{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-task-collaborative-network-for-joint","title":"Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation","arxiv_id":"2003.08813","date":"2020-03-19","proceeding":"CVPR 2020 6","authors":["Gen Luo","Yiyi Zhou","Xiaoshuai Sun","Liujuan Cao","Chenglin Wu","Cheng Deng","Rongrong Ji"],"abstract":"Referring expression comprehension (REC) and segmentation (RES) are two highly-related tasks, which both aim at identifying the referent according to a natural language expression. In this paper, we propose a novel Multi-task Collaborative Network (MCN) to achieve a joint learning of REC and RES for the first time. In MCN, RES can help REC to achieve better language-vision alignment, while REC can help RES to better locate the referent. In addition, we address a key challenge in this multi-task setup, i.e., the prediction conflict, with two innovative designs namely, Consistency Energy Maximization (CEM) and Adaptive Soft Non-Located Suppression (ASNLS). Specifically, CEM enables REC and RES to focus on similar visual regions by maximizing the consistency energy between two tasks. ASNLS supresses the response of unrelated regions in RES based on the prediction of REC. To validate our model, we conduct extensive experiments on three benchmark datasets of REC and RES, i.e., RefCOCO, RefCOCO+ and RefCOCOg. The experimental results report the significant performance gains of MCN over all existing methods, i.e., up to +7.13% for REC and +11.50% for RES over SOTA, which well confirm the validity of our model for joint REC and RES learning.","url_abs":"https://arxiv.org/abs/2003.08813v1","url_pdf":"https://arxiv.org/pdf/2003.08813v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-task-collaborative-network-for-joint","repo_url":"https://github.com/luogen1996/MCN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"multi-task-collaborative-network-for-joint","repo_url":"https://github.com/xurige1995/3shnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"generalized-referring-expression","task_name":"Generalized Referring Expression Comprehension"},{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/generalized-referring-expression","task":"Generalized Referring Expression Comprehension","dataset":"gRefCOCO","model":"MCN","rank_in_archive_order":5,"of":5,"metrics":{"N-acc.":"30.6","Precision@(F1=1, IoU≥0.5)":"28.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2003.08813","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2003.08813"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luogen1996/MCN","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xurige1995/3shnet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"91dce3275884c301","entry":"aspp_decoder","repo":"luogen1996/MCN","repo_kind":"official","path":"model/mcn_model.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/mcn_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91dce3275884c301"}},{"code_sha256_prefix":"2c78556cd9635a1d","entry":"bert_output_sum","repo":"luogen1996/MCN","repo_kind":"official","path":"model/bert.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c78556cd9635a1d"}},{"code_sha256_prefix":"136643a1b2ec6aa9","entry":"create_from_dict","repo":"luogen1996/MCN","repo_kind":"official","path":"utils/parse_config.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/utils/parse_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"136643a1b2ec6aa9"}},{"code_sha256_prefix":"47337bf91f204990","entry":"expand_and_tile","repo":"luogen1996/MCN","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47337bf91f204990"}},{"code_sha256_prefix":"a6d7706f7a9a3eca","entry":"expand_and_tile_1","repo":"luogen1996/MCN","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6d7706f7a9a3eca"}},{"code_sha256_prefix":"73debbc8d76d4ded","entry":"get_checkpoint_paths","repo":"luogen1996/MCN","repo_kind":"official","path":"model/bert.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73debbc8d76d4ded"}},{"code_sha256_prefix":"5d597490de9d1f49","entry":"global_attention_block","repo":"luogen1996/MCN","repo_kind":"official","path":"model/garan.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/garan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d597490de9d1f49"}},{"code_sha256_prefix":"fe7e8a0ae28d05e2","entry":"letterbox_image","repo":"luogen1996/MCN","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/utils/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe7e8a0ae28d05e2"}},{"code_sha256_prefix":"364718e6449b4023","entry":"normalize_by_dim","repo":"luogen1996/MCN","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"364718e6449b4023"}},{"code_sha256_prefix":"cdf3547dcf481932","entry":"rand","repo":"luogen1996/MCN","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/utils/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cdf3547dcf481932"}},{"code_sha256_prefix":"b0d7842e5e5a7f8d","entry":"simple_fusion","repo":"luogen1996/MCN","repo_kind":"official","path":"model/mcn_model.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/mcn_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0d7842e5e5a7f8d"}},{"code_sha256_prefix":"daccc76ced7a707b","entry":"up_proj_cat_proj","repo":"luogen1996/MCN","repo_kind":"official","path":"model/mcn_model.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/model/mcn_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daccc76ced7a707b"}},{"code_sha256_prefix":"c7ed0cfdeeed82ac","entry":"update_from_dict","repo":"luogen1996/MCN","repo_kind":"official","path":"utils/parse_config.py","file_url":"https://github.com/luogen1996/MCN/blob/HEAD/utils/parse_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7ed0cfdeeed82ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}