{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clevr-ref-diagnosing-visual-reasoning-with","title":"CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions","arxiv_id":"1901.00850","date":"2019-01-03","proceeding":"CVPR 2019 6","authors":["Runtao Liu","Chenxi Liu","Yutong Bai","Alan Yuille"],"abstract":"Referring object detection and referring image segmentation are important\ntasks that require joint understanding of visual information and natural\nlanguage. Yet there has been evidence that current benchmark datasets suffer\nfrom bias, and current state-of-the-art models cannot be easily evaluated on\ntheir intermediate reasoning process. To address these issues and complement\nsimilar efforts in visual question answering, we build CLEVR-Ref+, a synthetic\ndiagnostic dataset for referring expression comprehension. The precise\nlocations and attributes of the objects are readily available, and the\nreferring expressions are automatically associated with functional programs.\nThe synthetic nature allows control over dataset bias (through sampling\nstrategy), and the modular programs enable intermediate reasoning ground truth\nwithout human annotators.\n  In addition to evaluating several state-of-the-art models on CLEVR-Ref+, we\nalso propose IEP-Ref, a module network approach that significantly outperforms\nother models on our dataset. In particular, we present two interesting and\nimportant findings using IEP-Ref: (1) the module trained to transform feature\nmaps into segmentation masks can be attached to any intermediate module to\nreveal the entire reasoning process step-by-step; (2) even if all training data\nhas at least one object referred, IEP-Ref can correctly predict no-foreground\nwhen presented with false-premise referring expressions. To the best of our\nknowledge, this is the first direct and quantitative proof that neural modules\nbehave in the way they are intended.","url_abs":"http://arxiv.org/abs/1901.00850v2","url_pdf":"http://arxiv.org/pdf/1901.00850v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clevr-ref-diagnosing-visual-reasoning-with","repo_url":"https://github.com/arjunakula/neurips2021","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"clevr-ref-diagnosing-visual-reasoning-with","repo_url":"https://github.com/byahn2/clevr_ref","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"clevr-ref-diagnosing-visual-reasoning-with","repo_url":"https://github.com/ruotianluo/iep-ref","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[{"slug":"clevr-ref","name":"CLEVR-Ref+","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/referring-expression-segmentation-on-clevr","task":"Referring Expression Segmentation","dataset":"CLEVR-Ref+","model":"IEP-Ref (700K prog.)","rank_in_archive_order":1,"of":1,"metrics":{"IoU":"80.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1901.00850","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.00850"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruotianluo/iep-ref","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/byahn2/clevr_ref","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arjunakula/neurips2021","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"ran_honours":1,"unverified":1},"by_repo_kind":{"listed":{"samples":4,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"fba59b7f1bb4cc9e","entry":"compute_all_relationships","repo":"byahn2/clevr_ref","repo_kind":"listed","path":"image_generation/restore_render_images.py","file_url":"https://github.com/byahn2/clevr_ref/blob/HEAD/image_generation/restore_render_images.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fba59b7f1bb4cc9e"}},{"code_sha256_prefix":"9a69db254731308a","entry":"find_filter_options","repo":"byahn2/clevr_ref","repo_kind":"listed","path":"refexp_generation/generate_refexp.py","file_url":"https://github.com/byahn2/clevr_ref/blob/HEAD/refexp_generation/generate_refexp.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9a69db254731308a"}},{"code_sha256_prefix":"399457377d04215f","entry":"node_shallow_copy","repo":"byahn2/clevr_ref","repo_kind":"listed","path":"refexp_generation/generate_refexp.py","file_url":"https://github.com/byahn2/clevr_ref/blob/HEAD/refexp_generation/generate_refexp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"399457377d04215f"}},{"code_sha256_prefix":"394f2802fb88d6da","entry":"parse_int_list","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"394f2802fb88d6da"}},{"code_sha256_prefix":"1383aead1bcf4276","entry":"run_our_model_batch","repo":"ruotianluo/iep-ref","repo_kind":"listed","path":"scripts/run_model.py","file_url":"https://github.com/ruotianluo/iep-ref/blob/HEAD/scripts/run_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1383aead1bcf4276"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}