{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/super-clevr-a-virtual-benchmark-to-diagnose","title":"Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning","arxiv_id":"2212.00259","date":"2022-12-01","proceeding":"CVPR 2023 1","authors":["Zhuowan Li","Xingrui Wang","Elias Stengel-Eskin","Adam Kortylewski","Wufei Ma","Benjamin Van Durme","Alan Yuille"],"abstract":"Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.","url_abs":"https://arxiv.org/abs/2212.00259v2","url_pdf":"https://arxiv.org/pdf/2212.00259v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"super-clevr-a-virtual-benchmark-to-diagnose","repo_url":"https://github.com/lizw14/super-clevr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"super-clevr-a-virtual-benchmark-to-diagnose","repo_url":"https://github.com/XingruiWang/superclevr-3D-question","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[{"slug":"super-clevr","name":"Super-CLEVR","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.00259","atlas_url":"https://app.syntology.ai/?focus=2212.00259","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.00259"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lizw14/super-clevr","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/XingruiWang/superclevr-3D-question","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b5799be0850f8e3b","entry":"drop_redundant_filters","repo":"lizw14/super-clevr","repo_kind":"official","path":"question_generation/generate_questions.py","file_url":"https://github.com/lizw14/super-clevr/blob/HEAD/question_generation/generate_questions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b5799be0850f8e3b"}},{"code_sha256_prefix":"6783ec2ca3611f61","entry":"find_partfilter_options","repo":"lizw14/super-clevr","repo_kind":"official","path":"question_generation/generate_questions.py","file_url":"https://github.com/lizw14/super-clevr/blob/HEAD/question_generation/generate_questions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6783ec2ca3611f61"}},{"code_sha256_prefix":"0d9ea0a85da55fc6","entry":"subsumes","repo":"lizw14/super-clevr","repo_kind":"official","path":"question_generation/generate_questions.py","file_url":"https://github.com/lizw14/super-clevr/blob/HEAD/question_generation/generate_questions.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0d9ea0a85da55fc6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}