{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-question-based-biases-assessing","title":"Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering","arxiv_id":"2104.03149","date":"2021-04-07","proceeding":"ICCV 2021 10","authors":["Corentin Dancette","Remi Cadene","Damien Teney","Matthieu Cord"],"abstract":"We introduce an evaluation methodology for visual question answering (VQA) to better diagnose cases of shortcut learning. These cases happen when a model exploits spurious statistical regularities to produce correct answers but does not actually deploy the desired behavior. There is a need to identify possible shortcuts in a dataset and assess their use before deploying a model in the real world. The research community in VQA has focused exclusively on question-based shortcuts, where a model might, for example, answer \"What is the color of the sky\" with \"blue\" by relying mostly on the question-conditional training prior and give little weight to visual evidence. We go a step further and consider multimodal shortcuts that involve both questions and images. We first identify potential shortcuts in the popular VQA v2 training set by mining trivial predictive rules such as co-occurrences of words and visual elements. We then introduce VQA-CounterExamples (VQA-CE), an evaluation protocol based on our subset of CounterExamples i.e. image-question-answer triplets where our rules lead to incorrect answers. We use this new evaluation in a large-scale study of existing approaches for VQA. We demonstrate that even state-of-the-art models perform poorly and that existing techniques to reduce biases are largely ineffective in this context. Our findings suggest that past work on question-based biases in VQA has only addressed one facet of a complex issue. The code for our method is available at https://github.com/cdancette/detect-shortcuts.","url_abs":"https://arxiv.org/abs/2104.03149v3","url_pdf":"https://arxiv.org/pdf/2104.03149v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-question-based-biases-assessing","repo_url":"https://github.com/cdancette/detect-shortcuts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[{"slug":"vqa-ce","name":"VQA-CE","full_name":"VQA Counterexamples"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"RandImg","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy (Counterexamples)":"34.41"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"LMH + CSS","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy (Counterexamples)":"34.36"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"LFF","rank_in_archive_order":3,"of":9,"metrics":{"Accuracy (Counterexamples)":"34.27"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"LMH","rank_in_archive_order":4,"of":9,"metrics":{"Accuracy (Counterexamples)":"34.26"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"UpDown","rank_in_archive_order":5,"of":9,"metrics":{"Accuracy (Counterexamples)":"33.91"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"ESR","rank_in_archive_order":6,"of":9,"metrics":{"Accuracy (Counterexamples)":"33.26"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"LMH + RMFE","rank_in_archive_order":7,"of":9,"metrics":{"Accuracy (Counterexamples)":"33.14"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"BLOCK","rank_in_archive_order":8,"of":9,"metrics":{"Accuracy (Counterexamples)":"32.91"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-ce","task":"Visual Question Answering (VQA)","dataset":"VQA-CE","model":"RUBi","rank_in_archive_order":9,"of":9,"metrics":{"Accuracy (Counterexamples)":"32.25"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.03149","atlas_url":"https://app.syntology.ai/?focus=2104.03149","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.03149"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cdancette/detect-shortcuts","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"f151d3d0ad479012","entry":"loadjson","repo":"cdancette/detect-shortcuts","repo_kind":"official","path":"vqa.py","file_url":"https://github.com/cdancette/detect-shortcuts/blob/HEAD/vqa.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f151d3d0ad479012"}},{"code_sha256_prefix":"5d8f3b1c6d1dece9","entry":"run_gminer","repo":"cdancette/detect-shortcuts","repo_kind":"official","path":"rule_mining.py","file_url":"https://github.com/cdancette/detect-shortcuts/blob/HEAD/rule_mining.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d8f3b1c6d1dece9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}