{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sort-ing-vqa-models-contrastive-gradient","title":"SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency","arxiv_id":"2010.10038","date":"2020-10-20","proceeding":"NAACL 2021 4","authors":["Sameer Dharur","Purva Tendulkar","Dhruv Batra","Devi Parikh","Ramprasaath R. Selvaraju"],"abstract":"Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world -- they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level visual concepts in the image that models ideally should understand to be able to answer the higher level question correctly. To address this, we first present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image, and use this to evaluate VQA models on their ability to identify the relevant sub-questions needed to answer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant sub-questions higher than irrelevant questions for an <image, reasoning-question> pair. We show that SOrT improves model consistency by upto 6.5% points over existing baselines, while also improving visual grounding.","url_abs":"https://arxiv.org/abs/2010.10038v2","url_pdf":"https://arxiv.org/pdf/2010.10038v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sort-ing-vqa-models-contrastive-gradient","repo_url":"https://github.com/sameerdharur/sorting-vqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"interpretability","method_name":"Interpretability"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.10038","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.10038"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sameerdharur/sorting-vqa","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"099456faba640847","entry":"parse_annotation","repo":"sameerdharur/sorting-vqa","repo_kind":"official","path":"pythia/legacy/eval_model/eval_demo.py","file_url":"https://github.com/sameerdharur/sorting-vqa/blob/HEAD/pythia/legacy/eval_model/eval_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"099456faba640847"}},{"code_sha256_prefix":"2d308ec5cf958c0f","entry":"parse_ans","repo":"sameerdharur/sorting-vqa","repo_kind":"official","path":"pythia/legacy/eval_model/eval_demo.py","file_url":"https://github.com/sameerdharur/sorting-vqa/blob/HEAD/pythia/legacy/eval_model/eval_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2d308ec5cf958c0f"}},{"code_sha256_prefix":"8f1fbd47de11f9ff","entry":"Registry","repo":"sameerdharur/sorting-vqa","repo_kind":"official","path":"pythia/modules/losses.py","file_url":"https://github.com/sameerdharur/sorting-vqa/blob/HEAD/pythia/modules/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8f1fbd47de11f9ff"}},{"code_sha256_prefix":"7204aedbcd38ed89","entry":"SubQuestionLoss","repo":"sameerdharur/sorting-vqa","repo_kind":"official","path":"pythia/modules/losses.py","file_url":"https://github.com/sameerdharur/sorting-vqa/blob/HEAD/pythia/modules/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7204aedbcd38ed89"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}