{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncovering-the-full-potential-of-visual","title":"Uncovering the Full Potential of Visual Grounding Methods in VQA","arxiv_id":"2401.07803","date":"2024-01-15","proceeding":null,"authors":["Daniel Reich","Tanja Schultz"],"abstract":"Visual Grounding (VG) methods in Visual Question Answering (VQA) attempt to improve VQA performance by strengthening a model's reliance on question-relevant visual information. The presence of such relevant information in the visual input is typically assumed in training and testing. This assumption, however, is inherently flawed when dealing with imperfect image representations common in large-scale VQA, where the information carried by visual features frequently deviates from expected ground-truth contents. As a result, training and testing of VG-methods is performed with largely inaccurate data, which obstructs proper assessment of their potential benefits. In this study, we demonstrate that current evaluation schemes for VG-methods are problematic due to the flawed assumption of availability of relevant visual information. Our experiments show that these methods can be much more effective when evaluation conditions are corrected. Code is provided on GitHub.","url_abs":"https://arxiv.org/abs/2401.07803v2","url_pdf":"https://arxiv.org/pdf/2401.07803v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncovering-the-full-potential-of-visual","repo_url":"https://github.com/dreichcsl/truevg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.07803","atlas_url":"https://app.syntology.ai/?focus=2401.07803","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.07803"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/dreichCSL/TrueVG","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"89bdd6decba2b415","entry":"compute_score_with_logits","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"components/losses.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/components/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89bdd6decba2b415"}},{"code_sha256_prefix":"69f8cd8f0aee84d4","entry":"find_infusion_objects","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"preprocessing/truevg/prep_infusion.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/preprocessing/truevg/prep_infusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69f8cd8f0aee84d4"}},{"code_sha256_prefix":"ed7ea0c3706614fe","entry":"get_FIscores_INF_semantic_matching","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"preprocessing/truevg/prep_hints.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/preprocessing/truevg/prep_hints.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed7ea0c3706614fe"}},{"code_sha256_prefix":"bbab318c93ed2fb0","entry":"get_FIscores_INF_spatial_matching","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"preprocessing/truevg/prep_hints.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/preprocessing/truevg/prep_hints.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbab318c93ed2fb0"}},{"code_sha256_prefix":"47aac06436736844","entry":"get_iou","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"preprocessing/truevg/prep_evaluation.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/preprocessing/truevg/prep_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47aac06436736844"}},{"code_sha256_prefix":"77c2c6aa4beabb9b","entry":"get_overlap","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"preprocessing/truevg/prep_evaluation.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/preprocessing/truevg/prep_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77c2c6aa4beabb9b"}},{"code_sha256_prefix":"1a250cde60dc9077","entry":"get_prediction_scores","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"components/losses.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/components/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a250cde60dc9077"}},{"code_sha256_prefix":"f314ec8478f617c7","entry":"get_truevg_qids","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"preprocessing/truevg/prep_evaluation.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/preprocessing/truevg/prep_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f314ec8478f617c7"}},{"code_sha256_prefix":"2b00263b6afeb3ee","entry":"instance_bce_with_logits","repo":"dreichCSL/TrueVG","repo_kind":"official","path":"components/losses.py","file_url":"https://github.com/dreichCSL/TrueVG/blob/HEAD/components/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b00263b6afeb3ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}