{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visfis-visual-feature-importance-supervision","title":"VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives","arxiv_id":"2206.11212","date":"2022-06-22","proceeding":null,"authors":["Zhuofan Ying","Peter Hase","Mohit Bansal"],"abstract":"Many past works aim to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions. However, recent work has shown that performance gains from feature importance (FI) supervision for Visual Question Answering (VQA) tasks persist even with random supervision, suggesting that these methods do not meaningfully align model FI with human FI. In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason (RRR) metrics by optimizing for four key model objectives: (1) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance of predictions to changes in unimportant features (Invariance); and (4) alignment between model FI explanations and human FI explanations (Plausibility). Our best performing method, Visual Feature Importance Supervision (VisFIS), outperforms strong baselines on benchmark VQA datasets in terms of both in-distribution and out-of-distribution accuracy. While past work suggests that the mechanism for improved accuracy is through improved explanation plausibility, we show that this relationship depends crucially on explanation faithfulness (whether explanations truly represent the model's internal reasoning). Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful. Lastly, we show that, surprisingly, RRR metrics are not predictive of out-of-distribution model accuracy when controlling for a model's in-distribution accuracy, which calls into question the value of these metrics for evaluating model reasoning. All supporting code is available at https://github.com/zfying/visfis","url_abs":"https://arxiv.org/abs/2206.11212v2","url_pdf":"https://arxiv.org/pdf/2206.11212v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visfis-visual-feature-importance-supervision","repo_url":"https://github.com/zfying/visfis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"feature-importance","task_name":"Feature Importance"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.11212","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.11212"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zfying/visfis","reach":null}],"summary":{"ran_draft_wrong":3,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0950909878002e97","entry":"SHAP_weighting","repo":"zfying/visfis","repo_kind":"official","path":"components/feature_impt.py","file_url":"https://github.com/zfying/visfis/blob/HEAD/components/feature_impt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0950909878002e97"}},{"code_sha256_prefix":"511904da852d99b3","entry":"apply_mask","repo":"zfying/visfis","repo_kind":"official","path":"components/feature_impt.py","file_url":"https://github.com/zfying/visfis/blob/HEAD/components/feature_impt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"511904da852d99b3"}},{"code_sha256_prefix":"eec3cde903e4d38e","entry":"forward","repo":"zfying/visfis","repo_kind":"official","path":"components/feature_impt.py","file_url":"https://github.com/zfying/visfis/blob/HEAD/components/feature_impt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eec3cde903e4d38e"}},{"code_sha256_prefix":"9c5dc8aa19cdc86e","entry":"generate_masks","repo":"zfying/visfis","repo_kind":"official","path":"components/feature_impt.py","file_url":"https://github.com/zfying/visfis/blob/HEAD/components/feature_impt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c5dc8aa19cdc86e"}},{"code_sha256_prefix":"8884fda34ce90d8e","entry":"ncr","repo":"zfying/visfis","repo_kind":"official","path":"components/feature_impt.py","file_url":"https://github.com/zfying/visfis/blob/HEAD/components/feature_impt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8884fda34ce90d8e"}},{"code_sha256_prefix":"c3ee409c02ed594b","entry":"FI_forward","repo":"zfying/visfis","repo_kind":"official","path":"components/feature_impt.py","file_url":"https://github.com/zfying/visfis/blob/HEAD/components/feature_impt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ee409c02ed594b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}