{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/making-heads-or-tails-towards-semantically","title":"Making Heads or Tails: Towards Semantically Consistent Visual Counterfactuals","arxiv_id":"2203.12892","date":"2022-03-24","proceeding":null,"authors":["Simon Vandenhende","Dhruv Mahajan","Filip Radenovic","Deepti Ghadiyaram"],"abstract":"A visual counterfactual explanation replaces image regions in a query image with regions from a distractor image such that the system's decision on the transformed image changes to the distractor class. In this work, we present a novel framework for computing visual counterfactual explanations based on two key ideas. First, we enforce that the replaced and replacer regions contain the same semantic part, resulting in more semantically consistent explanations. Second, we use multiple distractor images in a computationally efficient way and obtain more discriminative explanations with fewer region replacements. Our approach is 27 % more semantically consistent and an order of magnitude faster than a competing method on three fine-grained image recognition datasets. We highlight the utility of our counterfactuals over existing works through machine teaching experiments where we teach humans to classify different bird species. We also complement our explanations with the vocabulary of parts and attributes that contributed the most to the system's decision. In this task as well, we obtain state-of-the-art results when using our counterfactual explanations relative to existing works, reinforcing the importance of semantically consistent explanations. Source code is available at https://github.com/facebookresearch/visual-counterfactuals.","url_abs":"https://arxiv.org/abs/2203.12892v2","url_pdf":"https://arxiv.org/pdf/2203.12892v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"making-heads-or-tails-towards-semantically","repo_url":"https://github.com/facebookresearch/visual-counterfactuals","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"counterfactual-explanation","task_name":"Counterfactual Explanation"},{"task_slug":"fine-grained-image-recognition","task_name":"Fine-Grained Image Recognition"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[{"method_slug":"counterfactuals","method_name":"Counterfactuals"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.12892","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.12892"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/facebookresearch/visual-counterfactuals","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"1f5bcf85e82b5309","entry":"_get_feature_representations_of_all_edits","repo":"facebookresearch/visual-counterfactuals","repo_kind":"official","path":"counterfactuals/explainer/counterfactuals.py","file_url":"https://github.com/facebookresearch/visual-counterfactuals/blob/HEAD/counterfactuals/explainer/counterfactuals.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1f5bcf85e82b5309"}},{"code_sha256_prefix":"7cb262eb5a948363","entry":"_find_knn_cells","repo":"facebookresearch/visual-counterfactuals","repo_kind":"official","path":"counterfactuals/explainer/counterfactuals.py","file_url":"https://github.com/facebookresearch/visual-counterfactuals/blob/HEAD/counterfactuals/explainer/counterfactuals.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7cb262eb5a948363"}},{"code_sha256_prefix":"f85a83243efdcbfc","entry":"_find_single_best_edit","repo":"facebookresearch/visual-counterfactuals","repo_kind":"official","path":"counterfactuals/explainer/counterfactuals.py","file_url":"https://github.com/facebookresearch/visual-counterfactuals/blob/HEAD/counterfactuals/explainer/counterfactuals.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f85a83243efdcbfc"}},{"code_sha256_prefix":"8d068706c4a0c0e1","entry":"compute_counterfactual","repo":"facebookresearch/visual-counterfactuals","repo_kind":"official","path":"counterfactuals/explainer/counterfactuals.py","file_url":"https://github.com/facebookresearch/visual-counterfactuals/blob/HEAD/counterfactuals/explainer/counterfactuals.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8d068706c4a0c0e1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}