{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/funnybirds-a-synthetic-vision-dataset-for-a","title":"FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methods","arxiv_id":"2308.06248","date":"2023-08-11","proceeding":"ICCV 2023 1","authors":["Robin Hesse","Simone Schaub-Meyer","Stefan Roth"],"abstract":"The field of explainable artificial intelligence (XAI) aims to uncover the inner workings of complex deep neural models. While being crucial for safety-critical domains, XAI inherently lacks ground-truth explanations, making its automatic evaluation an unsolved problem. We address this challenge by proposing a novel synthetic vision dataset, named FunnyBirds, and accompanying automatic evaluation protocols. Our dataset allows performing semantically meaningful image interventions, e.g., removing individual object parts, which has three important implications. First, it enables analyzing explanations on a part level, which is closer to human comprehension than existing methods that evaluate on a pixel level. Second, by comparing the model output for inputs with removed parts, we can estimate ground-truth part importances that should be reflected in the explanations. Third, by mapping individual explanations into a common space of part importances, we can analyze a variety of different explanation types in a single common framework. Using our tools, we report results for 24 different combinations of neural models and XAI methods, demonstrating the strengths and weaknesses of the assessed methods in a fully automatic and systematic manner.","url_abs":"https://arxiv.org/abs/2308.06248v1","url_pdf":"https://arxiv.org/pdf/2308.06248v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"funnybirds-a-synthetic-vision-dataset-for-a","repo_url":"https://github.com/visinf/funnybirds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"funnybirds-a-synthetic-vision-dataset-for-a","repo_url":"https://github.com/visinf/funnybirds-framework","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"xai","task_name":"Explainable Artificial Intelligence (XAI)"},{"task_slug":"explainable-artificial-intelligence","task_name":"Explainable artificial intelligence"}],"methods":[],"datasets_introduced":[{"slug":"funnybirds","name":"FunnyBirds","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2308.06248","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.06248"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/visinf/funnybirds","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/visinf/funnybirds-framework","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran":2,"ran_draft_wrong":2,"unverified":8},"by_repo_kind":{"official":{"samples":12,"ran":5,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"927e684e71ca3ee9","entry":"compute_rollout_attention","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/ViT/ViT_LRP.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/ViT/ViT_LRP.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"927e684e71ca3ee9"}},{"code_sha256_prefix":"5a98eb7226d25564","entry":"compute_rollout_attention","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/ViT/ViT_explanation_generator.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/ViT/ViT_explanation_generator.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a98eb7226d25564"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/resnet.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/resnet.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"5adf613fdebfcbd4","entry":"make_layers","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/vgg.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/vgg.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5adf613fdebfcbd4"}},{"code_sha256_prefix":"85b7b2881e0def7e","entry":"resnet18","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/resnet.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85b7b2881e0def7e"}},{"code_sha256_prefix":"628d6393478da40d","entry":"resnet18","repo":"visinf/funnybirds-framework","repo_kind":"listed","path":"models/resnet.py","file_url":"https://github.com/visinf/funnybirds-framework/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"628d6393478da40d"}},{"code_sha256_prefix":"76b1bf0c7e996f90","entry":"vgg11","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/vgg.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/vgg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76b1bf0c7e996f90"}},{"code_sha256_prefix":"df6cfaa1fbccd91a","entry":"vgg11_bn","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/vgg.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/vgg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df6cfaa1fbccd91a"}},{"code_sha256_prefix":"5fe2124317452946","entry":"vit_base_patch16_224","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/ViT/ViT_LRP.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/ViT/ViT_LRP.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fe2124317452946"}},{"code_sha256_prefix":"57e03ac784886a52","entry":"vit_base_patch16_224","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/ViT/ViT_new.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/ViT/ViT_new.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57e03ac784886a52"}},{"code_sha256_prefix":"005867bd71baa4bb","entry":"vit_large_patch16_224","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/ViT/ViT_LRP.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/ViT/ViT_LRP.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"005867bd71baa4bb"}},{"code_sha256_prefix":"fd65fde98f9bb5fc","entry":"vit_large_patch16_224","repo":"visinf/funnybirds","repo_kind":"official","path":"funnybirds_complete/models/ViT/ViT_new.py","file_url":"https://github.com/visinf/funnybirds/blob/HEAD/funnybirds_complete/models/ViT/ViT_new.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd65fde98f9bb5fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}