{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/interpretation-of-neural-networks-is-fragile","title":"Interpretation of Neural Networks is Fragile","arxiv_id":"1710.10547","date":"2017-10-29","proceeding":null,"authors":["Amirata Ghorbani","Abubakar Abid","James Zou"],"abstract":"In order for machine learning to be deployed and trusted in many\napplications, it is crucial to be able to reliably explain why the machine\nlearning algorithm makes certain predictions. For example, if an algorithm\nclassifies a given pathology image to be a malignant tumor, then the doctor may\nneed to know which parts of the image led the algorithm to this classification.\nHow to interpret black-box predictors is thus an important and active area of\nresearch. A fundamental question is: how much can we trust the interpretation\nitself? In this paper, we show that interpretation of deep learning predictions\nis extremely fragile in the following sense: two perceptively indistinguishable\ninputs with the same predicted label can be assigned very different\ninterpretations. We systematically characterize the fragility of several\nwidely-used feature-importance interpretation methods (saliency maps, relevance\npropagation, and DeepLIFT) on ImageNet and CIFAR-10. Our experiments show that\neven small random perturbation can change the feature importance and new\nsystematic perturbations can lead to dramatically different interpretations\nwithout changing the label. We extend these results to show that\ninterpretations based on exemplars (e.g. influence functions) are similarly\nfragile. Our analysis of the geometry of the Hessian matrix gives insight on\nwhy fragility could be a fundamental challenge to the current interpretation\napproaches.","url_abs":"http://arxiv.org/abs/1710.10547v2","url_pdf":"http://arxiv.org/pdf/1710.10547v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"interpretation-of-neural-networks-is-fragile","repo_url":"https://github.com/amiratag/InterpretationFragility","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"interpretation-of-neural-networks-is-fragile","repo_url":"https://github.com/pytorch/captum","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"feature-importance","task_name":"Feature Importance"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1710.10547","atlas_url":"https://app.syntology.ai/?focus=1710.10547","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1710.10547"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pytorch/captum","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amiratag/InterpretationFragility","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"listed":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a871f75d37d138b","entry":"back_tensor","repo":"amiratag/InterpretationFragility","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/amiratag/InterpretationFragility/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a871f75d37d138b"}},{"code_sha256_prefix":"837df2aae88525f1","entry":"propagetion","repo":"amiratag/InterpretationFragility","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/amiratag/InterpretationFragility/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"837df2aae88525f1"}},{"code_sha256_prefix":"04287f8ff546fc30","entry":"retrain_VGG_on_flowers","repo":"amiratag/InterpretationFragility","repo_kind":"listed","path":"influence_utils.py","file_url":"https://github.com/amiratag/InterpretationFragility/blob/HEAD/influence_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04287f8ff546fc30"}},{"code_sha256_prefix":"3694bffc9033dca4","entry":"run_model","repo":"amiratag/InterpretationFragility","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/amiratag/InterpretationFragility/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3694bffc9033dca4"}},{"code_sha256_prefix":"680e5f9678530e9f","entry":"train_squeeze_on_cats_dogs","repo":"amiratag/InterpretationFragility","repo_kind":"listed","path":"influence_utils.py","file_url":"https://github.com/amiratag/InterpretationFragility/blob/HEAD/influence_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"680e5f9678530e9f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}