{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-24970","title":"Don't Go Breaking My LLM: The Impact of Pruning Attention Layers on Explanation Faithfulness and Confidence Calibration","arxiv_id":"2606.24970","date":"2026-06-23","proceeding":null,"authors":["Pietro Tropeano","Maria Maistro","Tuukka Ruotsalo","Christina Lioma"],"abstract":"Pruning Large Language Models (LLMs) reduces memory and inference costs by removing parts of the network, producing smaller models that retain most of their accuracy. As attention layers are the most resource-intensive parts of LLMs, pruning them is a promising compression strategy. Prior work shows that up to 33% of attention layers can be pruned with minimal accuracy loss. Nevertheless, the impact of attention pruning on model interpretability, specifically faithfulness and confidence calibration, remains unstudied. To address this gap, we study how pruning attention layers affects explanation faithfulness and confidence calibration across five LLMs and eight datasets. While the pruned models often maintain high accuracy, we find that their faithfulness and calibration often degrade. Notably, faithfulness and calibration can fluctuate significantly, even when accuracy remains stable, highlighting a misalignment between model confidence, interpretability, and accuracy. Our findings suggest that layer pruning can affect LLMs' interpretability and reliability in ways not captured by accuracy and efficiency measures alone. We recommend including explainability and calibration metrics when evaluating pruned models.","url_abs":"https://arxiv.org/abs/2606.24970","url_pdf":"https://arxiv.org/pdf/2606.24970","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.24970"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM","reach":null}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"ba8962fcfa6fa907","entry":"cosine_similarity","repo":"pietrotrope/Dont_Go_Breaking_My_LLM","repo_kind":"found_in_text","path":"block_drop.py","file_url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM/blob/HEAD/block_drop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba8962fcfa6fa907"}},{"code_sha256_prefix":"088898428fc14c3e","entry":"find_layers","repo":"pietrotrope/Dont_Go_Breaking_My_LLM","repo_kind":"found_in_text","path":"track_block_activations.py","file_url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM/blob/HEAD/track_block_activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"088898428fc14c3e"}},{"code_sha256_prefix":"c6d6ad3f768e73e9","entry":"get_blocks_scores_online","repo":"pietrotrope/Dont_Go_Breaking_My_LLM","repo_kind":"found_in_text","path":"block_drop.py","file_url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM/blob/HEAD/block_drop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6d6ad3f768e73e9"}},{"code_sha256_prefix":"02d8846e19bb9d2d","entry":"get_model_specific_variables","repo":"pietrotrope/Dont_Go_Breaking_My_LLM","repo_kind":"found_in_text","path":"track_block_activations.py","file_url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM/blob/HEAD/track_block_activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02d8846e19bb9d2d"}},{"code_sha256_prefix":"fedbf4713e04d9b8","entry":"retrieve_blocks","repo":"pietrotrope/Dont_Go_Breaking_My_LLM","repo_kind":"found_in_text","path":"block_drop.py","file_url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM/blob/HEAD/block_drop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fedbf4713e04d9b8"}},{"code_sha256_prefix":"6f2e0f632c7ae2da","entry":"truncate_colormap","repo":"pietrotrope/Dont_Go_Breaking_My_LLM","repo_kind":"found_in_text","path":"experiment_effectiveness_pruned.py","file_url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM/blob/HEAD/experiment_effectiveness_pruned.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f2e0f632c7ae2da"}},{"code_sha256_prefix":"269c15e4af75b825","entry":"prepare_calibration_input","repo":"pietrotrope/Dont_Go_Breaking_My_LLM","repo_kind":"found_in_text","path":"track_block_activations.py","file_url":"https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM/blob/HEAD/track_block_activations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"269c15e4af75b825"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}