{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-effect-of-model-size-on-llm-post-hoc","title":"The Effect of Model Size on LLM Post-hoc Explainability via LIME","arxiv_id":"2405.05348","date":"2024-05-08","proceeding":null,"authors":["Henning Heyen","Amy Widdicombe","Noah Y. Siegel","Maria Perez-Ortiz","Philip Treleaven"],"abstract":"Large language models (LLMs) are becoming bigger to boost performance. However, little is known about how explainability is affected by this trend. This work explores LIME explanations for DeBERTaV3 models of four different sizes on natural language inference (NLI) and zero-shot classification (ZSC) tasks. We evaluate the explanations based on their faithfulness to the models' internal decision processes and their plausibility, i.e. their agreement with human explanations. The key finding is that increased model size does not correlate with plausibility despite improved model performance, suggesting a misalignment between the LIME explanations and the models' internal processes as model size increases. Our results further suggest limitations regarding faithfulness metrics in NLI contexts.","url_abs":"https://arxiv.org/abs/2405.05348v1","url_pdf":"https://arxiv.org/pdf/2405.05348v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-effect-of-model-size-on-llm-post-hoc","repo_url":"https://github.com/henningheyen/scalability-of-llm-posthoc-explanations","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"lime","method_name":"LIME"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.05348","atlas_url":"https://app.syntology.ai/?focus=2405.05348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.05348"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/henningheyen/scalability-of-llm-posthoc-explanations","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e1c087e8e537738f","entry":"make_test_set_esnli","repo":"henningheyen/scalability-of-llm-posthoc-explanations","repo_kind":"official","path":"utils.py","file_url":"https://github.com/henningheyen/scalability-of-llm-posthoc-explanations/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1c087e8e537738f"}},{"code_sha256_prefix":"2fbae2b2de24931f","entry":"make_test_set_cose","repo":"henningheyen/scalability-of-llm-posthoc-explanations","repo_kind":"official","path":"utils.py","file_url":"https://github.com/henningheyen/scalability-of-llm-posthoc-explanations/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fbae2b2de24931f"}},{"code_sha256_prefix":"d428e5283a97296e","entry":"make_test_set_mnli","repo":"henningheyen/scalability-of-llm-posthoc-explanations","repo_kind":"official","path":"utils.py","file_url":"https://github.com/henningheyen/scalability-of-llm-posthoc-explanations/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d428e5283a97296e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}