{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-measuring-faithfulness-of-natural-language","title":"On Measuring Faithfulness or Self-consistency of Natural Language Explanations","arxiv_id":"2311.07466","date":"2023-11-13","proceeding":null,"authors":["Letitia Parcalabescu","Anette Frank"],"abstract":"Large language models (LLMs) can explain their predictions through post-hoc or Chain-of-Thought (CoT) explanations. But an LLM could make up reasonably sounding explanations that are unfaithful to its underlying reasoning. Recent work has designed tests that aim to judge the faithfulness of post-hoc or CoT explanations. In this work we argue that these faithfulness tests do not measure faithfulness to the models' inner workings -- but rather their self-consistency at output level. Our contributions are three-fold: i) We clarify the status of faithfulness tests in view of model explainability, characterising them as self-consistency tests instead. This assessment we underline by ii) constructing a Comparative Consistency Bank for self-consistency tests that for the first time compares existing tests on a common suite of 11 open LLMs and 5 tasks -- including iii) our new self-consistency measure CC-SHAP. CC-SHAP is a fine-grained measure (not a test) of LLM self-consistency. It compares how a model's input contributes to the predicted answer and to generating the explanation. Our fine-grained CC-SHAP metric allows us iii) to compare LLM behaviour when making predictions and to analyse the effect of other consistency tests at a deeper level, which takes us one step further towards measuring faithfulness by bringing us closer to the internals of the model than strictly surface output-oriented tests. Our code is available at \\url{https://github.com/Heidelberg-NLP/CC-SHAP}","url_abs":"https://arxiv.org/abs/2311.07466v3","url_pdf":"https://arxiv.org/pdf/2311.07466v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-measuring-faithfulness-of-natural-language","repo_url":"https://github.com/heidelberg-nlp/cc-shap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"on-measuring-faithfulness-of-natural-language","repo_url":"https://github.com/heidelberg-nlp/cc-shap-vlm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.07466","atlas_url":"https://app.syntology.ai/?focus=2311.07466","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.07466"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heidelberg-nlp/cc-shap","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heidelberg-nlp/cc-shap-vlm","reach":null}],"summary":{"ran_honours":1,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"bb81e3e4a9f52ed1","entry":"compute_mm_score","repo":"heidelberg-nlp/cc-shap-vlm","repo_kind":"listed","path":"ccshapvlm/cc_shap.py","file_url":"https://github.com/heidelberg-nlp/cc-shap-vlm/blob/HEAD/ccshapvlm/cc_shap.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb81e3e4a9f52ed1"}},{"code_sha256_prefix":"2f372a731d99a85b","entry":"deeplift_grad","repo":"heidelberg-nlp/cc-shap","repo_kind":"official","path":"shap/explainers/_deep/deep_pytorch.py","file_url":"https://github.com/heidelberg-nlp/cc-shap/blob/HEAD/shap/explainers/_deep/deep_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f372a731d99a85b"}},{"code_sha256_prefix":"10adb065b8e203e6","entry":"deeplift_tensor_grad","repo":"heidelberg-nlp/cc-shap","repo_kind":"official","path":"shap/explainers/_deep/deep_pytorch.py","file_url":"https://github.com/heidelberg-nlp/cc-shap/blob/HEAD/shap/explainers/_deep/deep_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10adb065b8e203e6"}},{"code_sha256_prefix":"30930885360cc5fc","entry":"passthrough","repo":"heidelberg-nlp/cc-shap","repo_kind":"official","path":"shap/explainers/_deep/deep_pytorch.py","file_url":"https://github.com/heidelberg-nlp/cc-shap/blob/HEAD/shap/explainers/_deep/deep_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30930885360cc5fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}