{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-22582","title":"Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?","arxiv_id":"2603.22582","date":"2026-03-23","proceeding":null,"authors":["Richard J. Young"],"abstract":"Chain-of-thought (CoT) reasoning has been proposed as a transparency mechanism for large language models in safety-critical deployments, yet its effectiveness depends on faithfulness (whether models accurately verbalize the factors that actually influence their outputs), a property that prior evaluations have examined in only two proprietary models, finding acknowledgment rates as low as 25% for Claude 3.7 Sonnet and 39% for DeepSeek-R1. To extend this evaluation across the open-weight ecosystem, this study tests 12 open-weight reasoning models spanning 9 architectural families (7B-685B parameters) on 498 multiple-choice questions from MMLU and GPQA Diamond, injecting six categories of reasoning hints (sycophancy, consistency, visual pattern, metadata, grader hacking, and unethical information) and measuring the rate at which models acknowledge hint influence in their CoT when hints successfully alter answers. Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families, with consistency hints (35.5%) and sycophancy hints (53.9%) exhibiting the lowest acknowledgment rates. Training methodology and model family predict faithfulness more strongly than parameter count, and keyword-based analysis reveals a striking gap between thinking-token acknowledgment (approximately 87.5%) and answer-text acknowledgment (approximately 28.6%), suggesting that models internally recognize hint influence but systematically suppress this acknowledgment in their outputs. These findings carry direct implications for the viability of CoT monitoring as a safety mechanism and suggest that faithfulness is not a fixed property of reasoning models but varies systematically with architecture, training method, and the nature of the influencing cue.","url_abs":"https://arxiv.org/abs/2603.22582","url_pdf":"https://arxiv.org/pdf/2603.22582","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.22582","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.22582"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ricyoung/cot-faithfulness-open-models","reach":null}],"summary":{"unverified":7},"by_repo_kind":{"found_in_text":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"0ac531f976549c16","entry":"download_gpqa","repo":"ricyoung/cot-faithfulness-open-models","repo_kind":"found_in_text","path":"src/cot_faithfulness/data_prep.py","file_url":"https://github.com/ricyoung/cot-faithfulness-open-models/blob/HEAD/src/cot_faithfulness/data_prep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0ac531f976549c16"}},{"code_sha256_prefix":"837d830cf80fac49","entry":"download_mmlu","repo":"ricyoung/cot-faithfulness-open-models","repo_kind":"found_in_text","path":"src/cot_faithfulness/data_prep.py","file_url":"https://github.com/ricyoung/cot-faithfulness-open-models/blob/HEAD/src/cot_faithfulness/data_prep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"837d830cf80fac49"}},{"code_sha256_prefix":"8735a844d15f3a2a","entry":"extract_answer","repo":"ricyoung/cot-faithfulness-open-models","repo_kind":"found_in_text","path":"src/cot_faithfulness/answer_extraction.py","file_url":"https://github.com/ricyoung/cot-faithfulness-open-models/blob/HEAD/src/cot_faithfulness/answer_extraction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8735a844d15f3a2a"}},{"code_sha256_prefix":"e681e2bcbc608d2b","entry":"extract_answer_for_visual_pattern","repo":"ricyoung/cot-faithfulness-open-models","repo_kind":"found_in_text","path":"src/cot_faithfulness/answer_extraction.py","file_url":"https://github.com/ricyoung/cot-faithfulness-open-models/blob/HEAD/src/cot_faithfulness/answer_extraction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e681e2bcbc608d2b"}},{"code_sha256_prefix":"fb567031d6310663","entry":"make_question_id","repo":"ricyoung/cot-faithfulness-open-models","repo_kind":"found_in_text","path":"src/cot_faithfulness/schemas.py","file_url":"https://github.com/ricyoung/cot-faithfulness-open-models/blob/HEAD/src/cot_faithfulness/schemas.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fb567031d6310663"}},{"code_sha256_prefix":"ee3827fcc5cedd7c","entry":"split_thinking","repo":"ricyoung/cot-faithfulness-open-models","repo_kind":"found_in_text","path":"src/cot_faithfulness/clients.py","file_url":"https://github.com/ricyoung/cot-faithfulness-open-models/blob/HEAD/src/cot_faithfulness/clients.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ee3827fcc5cedd7c"}},{"code_sha256_prefix":"7ff6ba901e38b975","entry":"wilson_ci","repo":"ricyoung/cot-faithfulness-open-models","repo_kind":"found_in_text","path":"measuring-faithfulness/check_quality.py","file_url":"https://github.com/ricyoung/cot-faithfulness-open-models/blob/HEAD/measuring-faithfulness/check_quality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7ff6ba901e38b975"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}