{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-a-mechanistic-interpretation-of-multi","title":"Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models","arxiv_id":"2310.14491","date":"2023-10-23","proceeding":null,"authors":["Yifan Hou","Jiaoda Li","Yu Fei","Alessandro Stolfo","Wangchunshu Zhou","Guangtao Zeng","Antoine Bosselut","Mrinmaya Sachan"],"abstract":"Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. However, it is unclear whether LMs perform these tasks by cheating with answers memorized from pretraining corpus, or, via a multi-step reasoning mechanism. In this paper, we try to answer this question by exploring a mechanistic interpretation of LMs for multi-step reasoning tasks. Concretely, we hypothesize that the LM implicitly embeds a reasoning tree resembling the correct reasoning process within it. We test this hypothesis by introducing a new probing approach (called MechanisticProbe) that recovers the reasoning tree from the model's attention patterns. We use our probe to analyze two LMs: GPT-2 on a synthetic task (k-th smallest element), and LLaMA on two simple language-based reasoning tasks (ProofWriter & AI2 Reasoning Challenge). We show that MechanisticProbe is able to detect the information of the reasoning tree from the model's attentions for most examples, suggesting that the LM indeed is going through a process of multi-step reasoning within its architecture in many cases.","url_abs":"https://arxiv.org/abs/2310.14491v1","url_pdf":"https://arxiv.org/pdf/2310.14491v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-a-mechanistic-interpretation-of-multi","repo_url":"https://github.com/yifan-h/mechanisticprobe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"towards-a-mechanistic-interpretation-of-multi","repo_url":"https://github.com/eth-lre/llm_icl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"AI2 Reasoning Challenge"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.14491","atlas_url":"https://app.syntology.ai/?focus=2310.14491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.14491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yifan-h/mechanisticprobe","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eth-lre/llm_icl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"272fccc7faf0215a","entry":"avg","repo":"yifan-h/mechanisticprobe","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/yifan-h/mechanisticprobe/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"272fccc7faf0215a"}},{"code_sha256_prefix":"3ee83ac3291341c4","entry":"get_head_mask","repo":"yifan-h/mechanisticprobe","repo_kind":"official","path":"src/stask/prune.py","file_url":"https://github.com/yifan-h/mechanisticprobe/blob/HEAD/src/stask/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ee83ac3291341c4"}},{"code_sha256_prefix":"42783df89e16edd3","entry":"get_label_idx","repo":"yifan-h/mechanisticprobe","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/yifan-h/mechanisticprobe/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42783df89e16edd3"}},{"code_sha256_prefix":"c13d0749a5472646","entry":"get_proof_graph","repo":"yifan-h/mechanisticprobe","repo_kind":"official","path":"src/proofparser.py","file_url":"https://github.com/yifan-h/mechanisticprobe/blob/HEAD/src/proofparser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c13d0749a5472646"}},{"code_sha256_prefix":"5621c0d34ff2bff6","entry":"get_proof_graph_with_fail","repo":"yifan-h/mechanisticprobe","repo_kind":"official","path":"src/proofparser.py","file_url":"https://github.com/yifan-h/mechanisticprobe/blob/HEAD/src/proofparser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5621c0d34ff2bff6"}},{"code_sha256_prefix":"b70c417c1146f460","entry":"label_generator","repo":"yifan-h/mechanisticprobe","repo_kind":"official","path":"src/stask/probe.py","file_url":"https://github.com/yifan-h/mechanisticprobe/blob/HEAD/src/stask/probe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b70c417c1146f460"}},{"code_sha256_prefix":"5b2bb4a3ea7d3329","entry":"knn_classifier","repo":"yifan-h/mechanisticprobe","repo_kind":"official","path":"src/stask/probe.py","file_url":"https://github.com/yifan-h/mechanisticprobe/blob/HEAD/src/stask/probe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b2bb4a3ea7d3329"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}