{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/emergent-symbolic-mechanisms-support-abstract","title":"Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models","arxiv_id":"2502.20332","date":"2025-02-27","proceeding":null,"authors":["Yukang Yang","Declan Campbell","Kaixuan Huang","Mengdi Wang","Jonathan Cohen","Taylor Webb"],"abstract":"Many recent studies have found evidence for emergent reasoning capabilities in large language models, but debate persists concerning the robustness of these capabilities, and the extent to which they depend on structured reasoning mechanisms. To shed light on these issues, we perform a comprehensive study of the internal mechanisms that support abstract rule induction in an open-source language model (Llama3-70B). We identify an emergent symbolic architecture that implements abstract reasoning via a series of three computations. In early layers, symbol abstraction heads convert input tokens to abstract variables based on the relations between those tokens. In intermediate layers, symbolic induction heads perform sequence induction over these abstract variables. Finally, in later layers, retrieval heads predict the next token by retrieving the value associated with the predicted abstract variable. These results point toward a resolution of the longstanding debate between symbolic and neural network approaches, suggesting that emergent reasoning in neural networks depends on the emergence of symbolic mechanisms.","url_abs":"https://arxiv.org/abs/2502.20332v1","url_pdf":"https://arxiv.org/pdf/2502.20332v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"emergent-symbolic-mechanisms-support-abstract","repo_url":"https://github.com/yukang123/LLMSymbMech","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.20332","atlas_url":"https://app.syntology.ai/?focus=2502.20332","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.20332"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yukang123/LLMSymbMech","reach":null}],"summary":{"ran_draft_wrong":3,"ran_fixture":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"bb921e3c42f12f1c","entry":"cal_similarity","repo":"yukang123/LLMSymbMech","repo_kind":"official","path":"tasks/identity_rules/rsa.py","file_url":"https://github.com/yukang123/LLMSymbMech/blob/HEAD/tasks/identity_rules/rsa.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb921e3c42f12f1c"}},{"code_sha256_prefix":"80a59e73ae93bc92","entry":"compare_two_similarity_matrix","repo":"yukang123/LLMSymbMech","repo_kind":"official","path":"tasks/identity_rules/rsa.py","file_url":"https://github.com/yukang123/LLMSymbMech/blob/HEAD/tasks/identity_rules/rsa.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80a59e73ae93bc92"}},{"code_sha256_prefix":"656876ecdbda8a01","entry":"generate_prompts","repo":"yukang123/LLMSymbMech","repo_kind":"official","path":"tasks/identity_rules/cma.py","file_url":"https://github.com/yukang123/LLMSymbMech/blob/HEAD/tasks/identity_rules/cma.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"656876ecdbda8a01"}},{"code_sha256_prefix":"272afe9f573bef03","entry":"generate_prompts","repo":"yukang123/LLMSymbMech","repo_kind":"official","path":"tasks/identity_rules/rsa.py","file_url":"https://github.com/yukang123/LLMSymbMech/blob/HEAD/tasks/identity_rules/rsa.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"272afe9f573bef03"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}