{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evidence-of-meaning-in-language-models","title":"Emergent Representations of Program Semantics in Language Models Trained on Programs","arxiv_id":"2305.11169","date":"2023-05-18","proceeding":null,"authors":["Charles Jin","Martin Rinard"],"abstract":"We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code. Our code is available at https://github.com/charlesjin/emergent-semantics.","url_abs":"https://arxiv.org/abs/2305.11169v3","url_pdf":"https://arxiv.org/pdf/2305.11169v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evidence-of-meaning-in-language-models","repo_url":"https://github.com/charlesjin/emergent-semantics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"program-synthesis","task_name":"Program Synthesis"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.11169","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.11169"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/charlesjin/emergent-semantics","reach":null}],"summary":{"ran_draft_wrong":12,"unverified":1},"by_repo_kind":{"official":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"19289791bccfe4aa","entry":"opposite1","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19289791bccfe4aa"}},{"code_sha256_prefix":"06ef4785e27794a2","entry":"perm1","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"06ef4785e27794a2"}},{"code_sha256_prefix":"9c15d06bd3b8286a","entry":"perm_nomarks_1","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c15d06bd3b8286a"}},{"code_sha256_prefix":"57fa76126e713940","entry":"perm_nomarks_2","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57fa76126e713940"}},{"code_sha256_prefix":"45f490a781f1c4b4","entry":"perm_nomarks_3","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45f490a781f1c4b4"}},{"code_sha256_prefix":"e5a9b785d18c1264","entry":"perm_nomarks_4","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5a9b785d18c1264"}},{"code_sha256_prefix":"017dd7ef99ca3626","entry":"perm_nomarks_5","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"017dd7ef99ca3626"}},{"code_sha256_prefix":"55d8912d375aaabb","entry":"random1","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55d8912d375aaabb"}},{"code_sha256_prefix":"c785730b24eadf32","entry":"unsemantic1","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c785730b24eadf32"}},{"code_sha256_prefix":"a54437cd7c70b3e9","entry":"unsemantic2","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a54437cd7c70b3e9"}},{"code_sha256_prefix":"c860b95cbf28a9cc","entry":"unsemantic3","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c860b95cbf28a9cc"}},{"code_sha256_prefix":"46316ec582bba1bf","entry":"unsemantic4","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46316ec582bba1bf"}},{"code_sha256_prefix":"de1956b9f51d4ea7","entry":"semantics_transformer","repo":"charlesjin/emergent-semantics","repo_kind":"official","path":"probe/alt.py","file_url":"https://github.com/charlesjin/emergent-semantics/blob/HEAD/probe/alt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de1956b9f51d4ea7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}