{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/emergent-world-representations-exploring-a","title":"Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task","arxiv_id":"2210.13382","date":"2022-10-24","proceeding":null,"authors":["Kenneth Li","Aspen K. Hopkins","David Bau","Fernanda Viégas","Hanspeter Pfister","Martin Wattenberg"],"abstract":"Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create \"latent saliency maps\" that can help explain predictions in human terms.","url_abs":"https://arxiv.org/abs/2210.13382v5","url_pdf":"https://arxiv.org/pdf/2210.13382v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"emergent-world-representations-exploring-a","repo_url":"https://github.com/likenneth/othello_world","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"emergent-world-representations-exploring-a","repo_url":"https://github.com/alxndrtl/othello_mamba","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"emergent-world-representations-exploring-a","repo_url":"https://github.com/keyonvafa/world-model-evaluation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"emergent-world-representations-exploring-a","repo_url":"https://github.com/openmoss/language-model-saes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.13382","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.13382"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openmoss/language-model-saes","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keyonvafa/world-model-evaluation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alxndrtl/othello_mamba","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/likenneth/othello_world","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1a0f3a6c603d52ed","entry":"permit","repo":"likenneth/othello_world","repo_kind":"official","path":"mechanistic_interpretability/mech_interp_othello_utils.py","file_url":"https://github.com/likenneth/othello_world/blob/HEAD/mechanistic_interpretability/mech_interp_othello_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a0f3a6c603d52ed"}},{"code_sha256_prefix":"1484fe530bc561e5","entry":"permit_reverse","repo":"likenneth/othello_world","repo_kind":"official","path":"mechanistic_interpretability/mech_interp_othello_utils.py","file_url":"https://github.com/likenneth/othello_world/blob/HEAD/mechanistic_interpretability/mech_interp_othello_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1484fe530bc561e5"}},{"code_sha256_prefix":"958673f09e3e1cd0","entry":"sample","repo":"likenneth/othello_world","repo_kind":"official","path":"mingpt/utils.py","file_url":"https://github.com/likenneth/othello_world/blob/HEAD/mingpt/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"958673f09e3e1cd0"}},{"code_sha256_prefix":"371f505cf3526d45","entry":"str_to_int","repo":"likenneth/othello_world","repo_kind":"official","path":"mechanistic_interpretability/tl_initial_exploration.py","file_url":"https://github.com/likenneth/othello_world/blob/HEAD/mechanistic_interpretability/tl_initial_exploration.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"371f505cf3526d45"}},{"code_sha256_prefix":"0a2c0f55c57d9f48","entry":"to_board_label","repo":"likenneth/othello_world","repo_kind":"official","path":"mechanistic_interpretability/mech_interp_othello_utils.py","file_url":"https://github.com/likenneth/othello_world/blob/HEAD/mechanistic_interpretability/mech_interp_othello_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a2c0f55c57d9f48"}},{"code_sha256_prefix":"0f0021cdea13e4da","entry":"top_k_logits","repo":"likenneth/othello_world","repo_kind":"official","path":"mingpt/utils.py","file_url":"https://github.com/likenneth/othello_world/blob/HEAD/mingpt/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f0021cdea13e4da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}