{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mission-impossible-language-models","title":"Mission: Impossible Language Models","arxiv_id":"2401.06416","date":"2024-01-12","proceeding":null,"authors":["Julie Kallini","Isabel Papadimitriou","Richard Futrell","Kyle Mahowald","Christopher Potts"],"abstract":"Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. However, there is very little published experimental evidence to support such a claim. Here, we develop a set of synthetic impossible languages of differing complexity, each designed by systematically altering English data with unnatural word orders and grammar rules. These languages lie on an impossibility continuum: at one end are languages that are inherently impossible, such as random and irreversible shuffles of English words, and on the other, languages that may not be intuitively impossible but are often considered so in linguistics, particularly those with rules based on counting word positions. We report on a wide range of evaluations to assess the capacity of GPT-2 small models to learn these uncontroversially impossible languages, and crucially, we perform these assessments at various stages throughout training to compare the learning process for each language. Our core finding is that GPT-2 struggles to learn impossible languages when compared to English as a control, challenging the core claim. More importantly, we hope our approach opens up a productive line of inquiry in which different LLM architectures are tested on a variety of impossible languages in an effort to learn more about how LLMs can be used as tools for these cognitive and typological investigations.","url_abs":"https://arxiv.org/abs/2401.06416v2","url_pdf":"https://arxiv.org/pdf/2401.06416v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mission-impossible-language-models","repo_url":"https://github.com/jkallini/mission-impossible-language-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.06416","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.06416"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jkallini/mission-impossible-language-models","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a1988a1ab7fefaaf","entry":"compute_surprisals","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1988a1ab7fefaaf"}},{"code_sha256_prefix":"5bf34dc8d206d531","entry":"compute_token_probabilities","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bf34dc8d206d531"}},{"code_sha256_prefix":"ff77cf33dcba8fb3","entry":"create_attention_mask","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"perplexities/perplexities.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/perplexities/perplexities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff77cf33dcba8fb3"}},{"code_sha256_prefix":"547978906e547397","entry":"create_input_ids","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"perplexities/perplexities.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/perplexities/perplexities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"547978906e547397"}},{"code_sha256_prefix":"f954e3cc5ab8a96a","entry":"extract_phrases","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"edge_probing/load_phrase_data.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/edge_probing/load_phrase_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f954e3cc5ab8a96a"}},{"code_sha256_prefix":"50924a7f63ced3d3","entry":"get_perplexities","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"perplexities/perplexities.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/perplexities/perplexities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50924a7f63ced3d3"}},{"code_sha256_prefix":"74f6c2941d16cfcf","entry":"get_span","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"edge_probing/load_phrase_data.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/edge_probing/load_phrase_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74f6c2941d16cfcf"}},{"code_sha256_prefix":"bdb37b344c1fbfc5","entry":"max_pooling","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"edge_probing/edge_probing.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/edge_probing/edge_probing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdb37b344c1fbfc5"}},{"code_sha256_prefix":"7b8787922f993a93","entry":"mean_pooling","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"edge_probing/edge_probing.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/edge_probing/edge_probing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b8787922f993a93"}},{"code_sha256_prefix":"2dfc460bf3c3be3c","entry":"process_file","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"edge_probing/load_phrase_data.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/edge_probing/load_phrase_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dfc460bf3c3be3c"}},{"code_sha256_prefix":"6b2f4c65ccfed139","entry":"get_gpt2_tokenizer_with_markers","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b2f4c65ccfed139"}},{"code_sha256_prefix":"63f2e3ec2da6c808","entry":"get_layer_embedding","repo":"jkallini/mission-impossible-language-models","repo_kind":"official","path":"edge_probing/edge_probing.py","file_url":"https://github.com/jkallini/mission-impossible-language-models/blob/HEAD/edge_probing/edge_probing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63f2e3ec2da6c808"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}