{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/typical-decoding-for-natural-language","title":"Locally Typical Sampling","arxiv_id":"2202.00666","date":"2022-02-01","proceeding":null,"authors":["Clara Meister","Tiago Pimentel","Gian Wiher","Ryan Cotterell"],"abstract":"Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics, e.g., perplexity. This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language generation as a discrete stochastic process--which allows for an information-theoretic analysis--can provide new insights into the behavior of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We formally define the set of strings that meet this criterion: those for which each word has an information content close to the expected information content, i.e., the conditional entropy of our model. We then propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.","url_abs":"https://arxiv.org/abs/2202.00666v5","url_pdf":"https://arxiv.org/pdf/2202.00666v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"typical-decoding-for-natural-language","repo_url":"https://github.com/cimeister/typical-sampling","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"typical-decoding-for-natural-language","repo_url":"https://github.com/guinmoon/llmfarm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"typical-decoding-for-natural-language","repo_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/main/inference.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":"story-generation","task_name":"Story Generation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.00666","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.00666"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/guinmoon/llmfarm","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cimeister/typical-sampling","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/main/inference.py","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":19},"by_repo_kind":{"official":{"samples":15,"ran":6,"repositories":1},"listed":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"a727df5fd2f5599a","entry":"check_number_comma","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/convert_slow_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a727df5fd2f5599a"}},{"code_sha256_prefix":"d770e1dd1d799a47","entry":"ensure_valid_input","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d770e1dd1d799a47"}},{"code_sha256_prefix":"a4475703ff58ecf9","entry":"gelu_fast","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a4475703ff58ecf9"}},{"code_sha256_prefix":"45bb87451230d5e8","entry":"gelu_new","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"45bb87451230d5e8"}},{"code_sha256_prefix":"95f27ee4e1e8c5d3","entry":"gelu_python","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"95f27ee4e1e8c5d3"}},{"code_sha256_prefix":"7efe3db5c9e2f94d","entry":"generate_identified_filename","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7efe3db5c9e2f94d"}},{"code_sha256_prefix":"29095fa631853bbe","entry":"add_sq","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"src/backend.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/src/backend.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"29095fa631853bbe"}},{"code_sha256_prefix":"4e643fa0b277f39b","entry":"convert_slow_tokenizer","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/convert_slow_tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4e643fa0b277f39b"}},{"code_sha256_prefix":"52eb8468051d0996","entry":"decoder","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"52eb8468051d0996"}},{"code_sha256_prefix":"e187e5429b112df7","entry":"deepspeed_reinit","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/deepspeed.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/deepspeed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e187e5429b112df7"}},{"code_sha256_prefix":"c551a79d5bb1b062","entry":"detect_overflow","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/debug_utils.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/debug_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c551a79d5bb1b062"}},{"code_sha256_prefix":"58bc40af1d90b395","entry":"fn_if_dataclass","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"src/context.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/src/context.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"58bc40af1d90b395"}},{"code_sha256_prefix":"5c302780d36773f7","entry":"frame_encoder","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"script/quantized_vid2tfrecord.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/script/quantized_vid2tfrecord.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5c302780d36773f7"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"b5db92a796ef38f1","entry":"get_abs_min_max","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/debug_utils.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/debug_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b5db92a796ef38f1"}},{"code_sha256_prefix":"07e637f202f76ddb","entry":"get_configuration_file","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/configuration_utils.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/configuration_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"07e637f202f76ddb"}},{"code_sha256_prefix":"3449d7d975edfb89","entry":"infer_shapes","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3449d7d975edfb89"}},{"code_sha256_prefix":"5fbd4f093295aa97","entry":"init_class_copy","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"src/context.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/src/context.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5fbd4f093295aa97"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"e11d769ad67f0934","entry":"one_hot","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"inference.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e11d769ad67f0934"}},{"code_sha256_prefix":"f7c85a0d6b11bf74","entry":"promote_to","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"src/backend.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/src/backend.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f7c85a0d6b11bf74"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"cimeister/typical-sampling","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/cimeister/typical-sampling/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}},{"code_sha256_prefix":"417d7c14445e03e0","entry":"serialize","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"src/context.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/src/context.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"417d7c14445e03e0"}},{"code_sha256_prefix":"de13583a2c04b400","entry":"square_grad","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"src/backend.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/src/backend.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"de13583a2c04b400"}},{"code_sha256_prefix":"0a4d744fe302fd21","entry":"try_except","repo":"HomebrewNLP/HomebrewNLP-Jax","repo_kind":"listed","path":"script/quantized_vid2tfrecord.py","file_url":"https://github.com/HomebrewNLP/HomebrewNLP-Jax/blob/HEAD/script/quantized_vid2tfrecord.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0a4d744fe302fd21"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}