{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generalization-through-memorization-nearest","title":"Generalization through Memorization: Nearest Neighbor Language Models","arxiv_id":"1911.00172","date":"2019-11-01","proceeding":"ICLR 2020 1","authors":["Urvashi Khandelwal","Omer Levy","Dan Jurafsky","Luke Zettlemoyer","Mike Lewis"],"abstract":"We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.","url_abs":"https://arxiv.org/abs/1911.00172v2","url_pdf":"https://arxiv.org/pdf/1911.00172v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generalization-through-memorization-nearest","repo_url":"https://github.com/urvashik/knnlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"generalization-through-memorization-nearest","repo_url":"https://github.com/neulab/knn-transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"generalization-through-memorization-nearest","repo_url":"https://github.com/MS-Mind/MS-Code-06/tree/main/Neighbor2Neighbor","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"generalization-through-memorization-nearest","repo_url":"https://github.com/cordercorder/knn-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"generalization-through-memorization-nearest","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"memorization","task_name":"Memorization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"kNN-LM w/ Continuous Cache","rank_in_archive_order":10,"of":89,"metrics":{"Number of params":"247M","Test perplexity":"15.79","Validation perplexity":"15.81"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"kNN-LM","rank_in_archive_order":12,"of":89,"metrics":{"Number of params":"247M","Test perplexity":"16.12","Validation perplexity":"16.06"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1911.00172","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.00172"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MS-Mind/MS-Code-06/tree/main/Neighbor2Neighbor","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neulab/knn-transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cordercorder/knn-models","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/urvashik/knnlm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"05520f6527f04bde","entry":"get_dstore_path","repo":"neulab/knn-transformers","repo_kind":"listed","path":"knnlm.py","file_url":"https://github.com/neulab/knn-transformers/blob/HEAD/knnlm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05520f6527f04bde"}},{"code_sha256_prefix":"918cf1d7f24a77f5","entry":"get_index_path","repo":"neulab/knn-transformers","repo_kind":"listed","path":"knnlm.py","file_url":"https://github.com/neulab/knn-transformers/blob/HEAD/knnlm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"918cf1d7f24a77f5"}},{"code_sha256_prefix":"5b87ede004fa49b0","entry":"get_members_path","repo":"neulab/knn-transformers","repo_kind":"listed","path":"retomaton.py","file_url":"https://github.com/neulab/knn-transformers/blob/HEAD/retomaton.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b87ede004fa49b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}