{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/great-memory-shallow-reasoning-limits-of-k-nn","title":"Great Memory, Shallow Reasoning: Limits of $k$NN-LMs","arxiv_id":"2408.11815","date":"2024-08-21","proceeding":null,"authors":["Shangyi Geng","Wenting Zhao","Alexander M Rush"],"abstract":"$K$-nearest neighbor language models ($k$NN-LMs), which integrate retrieval with next-word prediction, have demonstrated strong performance in language modeling as well as downstream NLP benchmarks. These results have led researchers to argue that models trained on poor quality or outdated data could perform well by employing a $k$NN extension that has access to a higher-quality datastore. In this work, we ask whether this improved ability to recall information really translates into downstream abilities. We extensively evaluate $k$NN-LMs on a diverse set of tasks, ranging from sentiment classification and commonsense reasoning to multi-hop reasoning. Results show that $k$NN-LMs excel at memory-intensive tasks, where utilizing the patterns in the input is sufficient for determining the output, but struggle with reasoning tasks that require integrating multiple pieces of information to derive new knowledge. We further demonstrate through oracle experiments and qualitative analysis that even with perfect retrieval, $k$NN-LMs still fail to determine the correct answers, placing an upper bound on their reasoning performance. Code and datastores are released at https://github.com/GSYfate/knnlm-limits/.","url_abs":"https://arxiv.org/abs/2408.11815v1","url_pdf":"https://arxiv.org/pdf/2408.11815v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"great-memory-shallow-reasoning-limits-of-k-nn","repo_url":"https://github.com/gsyfate/knnlm-limits","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"sentiment-classification","task_name":"Sentiment Classification"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.11815","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.11815"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gsyfate/knnlm-limits","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"b14edf50d2544cc3","entry":"build_qa_prompt","repo":"gsyfate/knnlm-limits","repo_kind":"official","path":"eval_gsm8k.py","file_url":"https://github.com/gsyfate/knnlm-limits/blob/HEAD/eval_gsm8k.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b14edf50d2544cc3"}},{"code_sha256_prefix":"b81ff607da38bbac","entry":"extract_answer_and_chain","repo":"gsyfate/knnlm-limits","repo_kind":"official","path":"eval_gsm8k.py","file_url":"https://github.com/gsyfate/knnlm-limits/blob/HEAD/eval_gsm8k.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b81ff607da38bbac"}},{"code_sha256_prefix":"d42ce8e145a19d21","entry":"gen_prompt","repo":"gsyfate/knnlm-limits","repo_kind":"official","path":"eval_bbh.py","file_url":"https://github.com/gsyfate/knnlm-limits/blob/HEAD/eval_bbh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d42ce8e145a19d21"}},{"code_sha256_prefix":"7ab75c83ea401934","entry":"get_answer_from_model_output","repo":"gsyfate/knnlm-limits","repo_kind":"official","path":"eval_gsm8k.py","file_url":"https://github.com/gsyfate/knnlm-limits/blob/HEAD/eval_gsm8k.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ab75c83ea401934"}},{"code_sha256_prefix":"05520f6527f04bde","entry":"get_dstore_path","repo":"gsyfate/knnlm-limits","repo_kind":"official","path":"knnlm.py","file_url":"https://github.com/gsyfate/knnlm-limits/blob/HEAD/knnlm.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05520f6527f04bde"}},{"code_sha256_prefix":"918cf1d7f24a77f5","entry":"get_index_path","repo":"gsyfate/knnlm-limits","repo_kind":"official","path":"knnlm.py","file_url":"https://github.com/gsyfate/knnlm-limits/blob/HEAD/knnlm.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"918cf1d7f24a77f5"}},{"code_sha256_prefix":"699be993f9d76a78","entry":"evaluate","repo":"gsyfate/knnlm-limits","repo_kind":"official","path":"eval_bbh.py","file_url":"https://github.com/gsyfate/knnlm-limits/blob/HEAD/eval_bbh.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"699be993f9d76a78"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}