{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-not-to-trust-language-models","title":"When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories","arxiv_id":"2212.10511","date":"2022-12-20","proceeding":null,"authors":["Alex Mallen","Akari Asai","Victor Zhong","Rajarshi Das","Daniel Khashabi","Hannaneh Hajishirzi"],"abstract":"Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the limitations of relying solely on their parameters to encode a wealth of world knowledge. This paper aims to understand LMs' strengths and limitations in memorizing factual knowledge, by conducting large-scale knowledge probing experiments of 10 models and 4 augmentation methods on PopQA, our new open-domain QA dataset with 14k questions. We find that LMs struggle with less popular factual knowledge, and that scaling fails to appreciably improve memorization of factual knowledge in the long tail. We then show that retrieval-augmented LMs largely outperform orders of magnitude larger LMs, while unassisted LMs remain competitive in questions about high-popularity entities. Based on those findings, we devise a simple, yet effective, method for powerful and efficient retrieval-augmented LMs, which retrieves non-parametric memories only when necessary. Experimental results show that this significantly improves models' performance while reducing the inference costs.","url_abs":"https://arxiv.org/abs/2212.10511v4","url_pdf":"https://arxiv.org/pdf/2212.10511v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-not-to-trust-language-models","repo_url":"https://github.com/alextmallen/adaptive-retrieval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-probing","task_name":"Knowledge Probing"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[{"slug":"popqa","name":"PopQA","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.10511","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.10511"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alextmallen/adaptive-retrieval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f06c53e363e1ebaa","entry":"get_memory_footprint","repo":"alextmallen/adaptive-retrieval","repo_kind":"official","path":"util_clm.py","file_url":"https://github.com/alextmallen/adaptive-retrieval/blob/HEAD/util_clm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f06c53e363e1ebaa"}},{"code_sha256_prefix":"098c21c81fd6e9e8","entry":"call_model","repo":"alextmallen/adaptive-retrieval","repo_kind":"official","path":"run_model.py","file_url":"https://github.com/alextmallen/adaptive-retrieval/blob/HEAD/run_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"098c21c81fd6e9e8"}},{"code_sha256_prefix":"0fcc8c8e5c34db2a","entry":"call_request","repo":"alextmallen/adaptive-retrieval","repo_kind":"official","path":"run_model.py","file_url":"https://github.com/alextmallen/adaptive-retrieval/blob/HEAD/run_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0fcc8c8e5c34db2a"}},{"code_sha256_prefix":"7e4c366c4dbf9568","entry":"convert_model_to_int8_on_gpu","repo":"alextmallen/adaptive-retrieval","repo_kind":"official","path":"util_clm.py","file_url":"https://github.com/alextmallen/adaptive-retrieval/blob/HEAD/util_clm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e4c366c4dbf9568"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}