{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/elephants-never-forget-memorization-and","title":"Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models","arxiv_id":"2404.06209","date":"2024-04-09","proceeding":null,"authors":["Sebastian Bordt","Harsha Nori","Vanessa Rodrigues","Besmira Nushi","Rich Caruana"],"abstract":"While many have shown how Large Language Models (LLMs) can be applied to a diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Specifically, we introduce a variety of different techniques to assess whether a language model has seen a tabular dataset during training. This investigation reveals that LLMs have memorized many popular tabular datasets verbatim. We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training. We find that LLMs perform better on datasets seen during training, indicating that memorization leads to overfitting. At the same time, LLMs show non-trivial performance on novel datasets and are surprisingly robust to data transformations. We then investigate the in-context statistical learning abilities of LLMs. While LLMs are significantly better than random at solving statistical classification problems, the sample efficiency of few-shot learning lags behind traditional statistical learning algorithms, especially as the dimension of the problem increases. This suggests that much of the observed few-shot performance on novel real-world datasets is due to the LLM's world knowledge. Overall, our results highlight the importance of testing whether an LLM has seen an evaluation dataset during pre-training. We release the https://github.com/interpretml/LLM-Tabular-Memorization-Checker Python package to test LLMs for memorization of tabular datasets.","url_abs":"https://arxiv.org/abs/2404.06209v3","url_pdf":"https://arxiv.org/pdf/2404.06209v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"elephants-never-forget-memorization-and","repo_url":"https://github.com/interpretml/llm-tabular-memorization-checker","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.06209","atlas_url":"https://app.syntology.ai/?focus=2404.06209","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.06209"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/interpretml/llm-tabular-memorization-checker","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":11},"by_repo_kind":{"official":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16ba085a47db0f85","entry":"accuracy","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"colm-2024-paper-code/statutils.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/colm-2024-paper-code/statutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16ba085a47db0f85"}},{"code_sha256_prefix":"6bd8df53f13381fc","entry":"apply_transform","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"tabmemcheck/datasets/load.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/tabmemcheck/datasets/load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bd8df53f13381fc"}},{"code_sha256_prefix":"c2b405b9b0e0b6f9","entry":"claude_setup","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"tabmemcheck/llm.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/tabmemcheck/llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2b405b9b0e0b6f9"}},{"code_sha256_prefix":"c98c5fd398edb05e","entry":"fit_logistic_regression_cv","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"colm-2024-paper-code/statutils.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/colm-2024-paper-code/statutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c98c5fd398edb05e"}},{"code_sha256_prefix":"2691ff2e5953d672","entry":"format_data_point","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"colm-2024-paper-code/tabular_queries.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/colm-2024-paper-code/tabular_queries.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2691ff2e5953d672"}},{"code_sha256_prefix":"cce4367cc3f4f2d0","entry":"gemini_setup","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"tabmemcheck/llm.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/tabmemcheck/llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cce4367cc3f4f2d0"}},{"code_sha256_prefix":"671a4a5cd18f9f00","entry":"get_dataset_name","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"tabmemcheck/utils.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/tabmemcheck/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"671a4a5cd18f9f00"}},{"code_sha256_prefix":"cbcb22e2d695eba2","entry":"openai_setup","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"tabmemcheck/llm.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/tabmemcheck/llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbcb22e2d695eba2"}},{"code_sha256_prefix":"f9c04ea44364990e","entry":"rename_and_recode","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"tabmemcheck/datasets/load.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/tabmemcheck/datasets/load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9c04ea44364990e"}},{"code_sha256_prefix":"be58358dc4b19f7d","entry":"roc_auc","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"colm-2024-paper-code/statutils.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/colm-2024-paper-code/statutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be58358dc4b19f7d"}},{"code_sha256_prefix":"1f6b41345e9e24f0","entry":"string_strip","repo":"interpretml/llm-tabular-memorization-checker","repo_kind":"official","path":"tabmemcheck/analysis.py","file_url":"https://github.com/interpretml/llm-tabular-memorization-checker/blob/HEAD/tabmemcheck/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f6b41345e9e24f0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}