{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-neural-networks-memorize-and-why","title":"What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation","arxiv_id":"2008.03703","date":"2020-08-09","proceeding":"NeurIPS 2020 12","authors":["Vitaly Feldman","Chiyuan Zhang"],"abstract":"Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given. In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).","url_abs":"https://arxiv.org/abs/2008.03703v1","url_pdf":"https://arxiv.org/pdf/2008.03703v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-neural-networks-memorize-and-why","repo_url":"https://github.com/google-research/heldout-influence-estimation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"memorization","task_name":"Memorization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2008.03703","atlas_url":"https://app.syntology.ai/?focus=2008.03703","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2008.03703"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/heldout-influence-estimation","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":6},"by_repo_kind":{"listed":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a8f998b9ec355770","entry":"one_hot","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"mnist-example/mnist_infl_mem.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/mnist-example/mnist_infl_mem.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a8f998b9ec355770"}},{"code_sha256_prefix":"46fa247f2c3e6e4d","entry":"preprocess_for_eval","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"libdata/imagenet_preprocessing.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/libdata/imagenet_preprocessing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46fa247f2c3e6e4d"}},{"code_sha256_prefix":"c46504ca7d0285e1","entry":"preprocess_for_train","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"libdata/imagenet_preprocessing.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/libdata/imagenet_preprocessing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c46504ca7d0285e1"}},{"code_sha256_prefix":"84d9720661af92de","entry":"batch_correctness","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"mnist-example/mnist_infl_mem.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/mnist-example/mnist_infl_mem.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84d9720661af92de"}},{"code_sha256_prefix":"b54e5d5d7eccda19","entry":"distorted_bounding_box_crop","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"libdata/imagenet_preprocessing.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/libdata/imagenet_preprocessing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b54e5d5d7eccda19"}},{"code_sha256_prefix":"223b796c7c48a3c1","entry":"load_data","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"libdata/indexed_tfrecords.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/libdata/indexed_tfrecords.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"223b796c7c48a3c1"}},{"code_sha256_prefix":"8f1d8f0c8fede07a","entry":"loss","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"mnist-example/mnist_infl_mem.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/mnist-example/mnist_infl_mem.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8f1d8f0c8fede07a"}},{"code_sha256_prefix":"dd68508cbccc4bf8","entry":"parse_record","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"libdata/indexed_tfrecords.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/libdata/indexed_tfrecords.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd68508cbccc4bf8"}},{"code_sha256_prefix":"c00febd1ff3e9ed5","entry":"process_record_dataset","repo":"google-research/heldout-influence-estimation","repo_kind":"listed","path":"libdata/indexed_tfrecords.py","file_url":"https://github.com/google-research/heldout-influence-estimation/blob/HEAD/libdata/indexed_tfrecords.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c00febd1ff3e9ed5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}