{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-closer-look-at-memorization-in-deep","title":"A Closer Look at Memorization in Deep Networks","arxiv_id":"1706.05394","date":"2017-06-16","proceeding":"ICML 2017 8","authors":["Devansh Arpit","Stanisław Jastrzębski","Nicolas Ballas","David Krueger","Emmanuel Bengio","Maxinder S. Kanwal","Tegan Maharaj","Asja Fischer","Aaron Courville","Yoshua Bengio","Simon Lacoste-Julien"],"abstract":"We examine the role of memorization in deep learning, drawing connections to\ncapacity, generalization, and adversarial robustness. While deep networks are\ncapable of memorizing noise data, our results suggest that they tend to\nprioritize learning simple patterns first. In our experiments, we expose\nqualitative differences in gradient-based optimization of deep neural networks\n(DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned\nexplicit regularization (e.g., dropout) we can degrade DNN training performance\non noise datasets without compromising generalization on real data. Our\nanalysis suggests that the notions of effective capacity which are dataset\nindependent are unlikely to explain the generalization performance of deep\nnetworks when trained with gradient based methods because training data itself\nplays an important role in determining the degree of memorization.","url_abs":"http://arxiv.org/abs/1706.05394v2","url_pdf":"http://arxiv.org/pdf/1706.05394v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-closer-look-at-memorization-in-deep","repo_url":"https://github.com/jerermyyoung/rtlearning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-closer-look-at-memorization-in-deep","repo_url":"https://github.com/chenpf1025/IDN/blob/master/ops.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"memorization","task_name":"Memorization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1706.05394","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}