{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-importance-of-single-directions-for","title":"On the importance of single directions for generalization","arxiv_id":"1803.06959","date":"2018-03-19","proceeding":"ICLR 2018 1","authors":["Ari S. Morcos","David G. T. Barrett","Neil C. Rabinowitz","Matthew Botvinick"],"abstract":"Despite their ability to memorize large datasets, deep neural networks often\nachieve good generalization performance. However, the differences between the\nlearned solutions of networks which generalize and those which do not remain\nunclear. Additionally, the tuning properties of single directions (defined as\nthe activation of a single unit or some linear combination of units in response\nto some input) have been highlighted, but their importance has not been\nevaluated. Here, we connect these lines of inquiry to demonstrate that a\nnetwork's reliance on single directions is a good predictor of its\ngeneralization performance, across networks trained on datasets with different\nfractions of corrupted labels, across ensembles of networks trained on datasets\nwith unmodified labels, across different hyperparameters, and over the course\nof training. While dropout only regularizes this quantity up to a point, batch\nnormalization implicitly discourages single direction reliance, in part by\ndecreasing the class selectivity of individual units. Finally, we find that\nclass selectivity is a poor predictor of task importance, suggesting not only\nthat networks which generalize well minimize their dependence on individual\nunits by reducing their selectivity, but also that individually selective units\nmay not be necessary for strong network performance.","url_abs":"http://arxiv.org/abs/1803.06959v4","url_pdf":"http://arxiv.org/pdf/1803.06959v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-importance-of-single-directions-for","repo_url":"https://github.com/toshalpatel/Single-Directions","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1803.06959","atlas_url":"https://app.syntology.ai/?focus=1803.06959","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}