{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-understanding-learning","title":"Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation","arxiv_id":"1810.11750","date":"2018-10-28","proceeding":"NeurIPS 2018 12","authors":["Liwei Wang","Lunjia Hu","Jiayuan Gu","Yue Wu","Zhiqiang Hu","Kun He","John Hopcroft"],"abstract":"It is widely believed that learning good representations is one of the main\nreasons for the success of deep neural networks. Although highly intuitive,\nthere is a lack of theory and systematic approach quantitatively characterizing\nwhat representations do deep neural networks learn. In this work, we move a\ntiny step towards a theory and better understanding of the representations.\nSpecifically, we study a simpler problem: How similar are the representations\nlearned by two networks with identical architecture but trained from different\ninitializations. We develop a rigorous theory based on the neuron activation\nsubspace match model. The theory gives a complete characterization of the\nstructure of neuron activation subspace matches, where the core concepts are\nmaximum match and simple match which describe the overall and the finest\nsimilarity between sets of neurons in two networks respectively. We also\npropose efficient algorithms to find the maximum match and simple matches.\nFinally, we conduct extensive experiments using our algorithms. Experimental\nresults suggest that, surprisingly, representations learned by the same\nconvolutional layers of networks trained from different initializations are not\nas similar as prevalently expected, at least in terms of subspace match.","url_abs":"http://arxiv.org/abs/1810.11750v2","url_pdf":"http://arxiv.org/pdf/1810.11750v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-understanding-learning","repo_url":"https://github.com/MeckyWu/subspace-match","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1810.11750","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}