{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/insights-on-representational-similarity-in","title":"Insights on representational similarity in neural networks with canonical correlation","arxiv_id":"1806.05759","date":"2018-06-14","proceeding":"NeurIPS 2018 12","authors":["Ari S. Morcos","Maithra Raghu","Samy Bengio"],"abstract":"Comparing different neural network representations and determining how\nrepresentations evolve over time remain challenging open questions in our\nunderstanding of the function of neural networks. Comparing representations in\nneural networks is fundamentally difficult as the structure of representations\nvaries greatly, even across groups of networks trained on identical tasks, and\nover the course of training. Here, we develop projection weighted CCA\n(Canonical Correlation Analysis) as a tool for understanding neural networks,\nbuilding off of SVCCA, a recently proposed method (Raghu et al., 2017). We\nfirst improve the core method, showing how to differentiate between signal and\nnoise, and then apply this technique to compare across a group of CNNs,\ndemonstrating that networks which generalize converge to more similar\nrepresentations than networks which memorize, that wider networks converge to\nmore similar solutions than narrow networks, and that trained networks with\nidentical topology but different learning rates converge to distinct clusters\nwith diverse representations. We also investigate the representational dynamics\nof RNNs, across both training and sequential timesteps, finding that RNNs\nconverge in a bottom-up pattern over the course of training and that the hidden\nstate is highly variable over the course of a sequence, even when accounting\nfor linear transforms. Together, these results provide new insights into the\nfunction of CNNs and RNNs, and demonstrate the utility of using CCA to\nunderstand representations.","url_abs":"http://arxiv.org/abs/1806.05759v3","url_pdf":"http://arxiv.org/pdf/1806.05759v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"insights-on-representational-similarity-in","repo_url":"https://github.com/google/svcca","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"insights-on-representational-similarity-in","repo_url":"https://github.com/moskomule/cca.pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1806.05759","atlas_url":"https://app.syntology.ai/?focus=1806.05759","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1806.05759"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google/svcca","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/moskomule/cca.pytorch","reach":null}],"summary":{"ran_fixture":3,"ran_honours":1,"ran_draft_wrong":2},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"01472465bed8b268","entry":"compute_ccas","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"01472465bed8b268"}},{"code_sha256_prefix":"ea98389a1755bb3f","entry":"positivedef_matrix_sqrt","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ea98389a1755bb3f"}},{"code_sha256_prefix":"ffcbd13c7094948e","entry":"remove_small","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ffcbd13c7094948e"}},{"code_sha256_prefix":"38d282dc70b65fd7","entry":"svcca_distance","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"38d282dc70b65fd7"}},{"code_sha256_prefix":"25452d41e8e4d737","entry":"svd_reduction","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"25452d41e8e4d737"}},{"code_sha256_prefix":"eff74a29f37795b8","entry":"zero_mean","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"eff74a29f37795b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}