{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-14018","title":"Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth","arxiv_id":"2607.14018","date":"2026-07-15","proceeding":null,"authors":["Katie Everett"],"abstract":"We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank. We show that skip connections trade off rank collapse against ensemble-like behavior, controlled by the relative scales of the branch and the skip: skip connections route the gradient around the residual branch, where rank is lost, rather than along the long gradient paths that encourage the layers to compose. The placement of the normalization layer controls this same tradeoff by setting the branch-to-skip ratio across depth, unifying much of the normalization placement and depth scaling literature, in particular why rank collapses for Post-Norm but plateaus for Pre-Norm. Other aspects of the architecture, like the two-matrix structure that expands and contracts the width, use additional parameters to preserve the representation or branch Jacobian rank. The second matrix decorrelates a coherent mean spike that would grow across blocks with a single matrix and uncentered activation, preventing the residual representation from collapsing. The width expansion between the two matrices keeps the branch Jacobian full rank: applying the rank-reducing activation in this expanded space leaves enough directions to span the original, at a width that follows a Marchenko--Pastur law. The initialization rank of the input--output Jacobian predicts which networks train on CIFAR-10. Taken together, we recast architecture design for deep networks as navigating an intrinsic tradeoff among rank collapse, ensemble-like behavior, and parameter count.","url_abs":"https://arxiv.org/abs/2607.14018","url_pdf":"https://arxiv.org/pdf/2607.14018","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.14018"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/everettk/transforming-rank-paper","reach":null}],"summary":{"ran":11,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3da80da6181d87bb","entry":"act_derivative","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"width_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/width_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3da80da6181d87bb"}},{"code_sha256_prefix":"e663da71a75b4ef9","entry":"build_network","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"norm_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/norm_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e663da71a75b4ef9"}},{"code_sha256_prefix":"2729f733fbf261a7","entry":"download_cifar10","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"data_gen/cifar_train.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/data_gen/cifar_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2729f733fbf261a7"}},{"code_sha256_prefix":"f2c1202f997e67f8","entry":"effective_rank","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"width_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/width_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2c1202f997e67f8"}},{"code_sha256_prefix":"0339b8622c35a739","entry":"erank","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"coherence_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/coherence_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0339b8622c35a739"}},{"code_sha256_prefix":"c642f696dc78fddf","entry":"rcrit","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"data_gen/diag_law_slopes_width.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/data_gen/diag_law_slopes_width.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c642f696dc78fddf"}},{"code_sha256_prefix":"a0ae3224d5eb7e2a","entry":"scaled_add","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"path_decomposition.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/path_decomposition.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0ae3224d5eb7e2a"}},{"code_sha256_prefix":"67903a73d514d9c3","entry":"scaled_matmul_left","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"path_decomposition.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/path_decomposition.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67903a73d514d9c3"}},{"code_sha256_prefix":"e0e617ccbd55e472","entry":"stable_rank","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"coherence_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/coherence_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0e617ccbd55e472"}},{"code_sha256_prefix":"506ded6ec2e02fa9","entry":"survival_rate","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"width_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/width_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"506ded6ec2e02fa9"}},{"code_sha256_prefix":"8fc255e3bfb4fed6","entry":"to_scaled","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"path_decomposition.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/path_decomposition.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fc255e3bfb4fed6"}},{"code_sha256_prefix":"90cc0b9e90bf4298","entry":"build_network","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"coherence_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/coherence_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90cc0b9e90bf4298"}},{"code_sha256_prefix":"fedbaafedc485c30","entry":"composed_erank","repo":"everettk/transforming-rank-paper","repo_kind":"found_in_text","path":"norm_model.py","file_url":"https://github.com/everettk/transforming-rank-paper/blob/HEAD/norm_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fedbaafedc485c30"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}