{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-tangent-kernel-beyond-the-infinite","title":"Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization","arxiv_id":"2202.00553","date":"2022-02-01","proceeding":null,"authors":["Mariia Seleznova","Gitta Kutyniok"],"abstract":"Neural Tangent Kernel (NTK) is widely used to analyze overparametrized neural networks due to the famous result by Jacot et al. (2018): in the infinite-width limit, the NTK is deterministic and constant during training. However, this result cannot explain the behavior of deep networks, since it generally does not hold if depth and width tend to infinity simultaneously. In this paper, we study the NTK of fully-connected ReLU networks with depth comparable to width. We prove that the NTK properties depend significantly on the depth-to-width ratio and the distribution of parameters at initialization. In fact, our results indicate the importance of the three phases in the hyperparameter space identified in Poole et al. (2016): ordered, chaotic and the edge of chaos (EOC). We derive exact expressions for the NTK dispersion in the infinite-depth-and-width limit in all three phases and conclude that the NTK variability grows exponentially with depth at the EOC and in the chaotic phase but not in the ordered phase. We also show that the NTK of deep networks may stay constant during training only in the ordered phase and discuss how the structure of the NTK matrix changes during training.","url_abs":"https://arxiv.org/abs/2202.00553v2","url_pdf":"https://arxiv.org/pdf/2202.00553v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-tangent-kernel-beyond-the-infinite","repo_url":"https://github.com/mselezniova/ntk_beyond_limit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[],"methods":[{"method_slug":"ntk","method_name":"NTK"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.00553","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.00553"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mselezniova/ntk_beyond_limit","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"8d643619024dd1d7","entry":"mse","repo":"mselezniova/ntk_beyond_limit","repo_kind":"official","path":"ntk_train_dynamics.py","file_url":"https://github.com/mselezniova/ntk_beyond_limit/blob/HEAD/ntk_train_dynamics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d643619024dd1d7"}},{"code_sha256_prefix":"f959b7f0c5db0fd5","entry":"K","repo":"mselezniova/ntk_beyond_limit","repo_kind":"official","path":"ntk_sample_mlp.py","file_url":"https://github.com/mselezniova/ntk_beyond_limit/blob/HEAD/ntk_sample_mlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f959b7f0c5db0fd5"}},{"code_sha256_prefix":"bcf913f1f987e7a8","entry":"K_matr","repo":"mselezniova/ntk_beyond_limit","repo_kind":"official","path":"ntk_sample_mlp.py","file_url":"https://github.com/mselezniova/ntk_beyond_limit/blob/HEAD/ntk_sample_mlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcf913f1f987e7a8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}