{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-tangent-kernel-convergence-and","title":"Neural Tangent Kernel: Convergence and Generalization in Neural Networks","arxiv_id":"1806.07572","date":"2018-06-20","proceeding":"NeurIPS 2018 12","authors":["Arthur Jacot","Franck Gabriel","Clément Hongler"],"abstract":"At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_\\theta$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function $f_\\theta$ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.","url_abs":"https://arxiv.org/abs/1806.07572v4","url_pdf":"https://arxiv.org/pdf/1806.07572v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-tangent-kernel-convergence-and","repo_url":"https://github.com/Triple-L/Wartegg_test","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"neural-tangent-kernel-convergence-and","repo_url":"https://github.com/ajacot/NTK_utils","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-tangent-kernel-convergence-and","repo_url":"https://github.com/jxVmnLgedVwv6mNcGCBy/NNspectra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"neural-tangent-kernel-convergence-and","repo_url":"https://github.com/thegregyang/GP4A","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"neural-tangent-kernel-convergence-and","repo_url":"https://github.com/thegregyang/NNspectra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-tangent-kernel-convergence-and","repo_url":"https://github.com/thegregyang/NTK4A","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"gaussian-processes","task_name":"Gaussian Processes"}],"methods":[{"method_slug":"ntk","method_name":"NTK"}],"datasets_introduced":[],"methods_introduced":[{"slug":"ntk","name":"NTK","full_name":"Neural Tangent Kernel"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1806.07572","atlas_url":"https://app.syntology.ai/?focus=1806.07572","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1806.07572"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ajacot/NTK_utils","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thegregyang/NNspectra","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thegregyang/GP4A","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Triple-L/Wartegg_test","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thegregyang/NTK4A","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jxVmnLgedVwv6mNcGCBy/NNspectra","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_fixture":1},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"c312daf70b4bd755","entry":"double_factorial","repo":"ajacot/NTK_utils","repo_kind":"listed","path":"network.py","file_url":"https://github.com/ajacot/NTK_utils/blob/HEAD/network.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c312daf70b4bd755"}},{"code_sha256_prefix":"3b06102285a0af20","entry":"make_mask","repo":"ajacot/NTK_utils","repo_kind":"listed","path":"network.py","file_url":"https://github.com/ajacot/NTK_utils/blob/HEAD/network.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b06102285a0af20"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}