{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-do-neural-networks-outperform-kernel","title":"When Do Neural Networks Outperform Kernel Methods?","arxiv_id":"2006.13409","date":"2020-06-24","proceeding":"NeurIPS 2020 12","authors":["Behrooz Ghorbani","Song Mei","Theodor Misiakiewicz","Andrea Montanari"],"abstract":"For a certain scaling of the initialization of stochastic gradient descent (SGD), wide neural networks (NN) have been shown to be well approximated by reproducing kernel Hilbert space (RKHS) methods. Recent empirical work showed that, for some classification tasks, RKHS methods can replace NNs without a large loss in performance. On the other hand, two-layers NNs are known to encode richer smoothness classes than RKHS and we know of special examples for which SGD-trained NN provably outperform RKHS. This is true even in the wide network limit, for a different scaling of the initialization. How can we reconcile the above claims? For which tasks do NNs outperform RKHS? If covariates are nearly isotropic, RKHS methods suffer from the curse of dimensionality, while NNs can overcome it by learning the best low-dimensional representation. Here we show that this curse of dimensionality becomes milder if the covariates display the same low-dimensional structure as the target function, and we precisely characterize this tradeoff. Building on these results, we present the spiked covariates model that can capture in a unified framework both behaviors observed in earlier work. We hypothesize that such a latent low-dimensional structure is present in image classification. We test numerically this hypothesis by showing that specific perturbations of the training distribution degrade the performances of RKHS methods much more significantly than NNs.","url_abs":"https://arxiv.org/abs/2006.13409v2","url_pdf":"https://arxiv.org/pdf/2006.13409v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-do-neural-networks-outperform-kernel","repo_url":"https://github.com/bGhorbani/linearized_neural_networks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.13409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.13409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bGhorbani/linearized_neural_networks","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"03f5166a86390f22","entry":"NTK2","repo":"bGhorbani/linearized_neural_networks","repo_kind":"official","path":"Kernel_Fit.py","file_url":"https://github.com/bGhorbani/linearized_neural_networks/blob/HEAD/Kernel_Fit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"03f5166a86390f22"}},{"code_sha256_prefix":"96fa657135896791","entry":"RFK2","repo":"bGhorbani/linearized_neural_networks","repo_kind":"official","path":"Kernel_Fit.py","file_url":"https://github.com/bGhorbani/linearized_neural_networks/blob/HEAD/Kernel_Fit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96fa657135896791"}},{"code_sha256_prefix":"e5126224ca60d357","entry":"addHFNoise","repo":"bGhorbani/linearized_neural_networks","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/bGhorbani/linearized_neural_networks/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5126224ca60d357"}},{"code_sha256_prefix":"2aa0e5b8694c4755","entry":"cifar_input","repo":"bGhorbani/linearized_neural_networks","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/bGhorbani/linearized_neural_networks/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2aa0e5b8694c4755"}},{"code_sha256_prefix":"119e6c7e5139d644","entry":"fun","repo":"bGhorbani/linearized_neural_networks","repo_kind":"official","path":"RF_Kernel_Gen.py","file_url":"https://github.com/bGhorbani/linearized_neural_networks/blob/HEAD/RF_Kernel_Gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"119e6c7e5139d644"}},{"code_sha256_prefix":"306d40284d54ba3f","entry":"unpickle","repo":"bGhorbani/linearized_neural_networks","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/bGhorbani/linearized_neural_networks/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"306d40284d54ba3f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}