{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spectral-bias-outside-the-training-set-for","title":"Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime","arxiv_id":"2206.02927","date":"2022-06-06","proceeding":null,"authors":["Benjamin Bowman","Guido Montufar"],"abstract":"We provide quantitative bounds measuring the $L^2$ difference in function space between the trajectory of a finite-width network trained on finitely many samples from the idealized kernel dynamics of infinite width and infinite data. An implication of the bounds is that the network is biased to learn the top eigenfunctions of the Neural Tangent Kernel not just on the training set but over the entire input space. This bias depends on the model architecture and input distribution alone and thus does not depend on the target function which does not need to be in the RKHS of the kernel. The result is valid for deep architectures with fully connected, convolutional, and residual layers. Furthermore the width does not need to grow polynomially with the number of samples in order to obtain high probability bounds up to a stopping time. The proof exploits the low-effective-rank property of the Fisher Information Matrix at initialization, which implies a low effective dimension of the model (far smaller than the number of parameters). We conclude that local capacity control from the low effective rank of the Fisher Information Matrix is still underexplored theoretically.","url_abs":"https://arxiv.org/abs/2206.02927v2","url_pdf":"https://arxiv.org/pdf/2206.02927v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spectral-bias-outside-the-training-set-for","repo_url":"https://github.com/bbowman223/deepspec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.02927","atlas_url":"https://app.syntology.ai/?focus=2206.02927","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.02927"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/bbowman223/deepspec","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"043d8579b7468851","entry":"coor_loss","repo":"bbowman223/deepspec","repo_kind":"official","path":"deep/hercules/ntk_utils.py","file_url":"https://github.com/bbowman223/deepspec/blob/HEAD/deep/hercules/ntk_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"043d8579b7468851"}},{"code_sha256_prefix":"3a1adab214500167","entry":"grad_size","repo":"bbowman223/deepspec","repo_kind":"official","path":"deep/hercules/ntk_utils.py","file_url":"https://github.com/bbowman223/deepspec/blob/HEAD/deep/hercules/ntk_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a1adab214500167"}},{"code_sha256_prefix":"d5f71e934c65e691","entry":"is_supported","repo":"bbowman223/deepspec","repo_kind":"official","path":"deep/hercules/autograd_hacks.py","file_url":"https://github.com/bbowman223/deepspec/blob/HEAD/deep/hercules/autograd_hacks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5f71e934c65e691"}},{"code_sha256_prefix":"56afcb3496c8fd91","entry":"symsqrt","repo":"bbowman223/deepspec","repo_kind":"official","path":"deep/hercules/autograd_hacks.py","file_url":"https://github.com/bbowman223/deepspec/blob/HEAD/deep/hercules/autograd_hacks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56afcb3496c8fd91"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}