{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stochastic-marginal-likelihood-gradients","title":"Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels","arxiv_id":"2306.03968","date":"2023-06-06","proceeding":null,"authors":["Alexander Immer","Tycho F. A. van der Ouderaa","Mark van der Wilk","Gunnar Rätsch","Bernhard Schölkopf"],"abstract":"Selecting hyperparameters in deep learning greatly impacts its effectiveness but requires manual effort and expertise. Recent works show that Bayesian model selection with Laplace approximations can allow to optimize such hyperparameters just like standard neural network parameters using gradients and on the training data. However, estimating a single hyperparameter gradient requires a pass through the entire dataset, limiting the scalability of such algorithms. In this work, we overcome this issue by introducing lower bounds to the linearized Laplace approximation of the marginal likelihood. In contrast to previous estimators, these bounds are amenable to stochastic-gradient-based optimization and allow to trade off estimation accuracy against computational complexity. We derive them using the function-space form of the linearized Laplace, which can be estimated using the neural tangent kernel. Experimentally, we show that the estimators can significantly accelerate gradient-based hyperparameter optimization.","url_abs":"https://arxiv.org/abs/2306.03968v1","url_pdf":"https://arxiv.org/pdf/2306.03968v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stochastic-marginal-likelihood-gradients","repo_url":"https://github.com/aleximmer/ntk-marglik","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.03968","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.03968"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/AlexImmer/ntk-marglik","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aleximmer/ntk-marglik","reach":null}],"summary":{"ran_draft_wrong":3,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"64a6789a3a000193","entry":"expand_prior_precision","repo":"AlexImmer/ntk-marglik","repo_kind":"official","path":"ntkmarglik/marglik.py","file_url":"https://github.com/AlexImmer/ntk-marglik/blob/HEAD/ntkmarglik/marglik.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64a6789a3a000193"}},{"code_sha256_prefix":"dc30499117b3d20a","entry":"get_frame","repo":"aleximmer/ntk-marglik","repo_kind":"official","path":"generate_illustration_figures.py","file_url":"https://github.com/aleximmer/ntk-marglik/blob/HEAD/generate_illustration_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc30499117b3d20a"}},{"code_sha256_prefix":"e58f08a5db6faad6","entry":"get_prior_hyperparams","repo":"AlexImmer/ntk-marglik","repo_kind":"official","path":"ntkmarglik/marglik.py","file_url":"https://github.com/AlexImmer/ntk-marglik/blob/HEAD/ntkmarglik/marglik.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e58f08a5db6faad6"}},{"code_sha256_prefix":"a11cbf1c7fbeb03b","entry":"get_runtime_frame","repo":"aleximmer/ntk-marglik","repo_kind":"official","path":"generate_illustration_figures.py","file_url":"https://github.com/aleximmer/ntk-marglik/blob/HEAD/generate_illustration_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a11cbf1c7fbeb03b"}},{"code_sha256_prefix":"a8cb194eb229f7e7","entry":"valid_performance","repo":"AlexImmer/ntk-marglik","repo_kind":"official","path":"ntkmarglik/marglik.py","file_url":"https://github.com/AlexImmer/ntk-marglik/blob/HEAD/ntkmarglik/marglik.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8cb194eb229f7e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}