{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stochastic-gradient-descent-for-gaussian","title":"Stochastic Gradient Descent for Gaussian Processes Done Right","arxiv_id":"2310.20581","date":"2023-10-31","proceeding":null,"authors":["Jihao Andreas Lin","Shreyas Padhy","Javier Antorán","Austin Tripp","Alexander Terenin","Csaba Szepesvári","José Miguel Hernández-Lobato","David Janz"],"abstract":"As is well known, both sampling from the posterior and computing the mean of the posterior in Gaussian process regression reduces to solving a large linear system of equations. We study the use of stochastic gradient descent for solving this linear system, and show that when \\emph{done right} -- by which we mean using specific insights from the optimisation and kernel communities -- stochastic gradient descent is highly effective. To that end, we introduce a particularly simple \\emph{stochastic dual descent} algorithm, explain its design in an intuitive manner and illustrate the design choices through a series of ablation studies. Further experiments demonstrate that our new method is highly competitive. In particular, our evaluations on the UCI regression tasks and on Bayesian optimisation set our approach apart from preconditioned conjugate gradients and variational Gaussian process approximations. Moreover, our method places Gaussian process regression on par with state-of-the-art graph neural networks for molecular binding affinity prediction.","url_abs":"https://arxiv.org/abs/2310.20581v2","url_pdf":"https://arxiv.org/pdf/2310.20581v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stochastic-gradient-descent-for-gaussian","repo_url":"https://github.com/cambridge-mlg/sgd-gp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"bayesian-optimisation","task_name":"Bayesian Optimisation"},{"task_slug":"gaussian-processes","task_name":"Gaussian Processes"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"gaussian-process","method_name":"Gaussian Process"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.20581","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.20581"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cambridge-mlg/sgd-gp","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":15,"unverified":4},"by_repo_kind":{"official":{"samples":19,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"256df157025464e0","entry":"add_centroid","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/kmeans.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/kmeans.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"256df157025464e0"}},{"code_sha256_prefix":"e516cb910a18d1c4","entry":"custom_cg","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/custom_cg.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/custom_cg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e516cb910a18d1c4"}},{"code_sha256_prefix":"e0776f5f9f8da068","entry":"error","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/linear_model.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/linear_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0776f5f9f8da068"}},{"code_sha256_prefix":"902854c4bdd7e801","entry":"fp_dicts_to_arr","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/fingerprint_utils.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/fingerprint_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"902854c4bdd7e801"}},{"code_sha256_prefix":"182f12ef71a39631","entry":"freeze_non_variational_params","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/SVGP.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/SVGP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"182f12ef71a39631"}},{"code_sha256_prefix":"5aff9dfc1a529f98","entry":"generate_close_pair_dict","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/knn.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/knn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5aff9dfc1a529f98"}},{"code_sha256_prefix":"c95de35c70b9ab83","entry":"get_concentrating_toy_sin_dataset","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/data.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c95de35c70b9ab83"}},{"code_sha256_prefix":"bc4d5ad47891f330","entry":"get_pruned_indices","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/knn.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/knn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc4d5ad47891f330"}},{"code_sha256_prefix":"4eee6abddb7dac5c","entry":"get_split_toy_sin_dataset","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/data.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4eee6abddb7dac5c"}},{"code_sha256_prefix":"2bda1745f6d5e206","entry":"hilbert_space_RMSE","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/eval_utils.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bda1745f6d5e206"}},{"code_sha256_prefix":"776edbdfc8352a25","entry":"i_error","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/inducing_linear_model.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/inducing_linear_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"776edbdfc8352a25"}},{"code_sha256_prefix":"8940eaaee4dd71b2","entry":"kmeanspp","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/kmeans.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/kmeans.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8940eaaee4dd71b2"}},{"code_sha256_prefix":"5eca356948077f17","entry":"pred_space_RMSE","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/eval_utils.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5eca356948077f17"}},{"code_sha256_prefix":"5e56bc4f20bb7af5","entry":"subsample","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/data.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e56bc4f20bb7af5"}},{"code_sha256_prefix":"f9ffe6a03a54afb0","entry":"vector_quantize","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/kmeans.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/kmeans.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9ffe6a03a54afb0"}},{"code_sha256_prefix":"0f413768b7c2cd5a","entry":"i_error_grad_sample","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/inducing_linear_model.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/inducing_linear_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f413768b7c2cd5a"}},{"code_sha256_prefix":"c92ac444fbedc857","entry":"i_regularizer","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/inducing_linear_model.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/inducing_linear_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c92ac444fbedc857"}},{"code_sha256_prefix":"05b0c6b467ba7da1","entry":"regularizer","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/linear_model.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/linear_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05b0c6b467ba7da1"}},{"code_sha256_prefix":"e79e6d84397801b3","entry":"sample_from_qu","repo":"cambridge-mlg/sgd-gp","repo_kind":"official","path":"scalable_gps/SVGP.py","file_url":"https://github.com/cambridge-mlg/sgd-gp/blob/HEAD/scalable_gps/SVGP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e79e6d84397801b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}