{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-neural-networks-as-gaussian-processes","title":"Deep Neural Networks as Gaussian Processes","arxiv_id":"1711.00165","date":"2017-11-01","proceeding":"ICLR 2018 1","authors":["Jaehoon Lee","Yasaman Bahri","Roman Novak","Samuel S. Schoenholz","Jeffrey Pennington","Jascha Sohl-Dickstein"],"abstract":"It has long been known that a single-layer fully-connected neural network\nwith an i.i.d. prior over its parameters is equivalent to a Gaussian process\n(GP), in the limit of infinite network width. This correspondence enables exact\nBayesian inference for infinite width neural networks on regression tasks by\nmeans of evaluating the corresponding GP. Recently, kernel functions which\nmimic multi-layer random neural networks have been developed, but only outside\nof a Bayesian framework. As such, previous work has not identified that these\nkernels can be used as covariance functions for GPs and allow fully Bayesian\nprediction with a deep neural network.\n  In this work, we derive the exact equivalence between infinitely wide deep\nnetworks and GPs. We further develop a computationally efficient pipeline to\ncompute the covariance function for these GPs. We then use the resulting GPs to\nperform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10.\nWe observe that trained neural network accuracy approaches that of the\ncorresponding GP with increasing layer width, and that the GP uncertainty is\nstrongly correlated with trained network prediction error. We further find that\ntest performance increases as finite-width trained networks are made wider and\nmore similar to a GP, and thus that GP predictions typically outperform those\nof finite-width networks. Finally we connect the performance of these GPs to\nthe recent theory of signal propagation in random neural networks.","url_abs":"http://arxiv.org/abs/1711.00165v3","url_pdf":"http://arxiv.org/pdf/1711.00165v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-neural-networks-as-gaussian-processes","repo_url":"https://github.com/brain-research/nngp","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"deep-neural-networks-as-gaussian-processes","repo_url":"https://github.com/chrhenning/uncertainty_based_ood","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"deep-neural-networks-as-gaussian-processes","repo_url":"https://github.com/google/wide_bnn_sampling","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"deep-neural-networks-as-gaussian-processes","repo_url":"https://github.com/jxVmnLgedVwv6mNcGCBy/NNspectra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"deep-neural-networks-as-gaussian-processes","repo_url":"https://github.com/kmatt10/DNNGP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"deep-neural-networks-as-gaussian-processes","repo_url":"https://github.com/thegregyang/NNspectra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"deep-neural-networks-as-gaussian-processes","repo_url":"https://github.com/yumaloop/dnn_hsic","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"gaussian-processes","task_name":"Gaussian Processes"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.00165","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}