{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/laplacian-smoothing-gradient-descent","title":"Laplacian Smoothing Gradient Descent","arxiv_id":"1806.06317","date":"2018-06-17","proceeding":null,"authors":["Stanley Osher","Bao Wang","Penghang Yin","Xiyang Luo","Farzin Barekat","Minh Pham","Alex Lin"],"abstract":"We propose a class of very simple modifications of gradient descent and\nstochastic gradient descent. We show that when applied to a large variety of\nmachine learning problems, ranging from logistic regression to deep neural\nnets, the proposed surrogates can dramatically reduce the variance, allow to\ntake a larger step size, and improve the generalization accuracy. The methods\nonly involve multiplying the usual (stochastic) gradient by the inverse of a\npositive definitive matrix (which can be computed efficiently by FFT) with a\nlow condition number coming from a one-dimensional discrete Laplacian or its\nhigh order generalizations. It also preserves the mean and increases the\nsmallest component and decreases the largest component. The theory of\nHamilton-Jacobi partial differential equations demonstrates that the implicit\nversion of the new algorithm is almost the same as doing gradient descent on a\nnew function which (i) has the same global minima as the original function and\n(ii) is ``more convex\". Moreover, we show that optimization algorithms with\nthese surrogates converge uniformly in the discrete Sobolev $H_\\sigma^p$ sense\nand reduce the optimality gap for convex optimization problems. The code is\navailable at:\n\\url{https://github.com/BaoWangMath/LaplacianSmoothing-GradientDescent}","url_abs":"http://arxiv.org/abs/1806.06317v5","url_pdf":"http://arxiv.org/pdf/1806.06317v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"laplacian-smoothing-gradient-descent","repo_url":"https://github.com/BaoWangMath/LaplacianSmoothing-GradientDescent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"logistic-regression","method_name":"Logistic Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}