{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fluctuation-dissipation-relations-for","title":"Fluctuation-dissipation relations for stochastic gradient descent","arxiv_id":"1810.00004","date":"2018-09-28","proceeding":"ICLR 2019 5","authors":["Sho Yaida"],"abstract":"The notion of the stationary equilibrium ensemble has played a central role\nin statistical mechanics. In machine learning as well, training serves as\ngeneralized equilibration that drives the probability distribution of model\nparameters toward stationarity. Here, we derive stationary\nfluctuation-dissipation relations that link measurable quantities and\nhyperparameters in the stochastic gradient descent algorithm. These relations\nhold exactly for any stationary state and can in particular be used to\nadaptively set training schedule. We can further use the relations to\nefficiently extract information pertaining to a loss-function landscape such as\nthe magnitudes of its Hessian and anharmonicity. Our claims are empirically\nverified.","url_abs":"http://arxiv.org/abs/1810.00004v2","url_pdf":"http://arxiv.org/pdf/1810.00004v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fluctuation-dissipation-relations-for","repo_url":"https://github.com/cybertronai/pytorch-fd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"fluctuation-dissipation-relations-for","repo_url":"https://github.com/facebookresearch/FDR_scheduler","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1810.00004","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}