{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fadam-adam-is-a-natural-gradient-optimizer","title":"FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information","arxiv_id":"2405.12807","date":"2024-05-21","proceeding":null,"authors":["Dongseong Hwang"],"abstract":"This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We provide an accessible and detailed analysis of the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, adaptive epsilon, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.","url_abs":"https://arxiv.org/abs/2405.12807v10","url_pdf":"https://arxiv.org/pdf/2405.12807v10.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fadam-adam-is-a-natural-gradient-optimizer","repo_url":"https://github.com/lessw2020/fadam_pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"speech-recognition","task_name":"Speech Recognition"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"natural-gradient-descent","method_name":"Natural Gradient Descent"},{"method_slug":"vq-vae","method_name":"VQ-VAE"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/speech-recognition-on-librispeech-test-clean","task":"Speech Recognition","dataset":"LibriSpeech test-clean","model":"FAdam","rank_in_archive_order":3,"of":64,"metrics":{"Word Error Rate (WER)":"1.34"},"uses_additional_data":true},{"leaderboard":"/sota/speech-recognition-on-librispeech-test-other","task":"Speech Recognition","dataset":"LibriSpeech test-other","model":"FAdam","rank_in_archive_order":2,"of":53,"metrics":{"Word Error Rate (WER)":"2.49"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.12807","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}