{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/why-transformers-need-adam-a-hessian","title":"Why Transformers Need Adam: A Hessian Perspective","arxiv_id":"2402.16788","date":"2024-02-26","proceeding":null,"authors":["Yushun Zhang","Congliang Chen","Tian Ding","Ziniu Li","Ruoyu Sun","Zhi-Quan Luo"],"abstract":"SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear. In this work, we provide an explanation through the lens of Hessian: (i) Transformers are \"heterogeneous\": the Hessian spectrum across parameter blocks vary dramatically, a phenomenon we call \"block heterogeneity\"; (ii) Heterogeneity hampers SGD: SGD performs worse than Adam on problems with block heterogeneity. To validate (i) and (ii), we check various Transformers, CNNs, MLPs, and quadratic problems, and find that SGD can perform on par with Adam on problems without block heterogeneity, but performs worse than Adam when the heterogeneity exists. Our initial theoretical analysis indicates that SGD performs worse because it applies one single learning rate to all blocks, which cannot handle the heterogeneity among blocks. This limitation could be ameliorated if we use coordinate-wise learning rates, as designed in Adam.","url_abs":"https://arxiv.org/abs/2402.16788v4","url_pdf":"https://arxiv.org/pdf/2402.16788v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"why-transformers-need-adam-a-hessian","repo_url":"https://github.com/zyushun/hessian-spectrum","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"why-transformers-need-adam-a-hessian","repo_url":"https://github.com/zyushun/adam-mini","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16788","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16788"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zyushun/adam-mini","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zyushun/hessian-spectrum","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"d950bea4c9753cd4","entry":"accuracy","repo":"zyushun/hessian-spectrum","repo_kind":"official","path":"vision_models/helper.py","file_url":"https://github.com/zyushun/hessian-spectrum/blob/HEAD/vision_models/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d950bea4c9753cd4"}},{"code_sha256_prefix":"fafefcd33288b3e3","entry":"data_loader","repo":"zyushun/hessian-spectrum","repo_kind":"official","path":"vision_models/data_loader.py","file_url":"https://github.com/zyushun/hessian-spectrum/blob/HEAD/vision_models/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fafefcd33288b3e3"}},{"code_sha256_prefix":"aaad22245780ab53","entry":"get_lr","repo":"zyushun/hessian-spectrum","repo_kind":"official","path":"language_models/train_gpt2.py","file_url":"https://github.com/zyushun/hessian-spectrum/blob/HEAD/language_models/train_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aaad22245780ab53"}},{"code_sha256_prefix":"50afa2863ffd9fde","entry":"plot_overlap","repo":"zyushun/hessian-spectrum","repo_kind":"official","path":"language_models/logger.py","file_url":"https://github.com/zyushun/hessian-spectrum/blob/HEAD/language_models/logger.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50afa2863ffd9fde"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}