{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adafisher-adaptive-second-order-optimization","title":"AdaFisher: Adaptive Second Order Optimization via Fisher Information","arxiv_id":"2405.16397","date":"2024-05-26","proceeding":null,"authors":["Damien Martins Gomes","Yanlei Zhang","Eugene Belilovsky","Guy Wolf","Mahdi S. Hosseini"],"abstract":"First-order optimization methods are currently the mainstream in training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by employing the diagonal matrix preconditioning of the stochastic gradient during the training. Despite their widespread, second-order optimization algorithms exhibit superior convergence properties compared to their first-order counterparts e.g. Adam and SGD. However, their practicality in training DNNs is still limited due to increased per-iteration computations compared to the first-order methods. We present \\emph{AdaFisher}--an adaptive second-order optimizer that leverages a \\emph{diagonal block-Kronecker} approximation of the Fisher information matrix for adaptive gradient preconditioning. AdaFisher aims to bridge the gap between enhanced \\emph{convergence/generalization} capabilities and computational efficiency in second-order optimization framework for training DNNs. Despite the slow pace of second-order optimizers, we showcase that AdaFisher can be reliably adopted for image classification, language modeling and stands out for its stability and robustness in hyper-parameter tuning. We demonstrate that AdaFisher \\textbf{outperforms the SOTA optimizers} in terms of both accuracy and convergence speed. Code is available from https://github.com/AtlasAnalyticsLab/AdaFisher.","url_abs":"https://arxiv.org/abs/2405.16397v3","url_pdf":"https://arxiv.org/pdf/2405.16397v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adafisher-adaptive-second-order-optimization","repo_url":"https://github.com/AtlasAnalyticsLab/AdaFisher","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"adafisher","method_name":"AdaFisher"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[{"slug":"adafisher","name":"AdaFisher","full_name":"Adaptive Second Order Optimization via Fisher Information"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.16397","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16397"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/AtlasAnalyticsLab/AdaFisher","reach":null}],"summary":{"ran_draft_wrong":4,"ran":2,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"b2678c5c61eb8deb","entry":"smart_detect_inf","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b2678c5c61eb8deb"}},{"code_sha256_prefix":"4f43efbc35c93ca0","entry":"AdaFisher","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4f43efbc35c93ca0"}},{"code_sha256_prefix":"a98240a56c6339b6","entry":"Compute_S_D","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a98240a56c6339b6"}},{"code_sha256_prefix":"a95fe0f88dd753ff","entry":"MinMaxNormalization","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a95fe0f88dd753ff"}},{"code_sha256_prefix":"4eac4b80c07a7103","entry":"MinMaxNormalization","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4eac4b80c07a7103"}},{"code_sha256_prefix":"227d433fcf298d4e","entry":"_extract_patches","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"227d433fcf298d4e"}},{"code_sha256_prefix":"8c8c65f2ad1706c4","entry":"smart_detect_inf","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8c8c65f2ad1706c4"}},{"code_sha256_prefix":"1711986ac8070f59","entry":"AdaFisherBackBone","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1711986ac8070f59"}},{"code_sha256_prefix":"0803247e64ad1d52","entry":"Compute_H_D","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0803247e64ad1d52"}},{"code_sha256_prefix":"1b03719b988e4b7f","entry":"update_running_avg","repo":"AtlasAnalyticsLab/AdaFisher","repo_kind":"official","path":"optimizers/AdaFisher.py","file_url":"https://github.com/AtlasAnalyticsLab/AdaFisher/blob/HEAD/optimizers/AdaFisher.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1b03719b988e4b7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}