{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mkor-momentum-enabled-kronecker-factor-based","title":"MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates","arxiv_id":"2306.01685","date":"2023-06-02","proceeding":"NeurIPS 2023 11","authors":["Mohammad Mozaffari","Sikan Li","Zhao Zhang","Maryam Mehri Dehnavi"],"abstract":"This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 updates, called MKOR, that improves the training time and convergence properties of deep neural networks (DNNs). Second-order techniques, while enjoying higher convergence rates vs first-order counterparts, have cubic complexity with respect to either the model size and/or the training batch size. Hence they exhibit poor scalability and performance in transformer models, e.g. large language models (LLMs), because the batch sizes in these models scale by the attention mechanism sequence length, leading to large model size and batch sizes. MKOR's complexity is quadratic with respect to the model size, alleviating the computation bottlenecks in second-order methods. Because of their high computation complexity, state-of-the-art implementations of second-order methods can only afford to update the second order information infrequently, and thus do not fully exploit the promise of better convergence from these updates. By reducing the communication complexity of the second-order updates as well as achieving a linear communication complexity, MKOR increases the frequency of second order updates. We also propose a hybrid version of MKOR (called MKOR-H) that mid-training falls backs to a first order optimizer if the second order updates no longer accelerate convergence. Our experiments show that MKOR outperforms state -of-the-art first order methods, e.g. the LAMB optimizer, and best implementations of second-order methods, i.e. KAISA/KFAC, up to 2.57x and 1.85x respectively on BERT-Large-Uncased on 64 GPUs.","url_abs":"https://arxiv.org/abs/2306.01685v2","url_pdf":"https://arxiv.org/pdf/2306.01685v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mkor-momentum-enabled-kronecker-factor-based","repo_url":"https://github.com/mohammad-mozaffari/mkor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"second-order-methods","task_name":"Second-order methods"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"lamb","method_name":"LAMB"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.01685","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.01685"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mohammad-mozaffari/mkor","reach":null}],"summary":{"ran":6,"ran_fixture":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"8e5181cec3cb7d0c","entry":"ComputeCovA","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e5181cec3cb7d0c"}},{"code_sha256_prefix":"d2f1ea938929fecd","entry":"ComputeCovG","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2f1ea938929fecd"}},{"code_sha256_prefix":"fd12dec47efc6010","entry":"ComputeG","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd12dec47efc6010"}},{"code_sha256_prefix":"834785f37b636745","entry":"ComputeI","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"834785f37b636745"}},{"code_sha256_prefix":"1e2ee6e87930866f","entry":"EmptyBackend","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e2ee6e87930866f"}},{"code_sha256_prefix":"61eef187833440e6","entry":"Timer","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61eef187833440e6"}},{"code_sha256_prefix":"9a1d18d135e2e134","entry":"_extract_patches","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a1d18d135e2e134"}},{"code_sha256_prefix":"a2a278e05b22e5e5","entry":"sm_inverse","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2a278e05b22e5e5"}},{"code_sha256_prefix":"c37ec5fa3e9fd20c","entry":"try_contiguous","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c37ec5fa3e9fd20c"}},{"code_sha256_prefix":"c5fca7ca09dcf5c0","entry":"MKOR","repo":"mohammad-mozaffari/mkor","repo_kind":"official","path":"mkor/mkor.py","file_url":"https://github.com/mohammad-mozaffari/mkor/blob/HEAD/mkor/mkor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5fca7ca09dcf5c0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}