{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/magmax-leveraging-model-merging-for-seamless","title":"MagMax: Leveraging Model Merging for Seamless Continual Learning","arxiv_id":"2407.06322","date":"2024-07-08","proceeding":null,"authors":["Daniel Marczak","Bartłomiej Twardowski","Tomasz Trzciński","Sebastian Cygert"],"abstract":"This paper introduces a continual learning approach named MagMax, which utilizes model merging to enable large pre-trained models to continuously learn from new data without forgetting previously acquired knowledge. Distinct from traditional continual learning methods that aim to reduce forgetting during task training, MagMax combines sequential fine-tuning with a maximum magnitude weight selection for effective knowledge integration across tasks. Our initial contribution is an extensive examination of model merging techniques, revealing that simple approaches like weight averaging and random weight selection surprisingly hold up well in various continual learning contexts. More importantly, we present MagMax, a novel model-merging strategy that enables continual learning of large pre-trained models for successive tasks. Our thorough evaluation demonstrates the superiority of MagMax in various scenarios, including class- and domain-incremental learning settings. The code is available at this URL: https://github.com/danielm1405/magmax.","url_abs":"https://arxiv.org/abs/2407.06322v2","url_pdf":"https://arxiv.org/pdf/2407.06322v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"magmax-leveraging-model-merging-for-seamless","repo_url":"https://github.com/danielm1405/magmax","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"incremental-learning","task_name":"Incremental Learning"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.06322","atlas_url":"https://app.syntology.ai/?focus=2407.06322","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.06322"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/danielm1405/magmax","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"d123ccea453ac5db","entry":"calc_ewc_loss","repo":"danielm1405/magmax","repo_kind":"official","path":"ewc.py","file_url":"https://github.com/danielm1405/magmax/blob/HEAD/ewc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d123ccea453ac5db"}},{"code_sha256_prefix":"87c06b895809b585","entry":"cross_entropy","repo":"danielm1405/magmax","repo_kind":"official","path":"lwf.py","file_url":"https://github.com/danielm1405/magmax/blob/HEAD/lwf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87c06b895809b585"}},{"code_sha256_prefix":"02a07d36ea312362","entry":"compute_fisher_matrix_diag","repo":"danielm1405/magmax","repo_kind":"official","path":"ewc.py","file_url":"https://github.com/danielm1405/magmax/blob/HEAD/ewc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02a07d36ea312362"}},{"code_sha256_prefix":"1a7d014f38a3f7db","entry":"compute_fisher_matrix_diag","repo":"danielm1405/magmax","repo_kind":"official","path":"ewc_domain_splitted.py","file_url":"https://github.com/danielm1405/magmax/blob/HEAD/ewc_domain_splitted.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a7d014f38a3f7db"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}