{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/map-low-compute-model-merging-with-amortized","title":"MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation","arxiv_id":"2406.07529","date":"2024-06-11","proceeding":null,"authors":["Lu Li","Tianyu Zhang","Zhiqi Bu","Suyuchen Wang","Huan He","Jie Fu","Yonghui Wu","Jiang Bian","Yong Chen","Yoshua Bengio"],"abstract":"Model merging has emerged as an effective approach to combine multiple single-task models into a multitask model. This process typically involves computing a weighted average of the model parameters without any additional training. Existing model-merging methods focus on enhancing average task accuracy. However, interference and conflicts between the objectives of different tasks can lead to trade-offs during the merging process. In real-world applications, a set of solutions with various trade-offs can be more informative, helping practitioners make decisions based on diverse preferences. In this paper, we introduce a novel and low-compute algorithm, Model Merging with Amortized Pareto Front (MAP). MAP efficiently identifies a Pareto set of scaling coefficients for merging multiple models, reflecting the trade-offs involved. It amortizes the substantial computational cost of evaluations needed to estimate the Pareto front by using quadratic approximation surrogate models derived from a pre-selected set of scaling coefficients. Experimental results on vision and natural language processing tasks demonstrate that MAP can accurately identify the Pareto front, providing practitioners with flexible solutions to balance competing task objectives. We also introduce Bayesian MAP for scenarios with a relatively low number of tasks and Nested MAP for situations with a high number of tasks, further reducing the computational cost of evaluation.","url_abs":"https://arxiv.org/abs/2406.07529v4","url_pdf":"https://arxiv.org/pdf/2406.07529v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"map-low-compute-model-merging-with-amortized","repo_url":"https://github.com/luli-git/MAP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.07529","atlas_url":"https://app.syntology.ai/?focus=2406.07529","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07529"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luli-git/MAP","reach":{"status":"ok"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"56ff2891d1fdc557","entry":"create_objective_function","repo":"luli-git/MAP","repo_kind":"official","path":"MAP.py","file_url":"https://github.com/luli-git/MAP/blob/HEAD/MAP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56ff2891d1fdc557"}},{"code_sha256_prefix":"b4926d98ff6bbaa3","entry":"load_json_data","repo":"luli-git/MAP","repo_kind":"official","path":"utils.py","file_url":"https://github.com/luli-git/MAP/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4926d98ff6bbaa3"}},{"code_sha256_prefix":"15a7179959523297","entry":"load_pf","repo":"luli-git/MAP","repo_kind":"official","path":"utils.py","file_url":"https://github.com/luli-git/MAP/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15a7179959523297"}},{"code_sha256_prefix":"f72ecc9e2a1c54a8","entry":"load_pkl_data","repo":"luli-git/MAP","repo_kind":"official","path":"utils.py","file_url":"https://github.com/luli-git/MAP/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f72ecc9e2a1c54a8"}},{"code_sha256_prefix":"3e8f86d8ffb36ad0","entry":"sigmoid","repo":"luli-git/MAP","repo_kind":"official","path":"MAP.py","file_url":"https://github.com/luli-git/MAP/blob/HEAD/MAP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e8f86d8ffb36ad0"}},{"code_sha256_prefix":"e3dd7a56366f7f08","entry":"softplus","repo":"luli-git/MAP","repo_kind":"official","path":"MAP.py","file_url":"https://github.com/luli-git/MAP/blob/HEAD/MAP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3dd7a56366f7f08"}},{"code_sha256_prefix":"15267d00983c4051","entry":"tv_merge_to_pretrain","repo":"luli-git/MAP","repo_kind":"official","path":"merge.py","file_url":"https://github.com/luli-git/MAP/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15267d00983c4051"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}