{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowledge-fusion-of-large-language-models","title":"Knowledge Fusion of Large Language Models","arxiv_id":"2401.10491","date":"2024-01-19","proceeding":null,"authors":["Fanqi Wan","Xinting Huang","Deng Cai","Xiaojun Quan","Wei Bi","Shuming Shi"],"abstract":"While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures--Llama-2, MPT, and OpenLLaMA--across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at \\url{https://github.com/fanqiwan/FuseLLM}.","url_abs":"https://arxiv.org/abs/2401.10491v2","url_pdf":"https://arxiv.org/pdf/2401.10491v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowledge-fusion-of-large-language-models","repo_url":"https://github.com/fanqiwan/fusellm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"knowledge-fusion-of-large-language-models","repo_url":"https://github.com/fanqiwan/fuseai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"knowledge-fusion-of-large-language-models","repo_url":"https://github.com/songmzhang/dskd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.10491","atlas_url":"https://app.syntology.ai/?focus=2401.10491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.10491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/fanqiwan/FuseLLM","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/songmzhang/dskd","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fanqiwan/fuseai","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fanqiwan/fusellm","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"1082508e65f77331","entry":"get_logger","repo":"fanqiwan/FuseLLM","repo_kind":"official","path":"FuseLLM/src/utils/others.py","file_url":"https://github.com/fanqiwan/FuseLLM/blob/HEAD/FuseLLM/src/utils/others.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1082508e65f77331"}},{"code_sha256_prefix":"fe84e68815f8de20","entry":"DualSpaceKDWithCMA","repo":"songmzhang/dskd","repo_kind":"listed","path":"code/criterions/dual_space_kd_with_cross_model_attention.py","file_url":"https://github.com/songmzhang/dskd/blob/HEAD/code/criterions/dual_space_kd_with_cross_model_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe84e68815f8de20"}},{"code_sha256_prefix":"df472be23644090f","entry":"get_base_model","repo":"fanqiwan/FuseLLM","repo_kind":"official","path":"FuseLLM/src/utils/others.py","file_url":"https://github.com/fanqiwan/FuseLLM/blob/HEAD/FuseLLM/src/utils/others.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df472be23644090f"}},{"code_sha256_prefix":"503f70e78560aa9e","entry":"get_tokenizer","repo":"fanqiwan/FuseLLM","repo_kind":"official","path":"FuseLLM/src/utils/others.py","file_url":"https://github.com/fanqiwan/FuseLLM/blob/HEAD/FuseLLM/src/utils/others.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"503f70e78560aa9e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}