Papers › FuseChat: Knowledge Fusion of Chat Models

FuseChat: Knowledge Fusion of Chat Models

15 Aug 2024arXiv:2408.07990archive 2025-07-28

Fanqi Wan, Longguang Zhong, ZiYi Yang, Ruijun Chen, Xiaojun Quan

While training large language models (LLMs) from scratch can indeed lead to models with distinct capabilities and strengths, it incurs substantial costs and may lead to redundancy in competencies. Knowledge fusion aims to integrate existing LLMs of diverse architectures and capabilities into a more potent LLM through lightweight continual training, thereby reducing the need for costly LLM development. In this work, we propose a new framework for the knowledge fusion of chat LLMs through two main stages, resulting in FuseChat. Firstly, we conduct pairwise knowledge fusion on source chat LLMs of varying structures and scales to create multiple target LLMs with identical structure and size via lightweight fine-tuning. During this process, a statistics-based token alignment approach is introduced as the cornerstone for fusing LLMs with different structures. Secondly, we merge these target LLMs within the parameter space, where we propose a novel method for determining the merging coefficients based on the magnitude of parameter updates before and after fine-tuning. We implement and validate FuseChat using six prominent chat LLMs with diverse architectures and scales, including OpenChat-3.5-7B, Starling-LM-7B-alpha, NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and Qwen-1.5-Chat-72B. Experimental results on two instruction-following benchmarks, AlpacaEval 2.0 and MT-Bench, demonstrate the superiority of FuseChat-7B over baselines of various sizes. Our model is even comparable to the larger Mixtral-8x7B-Instruct and approaches GPT-3.5-Turbo-1106 on MT-Bench. Our code, model weights, and data are public at https://github.com/fanqiwan/FuseAI.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

fanqiwan/fuseai officialmentioned in papermentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Weighted-Reward Preference Optimization for Implicit Model Fusion (arXiv:2412.03187; that paper's own run record: 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 4 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 5 unverified) and Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output (arXiv:2606.10528; that paper's own run record: 31 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 3 honoured, 4 violated, 2 with no contract checked; 22 where Syntology's instrument failed) · 9 unverified); nothing checks that they implement this paper's method. report
fanqiwan/fusellm mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Knowledge Fusion of Large Language Models (arXiv:2401.10491; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified) and Weighted-Reward Preference Optimization for Implicit Model Fusion (arXiv:2412.03187; that paper's own run record: 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 4 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 5 unverified); nothing checks that they implement this paper's method. report
slit-ai/fusechat-3.0 mentioned on GitHubSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Instruction Following

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

AdamAttentionAttention DropoutBPECosine AnnealingDense ConnectionsDropoutGPT-3Layer NormalizationLinear LayerLinear Warmup With Cosine AnnealingMulti-Head AttentionResidual ConnectionSoftmaxWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections