{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformer-fusion-with-optimal-transport","title":"Transformer Fusion with Optimal Transport","arxiv_id":"2310.05719","date":"2023-10-09","proceeding":null,"authors":["Moritz Imfeld","Jacopo Graldi","Marco Giordano","Thomas Hofmann","Sotiris Anagnostidis","Sidak Pal Singh"],"abstract":"Fusion is a technique for merging multiple independently-trained neural networks in order to combine their capabilities. Past attempts have been restricted to the case of fully-connected, convolutional, and residual networks. This paper presents a systematic approach for fusing two or more transformer-based networks exploiting Optimal Transport to (soft-)align the various architectural components. We flesh out an abstraction for layer alignment, that can generalize to arbitrary architectures - in principle - and we apply this to the key ingredients of Transformers such as multi-head self-attention, layer-normalization, and residual connections, and we discuss how to handle them via various ablation studies. Furthermore, our method allows the fusion of models of different sizes (heterogeneous fusion), providing a new and efficient way to compress Transformers. The proposed approach is evaluated on both image classification tasks via Vision Transformer and natural language modeling tasks using BERT. Our approach consistently outperforms vanilla fusion, and, after a surprisingly short finetuning, also outperforms the individual converged parent models. In our analysis, we uncover intriguing insights about the significant role of soft alignment in the case of Transformers. Our results showcase the potential of fusing multiple Transformers, thus compounding their expertise, in the budding paradigm of model fusion and recombination. Code is available at https://github.com/graldij/transformer-fusion.","url_abs":"https://arxiv.org/abs/2310.05719v3","url_pdf":"https://arxiv.org/pdf/2310.05719v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformer-fusion-with-optimal-transport","repo_url":"https://github.com/graldij/transformer-fusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.05719","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05719"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/graldij/transformer-fusion","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":10,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"721bb9bc2a6879cd","entry":"dict_get","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"721bb9bc2a6879cd"}},{"code_sha256_prefix":"ea4d8e1f07bacca8","entry":"ln_fusion","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea4d8e1f07bacca8"}},{"code_sha256_prefix":"55a0789f11552e01","entry":"resid_policy","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55a0789f11552e01"}},{"code_sha256_prefix":"b47405ab6d7d2f03","entry":"GroundMetric","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b47405ab6d7d2f03"}},{"code_sha256_prefix":"d5028e23c26a673c","entry":"act_attention_fusion","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5028e23c26a673c"}},{"code_sha256_prefix":"ebd6da09478807df","entry":"dict_write","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebd6da09478807df"}},{"code_sha256_prefix":"49bcbae821502d39","entry":"encoder_fusion","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"49bcbae821502d39"}},{"code_sha256_prefix":"5b6a97a40329a03f","entry":"fc_fusion","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b6a97a40329a03f"}},{"code_sha256_prefix":"268fc860ec170df8","entry":"histogram_to_str","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"268fc860ec170df8"}},{"code_sha256_prefix":"2c6503476961b9d9","entry":"matrix_stats","repo":"graldij/transformer-fusion","repo_kind":"official","path":"otfusion/otfusion_lib.py","file_url":"https://github.com/graldij/transformer-fusion/blob/HEAD/otfusion/otfusion_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c6503476961b9d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}