{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pack-of-llms-model-fusion-at-test-time-via","title":"Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization","arxiv_id":"2404.11531","date":"2024-04-17","proceeding":null,"authors":["Costas Mavromatis","Petros Karypis","George Karypis"],"abstract":"Fusing knowledge from multiple Large Language Models (LLMs) can combine their diverse strengths to achieve improved performance on a given task. However, current fusion approaches either rely on learning-based fusers that do not generalize to new LLMs, or do not take into account how well each LLM understands the input. In this work, we study LLM fusion at test-time, which enables leveraging knowledge from arbitrary user-specified LLMs during inference. We introduce Pack of LLMs (PackLLM), an effective method for test-time fusion that leverages each LLM's expertise, given an input prompt. PackLLM performs model fusion by solving an optimization problem for determining each LLM's importance, so that perplexity over the input prompt is minimized. First, our simple PackLLM-sim variant validates that perplexity is a good indicator for measuring each LLM's expertise. Second, our PackLLM-opt variant approximately solves the perplexity minimization problem via a greedy algorithm. The derived importance weights are used to combine the LLMs during inference. We conduct experiments with over 100 total LLMs on a diverse set of tasks. Experimental results show that (i) perplexity is a reliable measure for LLM fusion, (ii) PackLLM outperforms test-time fusion baselines by 1.89% accuracy points, and (iii) PackLLM can leverage new LLMs to improve performance over learning-based fusion approaches by 3.92-11.94% accuracy points.","url_abs":"https://arxiv.org/abs/2404.11531v1","url_pdf":"https://arxiv.org/pdf/2404.11531v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pack-of-llms-model-fusion-at-test-time-via","repo_url":"https://github.com/cmavro/packllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.11531","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.11531"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cmavro/packllm","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"82fab1f6e6bddec2","entry":"format_dataset","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/get_task.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/get_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82fab1f6e6bddec2"}},{"code_sha256_prefix":"77f69af078e09830","entry":"mean_pooling","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/utils.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77f69af078e09830"}},{"code_sha256_prefix":"faa9fbf7cc4e66c1","entry":"prepro_sentence_pair","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/icl_data.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/icl_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"faa9fbf7cc4e66c1"}},{"code_sha256_prefix":"1f230cbd5842e2ab","entry":"prepro_sentence_pair_single","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/icl_data.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/icl_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f230cbd5842e2ab"}},{"code_sha256_prefix":"99a5947cc60a3cb3","entry":"process_mmlu_examples","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/get_task.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/get_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99a5947cc60a3cb3"}},{"code_sha256_prefix":"dc07ce32fc4a428a","entry":"process_openbookqa_examples","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/get_task.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/get_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc07ce32fc4a428a"}},{"code_sha256_prefix":"c0016ed0e6b96ff4","entry":"calculate_sentence_transformer_embedding","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/utils.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0016ed0e6b96ff4"}},{"code_sha256_prefix":"ce4a54944b0f471c","entry":"calculate_sentence_transformer_embedding_m","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/utils.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce4a54944b0f471c"}},{"code_sha256_prefix":"ddf793183ffe3659","entry":"get_instance_length","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/prompt_retrieval.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/prompt_retrieval.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddf793183ffe3659"}},{"code_sha256_prefix":"cbf1d84d85c303cc","entry":"setup_fp16","repo":"cmavro/packllm","repo_kind":"official","path":"downstream_tasks/icl_model.py","file_url":"https://github.com/cmavro/packllm/blob/HEAD/downstream_tasks/icl_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbf1d84d85c303cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}