{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fast-inference-of-mixture-of-experts-language","title":"Fast Inference of Mixture-of-Experts Language Models with Offloading","arxiv_id":"2312.17238","date":"2023-12-28","proceeding":null,"authors":["Artyom Eliseev","Denis Mazur"],"abstract":"With the widespread adoption of Large Language Models (LLMs), many deep learning practitioners are looking for strategies of running these models more efficiently. One such strategy is to use sparse Mixture-of-Experts (MoE) - a type of model architectures where only a fraction of model layers are active for any given input. This property allows MoE-based language models to generate tokens faster than their dense counterparts, but it also increases model size due to having multiple experts. Unfortunately, this makes state-of-the-art MoE language models difficult to run without high-end GPUs. In this work, we study the problem of running large MoE language models on consumer hardware with limited accelerator memory. We build upon parameter offloading algorithms and propose a novel strategy that accelerates offloading by taking advantage of innate properties of MoE LLMs. Using this strategy, we build can run Mixtral-8x7B with mixed quantization on desktop hardware and free-tier Google Colab instances.","url_abs":"https://arxiv.org/abs/2312.17238v1","url_pdf":"https://arxiv.org/pdf/2312.17238v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fast-inference-of-mixture-of-experts-language","repo_url":"https://github.com/dvmazur/mixtral-offloading","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.17238","atlas_url":"https://app.syntology.ai/?focus=2312.17238","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.17238"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dvmazur/mixtral-offloading","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":4},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"639811fcfbb1e7a8","entry":"is_namedtuple","repo":"dvmazur/mixtral-offloading","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/dvmazur/mixtral-offloading/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"639811fcfbb1e7a8"}},{"code_sha256_prefix":"d79294cb3c49a723","entry":"nested_compare","repo":"dvmazur/mixtral-offloading","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/dvmazur/mixtral-offloading/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d79294cb3c49a723"}},{"code_sha256_prefix":"7ebe56894626731f","entry":"nested_pack","repo":"dvmazur/mixtral-offloading","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/dvmazur/mixtral-offloading/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ebe56894626731f"}},{"code_sha256_prefix":"3ccd9ec0272b8111","entry":"pack_4bit_u8_common","repo":"dvmazur/mixtral-offloading","repo_kind":"official","path":"src/packing.py","file_url":"https://github.com/dvmazur/mixtral-offloading/blob/HEAD/src/packing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ccd9ec0272b8111"}},{"code_sha256_prefix":"6445d17b9878e024","entry":"unpack_4bit_u8_common","repo":"dvmazur/mixtral-offloading","repo_kind":"official","path":"src/packing.py","file_url":"https://github.com/dvmazur/mixtral-offloading/blob/HEAD/src/packing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6445d17b9878e024"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}