{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mixtral-of-experts","title":"Mixtral of Experts","arxiv_id":"2401.04088","date":"2024-01-08","proceeding":null,"authors":["Albert Q. Jiang","Alexandre Sablayrolles","Antoine Roux","Arthur Mensch","Blanche Savary","Chris Bamford","Devendra Singh Chaplot","Diego de Las Casas","Emma Bou Hanna","Florian Bressand","Gianna Lengyel","Guillaume Bour","Guillaume Lample","Lélio Renard Lavaud","Lucile Saulnier","Marie-Anne Lachaux","Pierre Stock","Sandeep Subramanian","Sophia Yang","Szymon Antoniak","Teven Le Scao","Théophile Gervet","Thibaut Lavril","Thomas Wang","Timothée Lacroix","William El Sayed"],"abstract":"We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.","url_abs":"https://arxiv.org/abs/2401.04088v1","url_pdf":"https://arxiv.org/pdf/2401.04088v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mixtral-of-experts","repo_url":"https://github.com/consequentai/fneval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"mixtral-of-experts","repo_url":"https://github.com/hit-scir/chinese-mixtral-8x7b","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"mixtral-of-experts","repo_url":"https://github.com/jingyaogong/minimind","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"mixtral-of-experts","repo_url":"https://github.com/kamanphoebe/look-into-moes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"mixtral-of-experts","repo_url":"https://github.com/ymcui/chinese-mixtral","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"mixtral-of-experts","repo_url":"https://github.com/pwc-1/Paper-9/tree/main/2/mixtral","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"math-word-problem-solving","task_name":"Math Word Problem Solving"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"Mixtral 8x7B (3-shot)","rank_in_archive_order":48,"of":99,"metrics":{"Accuracy":"60.7"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"Mixtral 8x7B (0-shot)","rank_in_archive_order":12,"of":47,"metrics":{"Accuracy":"83.1"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"Mistral 7B (0-shot)","rank_in_archive_order":14,"of":47,"metrics":{"Accuracy":"80.5"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"Mixtral 8x7B (0-shot)","rank_in_archive_order":20,"of":77,"metrics":{"Accuracy":"77.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"Mistral 7B (0-shot)","rank_in_archive_order":30,"of":77,"metrics":{"Accuracy":"74.2"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"Mixtral 8x7B (maj@4)","rank_in_archive_order":93,"of":135,"metrics":{"Accuracy":"28.4"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"Mistral 7B (maj@4)","rank_in_archive_order":113,"of":135,"metrics":{"Accuracy":"12.7","Parameters (Billions)":"7"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-mmlu","task":"Multi-task Language Understanding","dataset":"MML","model":"Mixtral 8x7B (5-shot)","rank_in_archive_order":11,"of":44,"metrics":{"Average (%)":"70.6"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-mmlu","task":"Multi-task Language Understanding","dataset":"MML","model":"Mistral 7B (5-shot)","rank_in_archive_order":17,"of":44,"metrics":{"Average (%)":"62.5"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"Mixtral 8x7B (0-shot)","rank_in_archive_order":13,"of":67,"metrics":{"Accuracy":"83.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"Mistral 7B (0-shot)","rank_in_archive_order":22,"of":67,"metrics":{"Accuracy":"82.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.04088","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.04088"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/consequentai/fneval","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-9/tree/main/2/mixtral","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingyaogong/minimind","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ymcui/chinese-mixtral","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kamanphoebe/look-into-moes","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hit-scir/chinese-mixtral-8x7b","reach":null}],"summary":{"ran":5},"by_repo_kind":{"listed":{"samples":5,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"549cdd919dd3f999","entry":"FeedForward","repo":"jingyaogong/minimind","repo_kind":"listed","path":"model/model_minimind.py","file_url":"https://github.com/jingyaogong/minimind/blob/HEAD/model/model_minimind.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"549cdd919dd3f999"}},{"code_sha256_prefix":"3678349ac16c8826","entry":"FeedForward","repo":"kamanphoebe/look-into-moes","repo_kind":"listed","path":"mixtral_instruct/modeling_mixtral_instruct.py","file_url":"https://github.com/kamanphoebe/look-into-moes/blob/HEAD/mixtral_instruct/modeling_mixtral_instruct.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3678349ac16c8826"}},{"code_sha256_prefix":"ca104baa2bcf757c","entry":"MOEFeedForward","repo":"jingyaogong/minimind","repo_kind":"listed","path":"model/model_minimind.py","file_url":"https://github.com/jingyaogong/minimind/blob/HEAD/model/model_minimind.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca104baa2bcf757c"}},{"code_sha256_prefix":"b2a516613396ed08","entry":"MiniMindConfig","repo":"jingyaogong/minimind","repo_kind":"listed","path":"model/model_minimind.py","file_url":"https://github.com/jingyaogong/minimind/blob/HEAD/model/model_minimind.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b2a516613396ed08"}},{"code_sha256_prefix":"253c6ed84c1f2c72","entry":"MoE","repo":"kamanphoebe/look-into-moes","repo_kind":"listed","path":"mixtral_instruct/modeling_mixtral_instruct.py","file_url":"https://github.com/kamanphoebe/look-into-moes/blob/HEAD/mixtral_instruct/modeling_mixtral_instruct.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"253c6ed84c1f2c72"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}