{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/emu3-next-token-prediction-is-all-you-need","title":"Emu3: Next-Token Prediction is All You Need","arxiv_id":"2409.18869","date":"2024-09-27","proceeding":null,"authors":["Xinlong Wang","Xiaosong Zhang","Zhengxiong Luo","Quan Sun","Yufeng Cui","Jinsheng Wang","Fan Zhang","Yueze Wang","Zhen Li","Qiying Yu","Yingli Zhao","Yulong Ao","Xuebin Min","Tao Li","Boya Wu","Bo Zhao","BoWen Zhang","Liangdong Wang","Guang Liu","Zheqi He","Xi Yang","Jingjing Liu","Yonghua Lin","Tiejun Huang","Zhongyuan Wang"],"abstract":"While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.","url_abs":"https://arxiv.org/abs/2409.18869v1","url_pdf":"https://arxiv.org/pdf/2409.18869v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"emu3-next-token-prediction-is-all-you-need","repo_url":"https://github.com/baaivision/emu3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"emu3-next-token-prediction-is-all-you-need","repo_url":"https://github.com/flagopen/flagscale","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/on-wise","task":"","dataset":"WISE","model":"Emu3","rank_in_archive_order":8,"of":11,"metrics":{"Biology":"0.41","Chemistry":"0.27","Cultural":"0.34","Overall":"0.39","Physics":"0.45","Space":"0.48","Time":"0.45"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-wise","task":"Image Generation","dataset":"WISE","model":"Emu3-gen","rank_in_archive_order":11,"of":14,"metrics":{"Biology":"0.41","Chemistry":"0.27","Cultural":"0.34","Overall":"0.39","Physics":"0.45","Space":"0.48","Time":"0.45"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-geneval","task":"Text-to-Image Generation","dataset":"GenEval","model":"Emu3","rank_in_archive_order":15,"of":20,"metrics":{"Overall":"0.66"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-t2i-compbench","task":"Text-to-Image Generation","dataset":"T2I-CompBench","model":"Emu3","rank_in_archive_order":1,"of":2,"metrics":{"Color":"0.7913","Shape":"0.5846","Texture":"0.7422"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"Emu3","rank_in_archive_order":138,"of":231,"metrics":{"GPT-4 score":"37.2"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2409.18869","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.18869"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flagopen/flagscale","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baaivision/emu3","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"listed":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"baaivision/emu3","repo_kind":"listed","path":"emu3/mllm/modeling_emu3.py","file_url":"https://github.com/baaivision/emu3/blob/HEAD/emu3/mllm/modeling_emu3.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"baaivision/emu3","repo_kind":"listed","path":"emu3/mllm/modeling_emu3.py","file_url":"https://github.com/baaivision/emu3/blob/HEAD/emu3/mllm/modeling_emu3.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"baaivision/emu3","repo_kind":"listed","path":"emu3/mllm/modeling_emu3.py","file_url":"https://github.com/baaivision/emu3/blob/HEAD/emu3/mllm/modeling_emu3.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"658cb59bfa54f294","entry":"image2str","repo":"baaivision/emu3","repo_kind":"listed","path":"gradio_demo.py","file_url":"https://github.com/baaivision/emu3/blob/HEAD/gradio_demo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"658cb59bfa54f294"}},{"code_sha256_prefix":"5ea70bedcc69c330","entry":"smart_resize","repo":"baaivision/emu3","repo_kind":"listed","path":"emu3/tokenizer/image_processing_emu3visionvq.py","file_url":"https://github.com/baaivision/emu3/blob/HEAD/emu3/tokenizer/image_processing_emu3visionvq.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ea70bedcc69c330"}},{"code_sha256_prefix":"8d1dbae469f6cafb","entry":"smart_resize","repo":"baaivision/emu3","repo_kind":"listed","path":"emu3/train/prepare_data.py","file_url":"https://github.com/baaivision/emu3/blob/HEAD/emu3/train/prepare_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d1dbae469f6cafb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}