{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generative-pretraining-in-multimodality","title":"Emu: Generative Pretraining in Multimodality","arxiv_id":"2307.05222","date":"2023-07-11","proceeding":null,"authors":["Quan Sun","Qiying Yu","Yufeng Cui","Fan Zhang","Xiaosong Zhang","Yueze Wang","Hongcheng Gao","Jingjing Liu","Tiejun Huang","Xinlong Wang"],"abstract":"We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the exploration of diverse pretraining data sources at scale, such as videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. Emu can serve as a generalist multimodal interface for both image-to-text and text-to-image tasks, and supports in-context image and text generation. Across a broad range of zero-shot/few-shot tasks including image captioning, visual question answering, video question answering and text-to-image generation, Emu demonstrates superb performance compared to state-of-the-art large multimodal models. Extended capabilities such as multimodal assistants via instruction tuning are also demonstrated with impressive performance.","url_abs":"https://arxiv.org/abs/2307.05222v2","url_pdf":"https://arxiv.org/pdf/2307.05222v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generative-pretraining-in-multimodality","repo_url":"https://github.com/baaivision/emu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"generative-pretraining-in-multimodality","repo_url":"https://github.com/doc-doc/NExT-OE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"temporal-casual-qa","task_name":"Temporal/Casual QA"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/temporal-casual-qa-on-next-qa","task":"Temporal/Casual QA","dataset":"NExT-QA","model":"Emu(0-shot)","rank_in_archive_order":8,"of":8,"metrics":{"WUPS":"23.4"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"Emu-14B","rank_in_archive_order":146,"of":231,"metrics":{"GPT-4 score":"36.3±0.3","Params":"14B"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet-w-o","task":"Visual Question Answering","dataset":"MM-Vet (w/o External Tools)","model":"Emu-14B","rank_in_archive_order":1,"of":1,"metrics":{"GPT-4 score":"36.3±0.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-1","task":"Visual Question Answering","dataset":"VQA v2","model":"Emu-I *","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"57.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vizwiz-1","task":"Visual Question Answering","dataset":"VizWiz","model":"Emu-I *","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"38.1"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on-core-mm","task":"Visual Question Answering (VQA)","dataset":"InfiMM-Eval","model":"Emu","rank_in_archive_order":7,"of":14,"metrics":{"Abductive":"36.57","Analogical":"18.19","Deductive":"28.9","Overall score":"28.24","Params":"14B"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.05222","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.05222"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/doc-doc/NExT-OE","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baaivision/emu","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d54f53c9fd4f3cf","entry":"SelfTransformer","repo":"doc-doc/NExT-OE","repo_kind":"listed","path":"networks/q_v_transformer.py","file_url":"https://github.com/doc-doc/NExT-OE/blob/HEAD/networks/q_v_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d54f53c9fd4f3cf"}},{"code_sha256_prefix":"7e88811227b46859","entry":"SelfTransformerLayer","repo":"doc-doc/NExT-OE","repo_kind":"listed","path":"networks/q_v_transformer.py","file_url":"https://github.com/doc-doc/NExT-OE/blob/HEAD/networks/q_v_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e88811227b46859"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}