{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/yi-open-foundation-models-by-01-ai","title":"Yi: Open Foundation Models by 01.AI","arxiv_id":"2403.04652","date":"2024-03-07","proceeding":null,"authors":["01. AI",":","Alex Young","Bei Chen","Chao Li","Chengen Huang","Ge Zhang","Guanwei Zhang","Guoyin Wang","Heng Li","Jiangcheng Zhu","Jianqun Chen","Jing Chang","Kaidong Yu","Peng Liu","Qiang Liu","Shawn Yue","Senbin Yang","Shiming Yang","Wen Xie","Wenhao Huang","Xiaohui Hu","Xiaoyi Ren","Xinyao Niu","Pengcheng Nie","Yanpeng Li","Yuchi Xu","Yudong Liu","Yue Wang","Yuxuan Cai","Zhenyu Gu","Zhiyuan Liu","Zonghong Dai"],"abstract":"We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.","url_abs":"https://arxiv.org/abs/2403.04652v3","url_pdf":"https://arxiv.org/pdf/2403.04652v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"yi-open-foundation-models-by-01-ai","repo_url":"https://github.com/01-ai/yi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"chatbot","task_name":"Chatbot"},{"task_slug":"continual-pretraining","task_name":"Continual Pretraining"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mmlu","task_name":"MMLU"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/chatbot-on-alpacaeval","task":"Chatbot","dataset":"AlpacaEval","model":"Yi 34B Chat","rank_in_archive_order":1,"of":1,"metrics":{"Average win rate":"27.2"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.04652","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.04652"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/01-ai/yi","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9fa178e6891384f1","entry":"build_vision_projector","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9fa178e6891384f1"}},{"code_sha256_prefix":"e016ec919cd88d1e","entry":"build_vision_tower","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e016ec919cd88d1e"}},{"code_sha256_prefix":"f324cbfd797797c5","entry":"CLIPVisionTower","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f324cbfd797797c5"}},{"code_sha256_prefix":"3c8ff5cd40f90abd","entry":"LlavaConfig","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c8ff5cd40f90abd"}},{"code_sha256_prefix":"3ce5244608be8933","entry":"LlavaLlamaForCausalLM","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ce5244608be8933"}},{"code_sha256_prefix":"1b3df989b0062240","entry":"LlavaLlamaModel","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b3df989b0062240"}},{"code_sha256_prefix":"cb4478f3b5e0497f","entry":"LlavaMetaForCausalLM","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb4478f3b5e0497f"}},{"code_sha256_prefix":"f55142d84e5ac3d0","entry":"LlavaMetaModel","repo":"01-ai/yi","repo_kind":"official","path":"VL/llava/model/llava_llama.py","file_url":"https://github.com/01-ai/yi/blob/HEAD/VL/llava/model/llava_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f55142d84e5ac3d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}