{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2-olmo-2-furious","title":"2 OLMo 2 Furious","arxiv_id":"2501.00656","date":"2024-12-31","proceeding":null,"authors":["Team OLMo","Pete Walsh","Luca Soldaini","Dirk Groeneveld","Kyle Lo","Shane Arora","Akshita Bhagia","Yuling Gu","Shengyi Huang","Matt Jordan","Nathan Lambert","Dustin Schwenk","Oyvind Tafjord","Taira Anderson","David Atkinson","Faeze Brahman","Christopher Clark","Pradeep Dasigi","Nouha Dziri","Michal Guerquin","Hamish Ivison","Pang Wei Koh","Jiacheng Liu","Saumya Malik","William Merrill","Lester James V. Miranda","Jacob Morrison","Tyler Murray","Crystal Nam","Valentina Pyatkin","Aman Rangapur","Michael Schmitz","Sam Skjonsberg","David Wadden","Christopher Wilhelm","Michael Wilson","Luke Zettlemoyer","Ali Farhadi","Noah A. Smith","Hannaneh Hajishirzi"],"abstract":"We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes dense autoregressive models with improved architecture and training recipe, pretraining data mixtures, and instruction tuning recipes. Our modified model architecture and training recipe achieve both better training stability and improved per-token efficiency. Our updated pretraining data mixture introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best practices from T\\\"ulu 3 to develop OLMo 2-Instruct, focusing on permissive data and extending our final-stage reinforcement learning with verifiable rewards (RLVR). Our OLMo 2 base models sit at the Pareto frontier of performance to compute, often matching or outperforming open-weight only models like Llama 3.1 and Qwen 2.5 while using fewer FLOPs and with fully transparent training data, code, and recipe. Our fully open OLMo 2-Instruct models are competitive with or surpassing open-weight only models of comparable size, including Qwen 2.5, Llama 3.1 and Gemma 2. We release all OLMo 2 artifacts openly -- models at 7B and 13B scales, both pretrained and post-trained, including their full training data, training code and recipes, training logs and thousands of intermediate checkpoints. The final instruction model is available on the Ai2 Playground as a free research demo.","url_abs":"https://arxiv.org/abs/2501.00656v2","url_pdf":"https://arxiv.org/pdf/2501.00656v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2-olmo-2-furious","repo_url":"https://github.com/allenai/OLMo-core","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"2-olmo-2-furious","repo_url":"https://github.com/allenai/olmes","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"2-olmo-2-furious","repo_url":"https://github.com/allenai/olmo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.00656","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.00656"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/OLMo-core","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/olmo","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/olmes","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":19},"by_repo_kind":{"official":{"samples":21,"ran":2,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f9abf6d74699cc1c","entry":"causal_attention_bias","repo":"allenai/olmo","repo_kind":"official","path":"olmo/model.py","file_url":"https://github.com/allenai/olmo/blob/HEAD/olmo/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9abf6d74699cc1c"}},{"code_sha256_prefix":"1e05670874fea1ef","entry":"longest_common_prefix","repo":"allenai/olmo","repo_kind":"official","path":"hf_olmo/convert_olmo_to_hf.py","file_url":"https://github.com/allenai/olmo/blob/HEAD/hf_olmo/convert_olmo_to_hf.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e05670874fea1ef"}},{"code_sha256_prefix":"eef0e57960f509e5","entry":"all_to_all","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/distributed/nn.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/distributed/nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eef0e57960f509e5"}},{"code_sha256_prefix":"60bd6b2361f5a8b0","entry":"all_to_all_single","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/distributed/nn.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/distributed/nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60bd6b2361f5a8b0"}},{"code_sha256_prefix":"903f4474fe15cdaf","entry":"beta_feature","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/doc_utils.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/doc_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"903f4474fe15cdaf"}},{"code_sha256_prefix":"f0c3a597e3ccfaf6","entry":"decode_key","repo":"allenai/olmo","repo_kind":"official","path":"olmo/safetensors_util.py","file_url":"https://github.com/allenai/olmo/blob/HEAD/olmo/safetensors_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0c3a597e3ccfaf6"}},{"code_sha256_prefix":"1325dacb19f6bd7e","entry":"delete_model","repo":"allenai/olmes","repo_kind":"official","path":"oe_eval/utilities/model_utils.py","file_url":"https://github.com/allenai/olmes/blob/HEAD/oe_eval/utilities/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1325dacb19f6bd7e"}},{"code_sha256_prefix":"e3246d51c7d7535c","entry":"encode_key","repo":"allenai/olmo","repo_kind":"official","path":"olmo/safetensors_util.py","file_url":"https://github.com/allenai/olmo/blob/HEAD/olmo/safetensors_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3246d51c7d7535c"}},{"code_sha256_prefix":"1ab94002f679ebd2","entry":"env_bool","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/utils.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ab94002f679ebd2"}},{"code_sha256_prefix":"379ba3f9c8a51ff3","entry":"extract_think_answer","repo":"allenai/olmes","repo_kind":"official","path":"oe_eval/models/output_processors.py","file_url":"https://github.com/allenai/olmes/blob/HEAD/oe_eval/models/output_processors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"379ba3f9c8a51ff3"}},{"code_sha256_prefix":"8c736d35652665c6","entry":"fix_optim_state_dict","repo":"allenai/olmo","repo_kind":"official","path":"olmo/optim.py","file_url":"https://github.com/allenai/olmo/blob/HEAD/olmo/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c736d35652665c6"}},{"code_sha256_prefix":"b593a4bbddcc66c3","entry":"flatten_dict","repo":"allenai/olmo","repo_kind":"official","path":"olmo/safetensors_util.py","file_url":"https://github.com/allenai/olmo/blob/HEAD/olmo/safetensors_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b593a4bbddcc66c3"}},{"code_sha256_prefix":"7162dfdf480a8995","entry":"get_completion","repo":"allenai/olmes","repo_kind":"official","path":"oe_eval/models/litellm.py","file_url":"https://github.com/allenai/olmes/blob/HEAD/oe_eval/models/litellm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7162dfdf480a8995"}},{"code_sha256_prefix":"32df15687ce14fd7","entry":"get_fs_local_rank","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/distributed/utils.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/distributed/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32df15687ce14fd7"}},{"code_sha256_prefix":"cf29bba61f4e7b45","entry":"get_global_rank","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/distributed/utils.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/distributed/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf29bba61f4e7b45"}},{"code_sha256_prefix":"b32994fbeacb1e9b","entry":"get_rank","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/distributed/utils.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/distributed/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b32994fbeacb1e9b"}},{"code_sha256_prefix":"b48e416f25d4cc2b","entry":"maybe_nvtx_annotate","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/_nvtx.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/_nvtx.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b48e416f25d4cc2b"}},{"code_sha256_prefix":"efa5007ea3ca9d06","entry":"move_to_device","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/utils.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efa5007ea3ca9d06"}},{"code_sha256_prefix":"173adb946721713b","entry":"normalize_mxfp8_scale_mode_value","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/mxfp8_config.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/mxfp8_config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"173adb946721713b"}},{"code_sha256_prefix":"c85e4b8ba614f924","entry":"process_batch","repo":"allenai/olmes","repo_kind":"official","path":"oe_eval/models/litellm.py","file_url":"https://github.com/allenai/olmes/blob/HEAD/oe_eval/models/litellm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c85e4b8ba614f924"}},{"code_sha256_prefix":"b4c909652686eb75","entry":"rank_matches_filter","repo":"allenai/OLMo-core","repo_kind":"official","path":"src/olmo_core/utils.py","file_url":"https://github.com/allenai/OLMo-core/blob/HEAD/src/olmo_core/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4c909652686eb75"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}