{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/octopack-instruction-tuning-code-large","title":"OctoPack: Instruction Tuning Code Large Language Models","arxiv_id":"2308.07124","date":"2023-08-14","proceeding":null,"authors":["Niklas Muennighoff","Qian Liu","Armel Zebaze","Qinkai Zheng","Binyuan Hui","Terry Yue Zhuo","Swayam Singh","Xiangru Tang","Leandro von Werra","Shayne Longpre"],"abstract":"Finetuning large language models (LLMs) on instructions leads to vast performance improvements on natural language tasks. We apply instruction tuning using code, leveraging the natural structure of Git commits, which pair code changes with human instructions. We compile CommitPack: 4 terabytes of Git commits across 350 programming languages. We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2% pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis) across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models, OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among all permissive models, demonstrating CommitPack's benefits in generalizing to a wider set of languages and natural coding tasks. Code, models and data are freely available at https://github.com/bigcode-project/octopack.","url_abs":"https://arxiv.org/abs/2308.07124v2","url_pdf":"https://arxiv.org/pdf/2308.07124v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"octopack-instruction-tuning-code-large","repo_url":"https://github.com/bigcode-project/bigcode-evaluation-harness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"octopack-instruction-tuning-code-large","repo_url":"https://github.com/bigcode-project/octopack","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"octopack-instruction-tuning-code-large","repo_url":"https://github.com/infi-coder/infibench-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"code-repair","task_name":"Code Repair"},{"task_slug":"humaneval","task_name":"HumanEval"}],"methods":[{"method_slug":"repair","method_name":"Repair"}],"datasets_introduced":[{"slug":"commitpack","name":"CommitPack","full_name":"CommitPack"},{"slug":"commitpackft","name":"CommitPackFT","full_name":"CommitPackFT"},{"slug":"humanevalpack","name":"HumanEvalPack","full_name":"HumanEvalPack"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.07124","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.07124"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigcode-project/bigcode-evaluation-harness","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/infi-coder/infibench-evaluation-harness","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigcode-project/octopack","reach":null}],"summary":{"ran_draft_wrong":5,"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7655561f064db48a","entry":"cpp_instruct","repo":"bigcode-project/octopack","repo_kind":"official","path":"evaluation/create/create_instructions.py","file_url":"https://github.com/bigcode-project/octopack/blob/HEAD/evaluation/create/create_instructions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7655561f064db48a"}},{"code_sha256_prefix":"db89171e76d09770","entry":"get_diff","repo":"bigcode-project/octopack","repo_kind":"official","path":"dataset/commitpack/scrape_github.py","file_url":"https://github.com/bigcode-project/octopack/blob/HEAD/dataset/commitpack/scrape_github.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db89171e76d09770"}},{"code_sha256_prefix":"2905235d73624835","entry":"get_file_contents","repo":"bigcode-project/octopack","repo_kind":"official","path":"dataset/commitpack/scrape_github.py","file_url":"https://github.com/bigcode-project/octopack/blob/HEAD/dataset/commitpack/scrape_github.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2905235d73624835"}},{"code_sha256_prefix":"4ba028fb30a021b2","entry":"get_test_results_json_path","repo":"bigcode-project/bigcode-evaluation-harness","repo_kind":"official","path":"bigcode_eval/tasks/custom_metrics/multiple_metrics/evaluation.py","file_url":"https://github.com/bigcode-project/bigcode-evaluation-harness/blob/HEAD/bigcode_eval/tasks/custom_metrics/multiple_metrics/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ba028fb30a021b2"}},{"code_sha256_prefix":"ef807f76cf003934","entry":"java_instruct","repo":"bigcode-project/octopack","repo_kind":"official","path":"evaluation/create/create_instructions.py","file_url":"https://github.com/bigcode-project/octopack/blob/HEAD/evaluation/create/create_instructions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef807f76cf003934"}},{"code_sha256_prefix":"ef34b9d45f6cbdf6","entry":"python_instruct","repo":"bigcode-project/octopack","repo_kind":"official","path":"evaluation/create/create_instructions.py","file_url":"https://github.com/bigcode-project/octopack/blob/HEAD/evaluation/create/create_instructions.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef34b9d45f6cbdf6"}},{"code_sha256_prefix":"29ee1d8c120f674a","entry":"remove_after_return","repo":"bigcode-project/bigcode-evaluation-harness","repo_kind":"official","path":"bigcode_eval/utils.py","file_url":"https://github.com/bigcode-project/bigcode-evaluation-harness/blob/HEAD/bigcode_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29ee1d8c120f674a"}},{"code_sha256_prefix":"3266895adde27ef7","entry":"run_in_shell","repo":"bigcode-project/octopack","repo_kind":"official","path":"dataset/commitpack/scrape_github.py","file_url":"https://github.com/bigcode-project/octopack/blob/HEAD/dataset/commitpack/scrape_github.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3266895adde27ef7"}},{"code_sha256_prefix":"ac86a6f136bea277","entry":"update_code_gens","repo":"bigcode-project/bigcode-evaluation-harness","repo_kind":"official","path":"bigcode_eval/utils.py","file_url":"https://github.com/bigcode-project/bigcode-evaluation-harness/blob/HEAD/bigcode_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac86a6f136bea277"}},{"code_sha256_prefix":"f2e0ef72cfe052c9","entry":"cache_get","repo":"bigcode-project/bigcode-evaluation-harness","repo_kind":"official","path":"bigcode_eval/tasks/custom_metrics/multiple_metrics/evaluation.py","file_url":"https://github.com/bigcode-project/bigcode-evaluation-harness/blob/HEAD/bigcode_eval/tasks/custom_metrics/multiple_metrics/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2e0ef72cfe052c9"}},{"code_sha256_prefix":"2bceb28a9eb6ff16","entry":"cached_eval_script","repo":"bigcode-project/bigcode-evaluation-harness","repo_kind":"official","path":"bigcode_eval/tasks/custom_metrics/multiple_metrics/evaluation.py","file_url":"https://github.com/bigcode-project/bigcode-evaluation-harness/blob/HEAD/bigcode_eval/tasks/custom_metrics/multiple_metrics/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2bceb28a9eb6ff16"}},{"code_sha256_prefix":"21177b52be3390ee","entry":"complete_code","repo":"bigcode-project/bigcode-evaluation-harness","repo_kind":"official","path":"bigcode_eval/utils.py","file_url":"https://github.com/bigcode-project/bigcode-evaluation-harness/blob/HEAD/bigcode_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21177b52be3390ee"}},{"code_sha256_prefix":"d611d112ec49cc8e","entry":"compute_codexglue_code_to_text_bleu","repo":"bigcode-project/bigcode-evaluation-harness","repo_kind":"official","path":"bigcode_eval/tasks/codexglue_code_to_text.py","file_url":"https://github.com/bigcode-project/bigcode-evaluation-harness/blob/HEAD/bigcode_eval/tasks/codexglue_code_to_text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d611d112ec49cc8e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}