{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kodcode-a-diverse-challenging-and-verifiable","title":"KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding","arxiv_id":"2503.02951","date":"2025-03-04","proceeding":null,"authors":["Zhangchen Xu","Yang Liu","Yueqin Yin","Mingyuan Zhou","Radha Poovendran"],"abstract":"We introduce KodCode, a synthetic dataset that addresses the persistent challenge of acquiring high-quality, verifiable training data across diverse difficulties and domains for training Large Language Models for coding. Existing code-focused resources typically fail to ensure either the breadth of coverage (e.g., spanning simple coding tasks to advanced algorithmic problems) or verifiable correctness (e.g., unit tests). In contrast, KodCode comprises question-solution-test triplets that are systematically validated via a self-verification procedure. Our pipeline begins by synthesizing a broad range of coding questions, then generates solutions and test cases with additional attempts allocated to challenging problems. Finally, post-training data synthesis is done by rewriting questions into diverse formats and generating responses under a test-based reject sampling procedure from a reasoning model (DeepSeek R1). This pipeline yields a large-scale, robust and diverse coding dataset. KodCode is suitable for supervised fine-tuning and the paired unit tests also provide great potential for RL tuning. Fine-tuning experiments on coding benchmarks (HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench) demonstrate that KodCode-tuned models achieve state-of-the-art performance, surpassing models like Qwen2.5-Coder-32B-Instruct and DeepSeek-R1-Distill-Llama-70B.","url_abs":"https://arxiv.org/abs/2503.02951v1","url_pdf":"https://arxiv.org/pdf/2503.02951v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kodcode-a-diverse-challenging-and-verifiable","repo_url":"https://github.com/KodCode-AI/kodcode","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":null,"task_name":"mbpp"}],"methods":[],"datasets_introduced":[{"slug":"kodcode-v1","name":"KodCode-V1","full_name":"KodCode/KodCode-V1"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.02951","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.02951"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KodCode-AI/kodcode","reach":null}],"summary":{"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"678e060316b39e74","entry":"energy_conversion","repo":"KodCode-AI/kodcode","repo_kind":"official","path":"demo/KodCode_algorithm_100_1741212598/self_verification_KodCode_algorithm_100_1741212598/00000000/trial_qwen2.5_0/solution.py","file_url":"https://github.com/KodCode-AI/kodcode/blob/HEAD/demo/KodCode_algorithm_100_1741212598/self_verification_KodCode_algorithm_100_1741212598/00000000/trial_qwen2.5_0/solution.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"678e060316b39e74"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}