{"url":"/sota/code-generation-on-codecontests","task":{"name":"Code Generation","url":"/task/code-generation","note":null},"dataset":{"name":"CodeContests","url":"/dataset/codecontests"},"category":"Natural Language Processing","categories":["Computer Code","Natural Language Processing","Reasoning"],"category_note":null,"description":"**Code Generation** is an important field to predict explicit code or program structure from multimodal data sources such as incomplete code, programs in another programming language, natural language descriptions or execution examples. Code Generation tools can assist the development of automatic programming tools to improve programming productivity.\r\n\r\n\r\n<span class=\"description-source\">Source: [Deep Learning for Source Code Modeling and Generation ](https://arxiv.org/abs/2002.05442)</span>\r\n\r\nImage source: [Measuring Coding Challenge Competence With APPS](https://paperswithcode.com/paper/measuring-coding-challenge-competence-with)","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["Test Set pass@1","Test Set pass@5","Val Set pass@1","Val Set pass@5"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction); null = not inferred, chart draws points only","by_metric":{"Test Set pass@1":null,"Test Set pass@5":null,"Val Set pass@1":null,"Val Set pass@5":null}},"counts":{"rows":8,"rows_with_code":8,"rows_with_paper_page":8,"rows_dated":8,"rows_using_additional_data":0},"rows":[{"rank_in_archive_order":1,"model":"EG-CFG (DeepSeek-V3-0324)","metrics":{"Test Set pass@1":"58.18"},"uses_additional_data":false,"paper_date":"2025-06-12","paper":"/paper/execution-guided-line-by-line-code-generation","paper_url":"https://arxiv.org/abs/2506.10948v1","paper_title":"Execution Guided Line-by-Line Code Generation","code":"https://github.com/boazlavon/eg_cfg","n_code_links":1,"syntology":null},{"rank_in_archive_order":2,"model":"LPW (GPT-4o)","metrics":{"Test Set pass@1":"34.7"},"uses_additional_data":false,"paper_date":"2024-11-21","paper":"/paper/planning-driven-programming-a-large-language","paper_url":"https://arxiv.org/abs/2411.14503v3","paper_title":"Planning-Driven Programming: A Large Language Model Programming Workflow","code":"https://github.com/you68681/lpw","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":9,"n_samples":9,"n_pointer_only_licence":0}},{"rank_in_archive_order":3,"model":"MapCoder (GPT-4)","metrics":{"Test Set pass@1":"28.5","Test Set pass@5":"35.2","Val Set pass@1":"28.5"},"uses_additional_data":false,"paper_date":"2024-05-18","paper":"/paper/mapcoder-multi-agent-code-generation-for","paper_url":"https://arxiv.org/abs/2405.11403v1","paper_title":"MapCoder: Multi-Agent Code Generation for Competitive Problem Solving","code":"https://github.com/md-ashraful-pramanik/mapcoder","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":10,"n_samples":10,"n_pointer_only_licence":0}},{"rank_in_archive_order":4,"model":"CodeSim (GPT4)","metrics":{"Test Set pass@1":"28.4"},"uses_additional_data":false,"paper_date":"2025-02-08","paper":"/paper/codesim-multi-agent-code-generation-and-1","paper_url":"https://arxiv.org/abs/2502.05664v1","paper_title":"CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging","code":"https://github.com/kagnlp/CodeGenerator","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":3,"n_samples":3,"n_pointer_only_licence":0}},{"rank_in_archive_order":5,"model":"MoTCoder-15B","metrics":{"Test Set pass@1":"26.34","Val Set pass@1":"20.35"},"uses_additional_data":false,"paper_date":"2023-12-26","paper":"/paper/motcoder-elevating-large-language-models-with","paper_url":"https://arxiv.org/abs/2312.15960v5","paper_title":"MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks","code":"https://github.com/dvlab-research/motcoder","n_code_links":1,"syntology":{"n_ran":4,"n_unverified":4,"n_samples":8,"n_pointer_only_licence":8}},{"rank_in_archive_order":6,"model":"MoTCoder-7B-v1.5","metrics":{"Test Set pass@1":"20.77","Val Set pass@1":"16.72"},"uses_additional_data":false,"paper_date":"2023-12-26","paper":"/paper/motcoder-elevating-large-language-models-with","paper_url":"https://arxiv.org/abs/2312.15960v5","paper_title":"MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks","code":"https://github.com/dvlab-research/motcoder","n_code_links":1,"syntology":{"n_ran":4,"n_unverified":4,"n_samples":8,"n_pointer_only_licence":8}},{"rank_in_archive_order":7,"model":"CodeChain + WizardCoder-15B","metrics":{"Test Set pass@1":"2.35","Test Set pass@5":"3.29","Val Set pass@1":"2.48","Val Set pass@5":"3.30"},"uses_additional_data":false,"paper_date":"2023-10-13","paper":"/paper/codechain-towards-modular-code-generation","paper_url":"https://arxiv.org/abs/2310.08992v3","paper_title":"CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules","code":"https://github.com/SalesforceAIResearch/CodeChain","n_code_links":1,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":0}},{"rank_in_archive_order":8,"model":"WizardCoder-15B","metrics":{"Test Set pass@1":"1.11","Test Set pass@5":"3.18","Val Set pass@1":"1.98","Val Set pass@5":"3.27"},"uses_additional_data":false,"paper_date":"2023-06-14","paper":"/paper/wizardcoder-empowering-code-large-language","paper_url":"https://arxiv.org/abs/2306.08568v1","paper_title":"WizardCoder: Empowering Code Large Language Models with Evol-Instruct","code":"https://github.com/nlpxucan/wizardlm","n_code_links":4,"syntology":{"n_ran":3,"n_unverified":4,"n_samples":7,"n_pointer_only_licence":1}}],"since_archive":{"present":false,"note":"No Syntology-extracted rows are published in this build."},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per row: N of M harvested code samples from that row's paper executed on a synthesized fixture; the other M-N are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","rows_with_graph_line":7,"rows_with_any_sample_ran":4,"distinct_papers_with_graph_line":6,"distinct_papers_with_any_sample_ran":3,"samples_over_distinct_papers":{"n_ran":10,"n_unverified":30,"n_samples":40,"n_pointer_only_licence":9,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":14,"n_unverified":34,"n_samples":48,"n_pointer_only_licence":17,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}}}