{"url":"/sota/code-generation-on-humaneval","task":{"name":"Code Generation","url":"/task/code-generation","note":null},"dataset":{"name":"HumanEval","url":"/dataset/humaneval"},"category":"Natural Language Processing","categories":["Computer Code","Natural Language Processing","Reasoning"],"category_note":null,"description":"**Code Generation** is an important field to predict explicit code or program structure from multimodal data sources such as incomplete code, programs in another programming language, natural language descriptions or execution examples. Code Generation tools can assist the development of automatic programming tools to improve programming productivity.\r\n\r\n\r\n<span class=\"description-source\">Source: [Deep Learning for Source Code Modeling and Generation ](https://arxiv.org/abs/2002.05442)</span>\r\n\r\nImage source: [Measuring Coding Challenge Competence With APPS](https://paperswithcode.com/paper/measuring-coding-challenge-competence-with)","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["Pass@1"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction); null = not inferred, chart draws points only","by_metric":{"Pass@1":null}},"counts":{"rows":8,"rows_with_code":6,"rows_with_paper_page":7,"rows_dated":7,"rows_using_additional_data":0},"rows":[{"rank_in_archive_order":1,"model":"DeepSeek-R1 (MGDebugger)","metrics":{"Pass@1":"100"},"uses_additional_data":false,"paper_date":"2024-10-02","paper":"/paper/from-code-to-correctness-closing-the-last","paper_url":"https://arxiv.org/abs/2410.01215v2","paper_title":"From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging","code":"https://github.com/YerbaPage/MGDebugger","n_code_links":1,"syntology":{"n_ran":8,"n_unverified":1,"n_samples":9,"n_pointer_only_licence":0}},{"rank_in_archive_order":2,"model":"LLaMA 3","metrics":{"Pass@1":"99.4"},"uses_additional_data":false,"paper_date":"2024-02-25","paper":"/paper/ldb-a-large-language-model-debugger-via","paper_url":"https://arxiv.org/abs/2402.16906v6","paper_title":"Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step","code":"https://github.com/floridsleeves/llmdebugger","n_code_links":1,"syntology":{"n_ran":8,"n_unverified":4,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":3,"model":"QualityFlow (Sonnet-3.5)","metrics":{"Pass@1":"98.8"},"uses_additional_data":false,"paper_date":"2025-01-20","paper":"/paper/qualityflow-an-agentic-workflow-for-program","paper_url":"https://arxiv.org/abs/2501.17167v2","paper_title":"QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":4,"model":"Phi-2","metrics":{"Pass@1":"98.2"},"uses_additional_data":false,"paper_date":"2024-11-21","paper":"/paper/planning-driven-programming-a-large-language","paper_url":"https://arxiv.org/abs/2411.14503v3","paper_title":"Planning-Driven Programming: A Large Language Model Programming Workflow","code":"https://github.com/you68681/lpw","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":9,"n_samples":9,"n_pointer_only_licence":0}},{"rank_in_archive_order":5,"model":"EG-CFG (DeepSeek-V3-0324)","metrics":{"Pass@1":"96.95"},"uses_additional_data":false,"paper_date":"2025-06-12","paper":"/paper/execution-guided-line-by-line-code-generation","paper_url":"https://arxiv.org/abs/2506.10948v1","paper_title":"Execution Guided Line-by-Line Code Generation","code":"https://github.com/boazlavon/eg_cfg","n_code_links":1,"syntology":null},{"rank_in_archive_order":6,"model":"Mistral 7B","metrics":{"Pass@1":"93.9"},"uses_additional_data":false,"paper_date":"2024-05-18","paper":"/paper/mapcoder-multi-agent-code-generation-for","paper_url":"https://arxiv.org/abs/2405.11403v1","paper_title":"MapCoder: Multi-Agent Code Generation for Competitive Problem Solving","code":"https://github.com/md-ashraful-pramanik/mapcoder","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":10,"n_samples":10,"n_pointer_only_licence":0}},{"rank_in_archive_order":7,"model":"Claude Sonnet 3.5","metrics":{"Pass@1":"90.85"},"uses_additional_data":false,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":8,"model":"L2MAC (GPT-4)","metrics":{"Pass@1":"90.2"},"uses_additional_data":false,"paper_date":"2023-10-02","paper":"/paper/l2mac-large-language-model-automatic-computer","paper_url":"https://arxiv.org/abs/2310.02003v5","paper_title":"L2MAC: Large Language Model Automatic Computer for Extensive Code Generation","code":"https://github.com/samholt/l2mac","n_code_links":2,"syntology":{"n_ran":10,"n_unverified":4,"n_samples":14,"n_pointer_only_licence":0}}],"since_archive":{"present":false,"note":"No Syntology-extracted rows are published in this build."},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per row: N of M harvested code samples from that row's paper executed on a synthesized fixture; the other M-N are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","rows_with_graph_line":5,"rows_with_any_sample_ran":3,"distinct_papers_with_graph_line":5,"distinct_papers_with_any_sample_ran":3,"samples_over_distinct_papers":{"n_ran":26,"n_unverified":28,"n_samples":54,"n_pointer_only_licence":0,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":26,"n_unverified":28,"n_samples":54,"n_pointer_only_licence":0,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}}}