{"url":"/task/code-generation","name":"Code Generation","slug":"code-generation","description_markdown":"**Code Generation** is an important field to predict explicit code or program structure from multimodal data sources such as incomplete code, programs in another programming language, natural language descriptions or execution examples. Code Generation tools can assist the development of automatic programming tools to improve programming productivity.\r\n\r\n\r\n<span class=\"description-source\">Source: [Deep Learning for Source Code Modeling and Generation ](https://arxiv.org/abs/2002.05442)</span>\r\n\r\nImage source: [Measuring Coding Challenge Competence With APPS](https://paperswithcode.com/paper/measuring-coding-challenge-competence-with)","categories":[{"name":"Computer Code","url":"/area/computer-code"},{"name":"Natural Language Processing","url":"/area/natural-language-processing"},{"name":"Reasoning","url":"/area/reasoning"}],"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","slug_source":"archive_url"},"counts":{"papers_tagged":1697,"papers_with_code":745,"benchmarks":27,"benchmark_tables_in_archive":27,"benchmark_tables_shown":27,"benchmark_tables_withheld_as_spam":0,"benchmark_definition":"a leaderboard table with at least one row; benchmark_tables_shown also counts the zero-row tables; benchmark_tables_in_archive adds the tables withheld as spam","datasets":70,"subtasks":5,"parent_tasks":0},"benchmarks":[{"leaderboard":"/sota/code-generation-on-mbpp","slug":"code-generation-on-mbpp","dataset":"MBPP","dataset_url":"/dataset/mbpp","rows_in_archive":99,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"EG-CFG (DeepSeek-V3-0324)","paper_title":"Execution Guided Line-by-Line Code Generation","paper_url":"/paper/execution-guided-line-by-line-code-generation","paper_date":"2025-06-12","arxiv_id":"2506.10948","code_links":[{"title":"boazlavon/eg_cfg","url":"https://github.com/boazlavon/eg_cfg"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-apps","slug":"code-generation-on-apps","dataset":"APPS","dataset_url":"/dataset/apps","rows_in_archive":18,"metrics":["Introductory Pass@1","Interview Pass@1","Competition Pass@1","Competition Pass@any","Interview Pass@any","Introductory Pass@any","Competition Pass@5","Interview Pass@5","Introductory Pass@5","Competition Pass@1000","Interview Pass@1000","Introductory Pass@1000","Pass@1"],"first_row_in_archive_order":{"model":"LPW (GPT-4o)","paper_title":"Planning-Driven Programming: A Large Language Model Programming Workflow","paper_url":"/paper/planning-driven-programming-a-large-language","paper_date":"2024-11-21","arxiv_id":"2411.14503","code_links":[{"title":"you68681/lpw","url":"https://github.com/you68681/lpw"}],"syntology":{"n":9,"n_ran":0,"n_unverified":9,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-conala","slug":"code-generation-on-conala","dataset":"CoNaLa","dataset_url":"/dataset/conala","rows_in_archive":14,"metrics":["BLEU","Exact Match Accuracy"],"first_row_in_archive_order":{"model":"PanGu-Coder-FT-I","paper_title":"Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets","paper_url":"/paper/fine-tuning-large-language-models-for","paper_date":"2023-06-26","arxiv_id":null,"code_links":[],"syntology":null}},{"leaderboard":"/sota/code-generation-on-django","slug":"code-generation-on-django","dataset":"Django","dataset_url":"/dataset/django","rows_in_archive":11,"metrics":["Accuracy","BLEU Score"],"first_row_in_archive_order":{"model":"MarianCG","paper_title":"MarianCG: a code generation transformer model inspired by machine translation","paper_url":"/paper/mariancg-a-code-generation-transformer-model","paper_date":"2022-11-22","arxiv_id":null,"code_links":[{"title":"AhmedSSoliman/MarianCG-NL-to-Code","url":"https://github.com/AhmedSSoliman/MarianCG-NL-to-Code"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-wikisql","slug":"code-generation-on-wikisql","dataset":"WikiSQL","dataset_url":"/dataset/wikisql","rows_in_archive":10,"metrics":["Execution Accuracy","Exact Match Accuracy"],"first_row_in_archive_order":{"model":"NL2SQL-RULE","paper_title":"Content Enhanced BERT-based Text-to-SQL Generation","paper_url":"/paper/content-enhanced-bert-based-text-to-sql","paper_date":"2019-10-16","arxiv_id":"1910.07179","code_links":[{"title":"guotong1988/NL2SQL-RULE","url":"https://github.com/guotong1988/NL2SQL-RULE"},{"title":"guotong1988/NL2SQL-BERT","url":"https://github.com/guotong1988/NL2SQL-BERT"},{"title":"shivam017arora/Conversational-BI","url":"https://github.com/shivam017arora/Conversational-BI"},{"title":"yangyucheng000/University","url":"https://github.com/yangyucheng000/University/tree/main/model-2/bert_generation"},{"title":"realsonalkumar/Mish-Mash-Hackathon","url":"https://github.com/realsonalkumar/Mish-Mash-Hackathon"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-res-q","slug":"code-generation-on-res-q","dataset":"RES-Q","dataset_url":"/dataset/res-q","rows_in_archive":9,"metrics":["pass@1"],"first_row_in_archive_order":{"model":"QurrentOS-coder + Claude 3.5 Sonnet","paper_title":"RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale","paper_url":"/paper/res-q-evaluating-code-editing-large-language","paper_date":"2024-06-24","arxiv_id":"2406.16801","code_links":[{"title":"qurrent-ai/res-q","url":"https://github.com/qurrent-ai/res-q"}],"syntology":{"n":6,"n_ran":6,"n_unverified":0,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-codecontests","slug":"code-generation-on-codecontests","dataset":"CodeContests","dataset_url":"/dataset/codecontests","rows_in_archive":8,"metrics":["Test Set pass@1","Test Set pass@5","Val Set pass@1","Val Set pass@5"],"first_row_in_archive_order":{"model":"EG-CFG (DeepSeek-V3-0324)","paper_title":"Execution Guided Line-by-Line Code Generation","paper_url":"/paper/execution-guided-line-by-line-code-generation","paper_date":"2025-06-12","arxiv_id":"2506.10948","code_links":[{"title":"boazlavon/eg_cfg","url":"https://github.com/boazlavon/eg_cfg"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-humaneval","slug":"code-generation-on-humaneval","dataset":"HumanEval","dataset_url":"/dataset/humaneval","rows_in_archive":8,"metrics":["Pass@1"],"first_row_in_archive_order":{"model":"DeepSeek-R1 (MGDebugger)","paper_title":"From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging","paper_url":"/paper/from-code-to-correctness-closing-the-last","paper_date":"2024-10-02","arxiv_id":"2410.01215","code_links":[{"title":"YerbaPage/MGDebugger","url":"https://github.com/YerbaPage/MGDebugger"}],"syntology":{"n":9,"n_ran":8,"n_unverified":1,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-pecc","slug":"code-generation-on-pecc","dataset":"PECC","dataset_url":"/dataset/pecc","rows_in_archive":8,"metrics":["Pass@3"],"first_row_in_archive_order":{"model":"Claude 3 Haiku","paper_title":"PECC: Problem Extraction and Coding Challenges","paper_url":"/paper/pecc-problem-extraction-and-coding-challenges","paper_date":"2024-04-29","arxiv_id":"2404.18766","code_links":[{"title":"hallerpatrick/pecc","url":"https://github.com/hallerpatrick/pecc"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-webapp1k-react","slug":"code-generation-on-webapp1k-react","dataset":"WebApp1K-React","dataset_url":"/dataset/webapp1k-react","rows_in_archive":8,"metrics":["pass@1"],"first_row_in_archive_order":{"model":"o1-preview","paper_title":"A Case Study of Web App Coding with OpenAI Reasoning Models","paper_url":"/paper/a-case-study-of-web-app-coding-with-openai","paper_date":"2024-09-19","arxiv_id":"2409.13773","code_links":[{"title":"onekq/webapp1k","url":"https://github.com/onekq/webapp1k"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-conala-ext","slug":"code-generation-on-conala-ext","dataset":"CoNaLa-Ext","dataset_url":"/dataset/conala-ext","rows_in_archive":6,"metrics":["BLEU"],"first_row_in_archive_order":{"model":"BART W/ Mined","paper_title":"Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation","paper_url":"/paper/reading-stackoverflow-encourages-cheating","paper_date":"2021-06-08","arxiv_id":"2106.04447","code_links":[{"title":"gabeorlanski/stackoverflow-encourages-cheating","url":"https://github.com/gabeorlanski/stackoverflow-encourages-cheating"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-webapp1k-duo-react","slug":"code-generation-on-webapp1k-duo-react","dataset":"WebApp1k-Duo-React","dataset_url":"/dataset/webapp1k-duo-react","rows_in_archive":6,"metrics":["pass@1"],"first_row_in_archive_order":{"model":"claude-3-5-sonnet","paper_title":"A Case Study of Web App Coding with OpenAI Reasoning Models","paper_url":"/paper/a-case-study-of-web-app-coding-with-openai","paper_date":"2024-09-19","arxiv_id":"2409.13773","code_links":[{"title":"onekq/webapp1k","url":"https://github.com/onekq/webapp1k"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-dseval-leetcode","slug":"code-generation-on-dseval-leetcode","dataset":"DSEval-LeetCode","dataset_url":"/dataset/dseval-leetcode","rows_in_archive":5,"metrics":["Pass Rate","w/o Intact","w/o PE"],"first_row_in_archive_order":{"model":"Jupyter-AI","paper_title":null,"paper_url":null,"paper_date":"","arxiv_id":null,"code_links":[],"syntology":null}},{"leaderboard":"/sota/code-generation-on-turbulence","slug":"code-generation-on-turbulence","dataset":"Turbulence","dataset_url":"/dataset/turbulence","rows_in_archive":5,"metrics":["CorrSc"],"first_row_in_archive_order":{"model":"GPT-4","paper_title":"Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code","paper_url":"/paper/turbulence-systematically-and-automatically","paper_date":"2023-12-22","arxiv_id":"2312.14856","code_links":[{"title":"shahinhonarvar/turbulence-benchmark","url":"https://github.com/shahinhonarvar/turbulence-benchmark"}],"syntology":{"n":4,"n_ran":4,"n_unverified":0,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-livecodebench","slug":"code-generation-on-livecodebench","dataset":"Livecodebench","dataset_url":null,"rows_in_archive":3,"metrics":["Acc"],"first_row_in_archive_order":{"model":"Xolver","paper_title":"Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Team","paper_url":"/paper/xolver-multi-agent-reasoning-with-holistic","paper_date":"2025-06-17","arxiv_id":"2506.14234","code_links":[{"title":"kagnlp/Xolver","url":"https://github.com/kagnlp/Xolver"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-shellcode-ia32","slug":"code-generation-on-shellcode-ia32","dataset":"Shellcode_IA32","dataset_url":"/dataset/shellcode-ia32","rows_in_archive":3,"metrics":["BLEU-4","Exact Match Accuracy"],"first_row_in_archive_order":{"model":"CodeBERT","paper_title":"Can We Generate Shellcodes via Natural Language? An Empirical Study","paper_url":"/paper/can-we-generate-shellcodes-via-natural","paper_date":"2022-02-08","arxiv_id":"2202.03755","code_links":[{"title":"dessertlab/Shellcode_IA32","url":"https://github.com/dessertlab/Shellcode_IA32"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-taco-code","slug":"code-generation-on-taco-code","dataset":"TACO-BAAI","dataset_url":"/dataset/taco-topics-in-algorithmic-code-generation","rows_in_archive":3,"metrics":["easy pass@1"],"first_row_in_archive_order":{"model":"GPT-4","paper_title":"TACO: Topics in Algorithmic COde generation dataset","paper_url":"/paper/taco-topics-in-algorithmic-code-generation","paper_date":"2023-12-22","arxiv_id":"2312.14852","code_links":[{"title":"flagopen/taco","url":"https://github.com/flagopen/taco"}],"syntology":{"n":2,"n_ran":1,"n_unverified":1,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-bigcodebench-complete","slug":"code-generation-on-bigcodebench-complete","dataset":"BigCodeBench-Complete","dataset_url":"/dataset/bigcodebench","rows_in_archive":2,"metrics":["Pass@1"],"first_row_in_archive_order":{"model":"GPT-4o-2024-05-13","paper_title":"BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions","paper_url":"/paper/bigcodebench-benchmarking-code-generation","paper_date":"2024-06-22","arxiv_id":"2406.15877","code_links":[{"title":"mlfoundations/Evalchemy","url":"https://github.com/mlfoundations/Evalchemy"},{"title":"bigcode-project/bigcodebench","url":"https://github.com/bigcode-project/bigcodebench"},{"title":"bigcode-project/bigcodebench-annotation","url":"https://github.com/bigcode-project/bigcodebench-annotation"},{"title":"stovecat/convcodeworld","url":"https://github.com/stovecat/convcodeworld"}],"syntology":{"n":6,"n_ran":4,"n_unverified":2,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-bigcodebench-instruct","slug":"code-generation-on-bigcodebench-instruct","dataset":"BigCodeBench-Instruct","dataset_url":"/dataset/bigcodebench","rows_in_archive":2,"metrics":["Pass@1"],"first_row_in_archive_order":{"model":"GPT-4o-2024-05-13","paper_title":"BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions","paper_url":"/paper/bigcodebench-benchmarking-code-generation","paper_date":"2024-06-22","arxiv_id":"2406.15877","code_links":[{"title":"mlfoundations/Evalchemy","url":"https://github.com/mlfoundations/Evalchemy"},{"title":"bigcode-project/bigcodebench","url":"https://github.com/bigcode-project/bigcodebench"},{"title":"bigcode-project/bigcodebench-annotation","url":"https://github.com/bigcode-project/bigcodebench-annotation"},{"title":"stovecat/convcodeworld","url":"https://github.com/stovecat/convcodeworld"}],"syntology":{"n":6,"n_ran":4,"n_unverified":2,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-codexglue-codesearchnet","slug":"code-generation-on-codexglue-codesearchnet","dataset":"CodeXGLUE - CodeSearchNet","dataset_url":"/dataset/codexglue","rows_in_archive":2,"metrics":["Java/EM","Python/EM","Java/BLEU","Python/BLEU","Java/CodeBLEU","Python/CodeBLEU"],"first_row_in_archive_order":{"model":"Redcoder-ext","paper_title":"Retrieval Augmented Code Generation and Summarization","paper_url":"/paper/retrieval-augmented-code-generation-and","paper_date":"2021-08-26","arxiv_id":"2108.11601","code_links":[{"title":"kagnlp/CodeGenerator","url":"https://github.com/kagnlp/CodeGenerator"},{"title":"rizwan09/redcoder","url":"https://github.com/rizwan09/redcoder"}],"syntology":{"n":3,"n_ran":2,"n_unverified":1,"n_pointer_only":3}}},{"leaderboard":"/sota/code-generation-on-concode","slug":"code-generation-on-concode","dataset":"CONCODE","dataset_url":"/dataset/concode","rows_in_archive":2,"metrics":["Exact Match","BLEU","CodeBLEU"],"first_row_in_archive_order":{"model":"Redcoder-ext","paper_title":"Retrieval Augmented Code Generation and Summarization","paper_url":"/paper/retrieval-augmented-code-generation-and","paper_date":"2021-08-26","arxiv_id":"2108.11601","code_links":[{"title":"kagnlp/CodeGenerator","url":"https://github.com/kagnlp/CodeGenerator"},{"title":"rizwan09/redcoder","url":"https://github.com/rizwan09/redcoder"}],"syntology":{"n":3,"n_ran":2,"n_unverified":1,"n_pointer_only":3}}},{"leaderboard":"/sota/code-generation-on-humaneval-et","slug":"code-generation-on-humaneval-et","dataset":"HumanEval-ET","dataset_url":"/dataset/humaneval-et","rows_in_archive":2,"metrics":["Pass@1"],"first_row_in_archive_order":{"model":"EG-CFG (DeepSeek-V3-0324)","paper_title":"Execution Guided Line-by-Line Code Generation","paper_url":"/paper/execution-guided-line-by-line-code-generation","paper_date":"2025-06-12","arxiv_id":"2506.10948","code_links":[{"title":"boazlavon/eg_cfg","url":"https://github.com/boazlavon/eg_cfg"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-mbpp-et","slug":"code-generation-on-mbpp-et","dataset":"MBPP-ET","dataset_url":"/dataset/mbpp-et","rows_in_archive":2,"metrics":["Pass@1"],"first_row_in_archive_order":{"model":"EG-CFG (DeepSeek-V3-0324)","paper_title":"Execution Guided Line-by-Line Code Generation","paper_url":"/paper/execution-guided-line-by-line-code-generation","paper_date":"2025-06-12","arxiv_id":"2506.10948","code_links":[{"title":"boazlavon/eg_cfg","url":"https://github.com/boazlavon/eg_cfg"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-verified-smart-contract","slug":"code-generation-on-verified-smart-contract","dataset":"Verified Smart Contract Code Comments","dataset_url":"/dataset/verified-smart-contract-code-comments","rows_in_archive":2,"metrics":["BLEU score"],"first_row_in_archive_order":{"model":"GPT-J 6B Smart Contract","paper_title":"Efficient Avoidance of Vulnerabilities in Auto-completed Smart Contract Code Using Vulnerability-constrained Decoding","paper_url":"/paper/efficient-avoidance-of-vulnerabilities-in","paper_date":"2023-09-18","arxiv_id":"2309.09826","code_links":[],"syntology":null}},{"leaderboard":"/sota/code-generation-on-android-repos","slug":"code-generation-on-android-repos","dataset":"Android Repos","dataset_url":null,"rows_in_archive":1,"metrics":["Perplexity"],"first_row_in_archive_order":{"model":"Entity Type Model","paper_title":"Building Language Models for Text with Named Entities","paper_url":"/paper/building-language-models-for-text-with-named","paper_date":"2018-05-13","arxiv_id":"1805.04836","code_links":[{"title":"uclanlp/NamedEntityLanguageModel","url":"https://github.com/uclanlp/NamedEntityLanguageModel"},{"title":"devanshbatra04/Named-Entity-Language-Models","url":"https://github.com/devanshbatra04/Named-Entity-Language-Models"}],"syntology":{"n":1,"n_ran":0,"n_unverified":1,"n_pointer_only":0}}},{"leaderboard":"/sota/code-generation-on-floco","slug":"code-generation-on-floco","dataset":"FloCo","dataset_url":"/dataset/floco","rows_in_archive":1,"metrics":["BLEU","CodeBLEU"],"first_row_in_archive_order":{"model":"FloCo-T5","paper_title":"Towards Making Flowchart Images Machine Interpretable","paper_url":"/paper/towards-making-flowchart-images-machine-1","paper_date":"2025-01-29","arxiv_id":"2501.17441","code_links":[{"title":"vl2g/floco","url":"https://github.com/vl2g/floco"}],"syntology":null}},{"leaderboard":"/sota/code-generation-on-verilogeval","slug":"code-generation-on-verilogeval","dataset":"VerilogEval","dataset_url":"/dataset/verilogeval","rows_in_archive":1,"metrics":["Pass Rate"],"first_row_in_archive_order":{"model":"Nexus (Claude 3.5 Sonnet)","paper_title":"Nexus: A Lightweight and Scalable Multi-Agent Framework for Complex Tasks Automation","paper_url":"/paper/nexus-a-lightweight-and-scalable-multi-agent","paper_date":"2025-02-26","arxiv_id":"2502.19091","code_links":[{"title":"PrimisAI/nexus","url":"https://github.com/PrimisAI/nexus"}],"syntology":null}}],"datasets":[{"url":"/dataset/humaneval","name":"HumanEval","full_name":"","num_papers_in_archive":1201},{"url":"/dataset/mbpp","name":"MBPP","full_name":"Mostly Basic Python Programming","num_papers_in_archive":666},{"url":"/dataset/wikisql","name":"WikiSQL","full_name":"WikiSQL","num_papers_in_archive":267},{"url":"/dataset/codexglue","name":"CodeXGLUE","full_name":"","num_papers_in_archive":205},{"url":"/dataset/apps","name":"APPS","full_name":"Automated Programming Progress Standard","num_papers_in_archive":180},{"url":"/dataset/codecontests","name":"CodeContests","full_name":"","num_papers_in_archive":84},{"url":"/dataset/conala","name":"CoNaLa","full_name":"CMU CoNaLa, the Code/Natural Language Challenge","num_papers_in_archive":77},{"url":"/dataset/ds-1000","name":"DS-1000","full_name":"","num_papers_in_archive":72},{"url":"/dataset/concode","name":"CONCODE","full_name":"","num_papers_in_archive":46},{"url":"/dataset/bigcodebench","name":"BigCodeBench","full_name":"","num_papers_in_archive":38},{"url":"/dataset/classeval","name":"ClassEval","full_name":"","num_papers_in_archive":27},{"url":"/dataset/django","name":"Django","full_name":"Django","num_papers_in_archive":22},{"url":"/dataset/hearthstone","name":"Hearthstone","full_name":"","num_papers_in_archive":22},{"url":"/dataset/securityeval","name":"SecurityEval","full_name":"","num_papers_in_archive":17},{"url":"/dataset/humaneval-x","name":"HumanEval-X","full_name":"","num_papers_in_archive":16},{"url":"/dataset/juice","name":"JuICe","full_name":"JuICe Dataset","num_papers_in_archive":16},{"url":"/dataset/humaneval-et","name":"HumanEval-ET","full_name":"","num_papers_in_archive":15},{"url":"/dataset/mbpp-et","name":"MBPP-ET","full_name":"","num_papers_in_archive":15},{"url":"/dataset/llamea","name":"LLaMEA","full_name":"algorithms and experiments from the paper","num_papers_in_archive":12},{"url":"/dataset/criticbench","name":"CriticBench","full_name":"","num_papers_in_archive":10},{"url":"/dataset/mconala","name":"MCoNaLa","full_name":"Multilingual CoNaLa","num_papers_in_archive":10},{"url":"/dataset/data-science-problems","name":"Data Science Problems","full_name":"","num_papers_in_archive":7},{"url":"/dataset/humaneval-xl","name":"HumanEval-XL","full_name":"","num_papers_in_archive":6},{"url":"/dataset/lyra","name":"Lyra","full_name":"","num_papers_in_archive":6},{"url":"/dataset/shellcode-ia32","name":"Shellcode_IA32","full_name":"Shellcode_IA32","num_papers_in_archive":6},{"url":"/dataset/biocoder","name":"BioCoder","full_name":"","num_papers_in_archive":5},{"url":"/dataset/pytorrent","name":"PyTorrent","full_name":"PyTorrent","num_papers_in_archive":5},{"url":"/dataset/safim","name":"SAFIM","full_name":"Syntax-Aware Fill-In-the-Middle","num_papers_in_archive":5},{"url":"/dataset/conala-ext","name":"CoNaLa-Ext","full_name":"CoNaLa Extended With Question Text","num_papers_in_archive":4},{"url":"/dataset/dseval-leetcode","name":"DSEval-LeetCode","full_name":"","num_papers_in_archive":4},{"url":"/dataset/leetcode-hard","name":"LeetCode-Hard","full_name":"","num_papers_in_archive":4},{"url":"/dataset/mmcode","name":"MMCode","full_name":"","num_papers_in_archive":3},{"url":"/dataset/spectre-v1","name":"Spectre-v1","full_name":"","num_papers_in_archive":3},{"url":"/dataset/webapp1k-react","name":"WebApp1K-React","full_name":"","num_papers_in_archive":3},{"url":"/dataset/disl","name":"DISL","full_name":"Fueling Research with A Large Dataset of Solidity Smart Contracts","num_papers_in_archive":2},{"url":"/dataset/dseval-exercise","name":"DSEval-Exercise","full_name":"","num_papers_in_archive":2},{"url":"/dataset/dseval","name":"DSEval-Kaggle","full_name":"","num_papers_in_archive":2},{"url":"/dataset/dseval-so","name":"DSEval-SO","full_name":"","num_papers_in_archive":2},{"url":"/dataset/evil","name":"EVIL","full_name":"","num_papers_in_archive":2},{"url":"/dataset/planetarium","name":"Planetarium","full_name":"","num_papers_in_archive":2},{"url":"/dataset/rmcbench","name":"RMCBench","full_name":"","num_papers_in_archive":2},{"url":"/dataset/sltrans","name":"SLTrans","full_name":"","num_papers_in_archive":2},{"url":"/dataset/verilogeval","name":"VerilogEval","full_name":"","num_papers_in_archive":2},{"url":"/dataset/codegen4libs-dataset","name":"CodeGen4Libs Dataset","full_name":"","num_papers_in_archive":1},{"url":"/dataset/coffe","name":"COFFE","full_name":"COFFE: A Code Efficiency Benchmark for Code Generation","num_papers_in_archive":1},{"url":"/dataset/crust-bench","name":"CRUST-bench","full_name":"","num_papers_in_archive":1},{"url":"/dataset/synthetic-grade-school-math-sgsm","name":"Educational Grade School Math (EGSM)","full_name":"","num_papers_in_archive":1},{"url":"/dataset/evil-decoders","name":"EVIL-Decoders","full_name":"","num_papers_in_archive":1},{"url":"/dataset/evil-encoders","name":"EVIL-Encoders","full_name":"","num_papers_in_archive":1},{"url":"/dataset/flat-real-world-simulink-models","name":"Flat Real World Simulink Models","full_name":"","num_papers_in_archive":1},{"url":"/dataset/floco","name":"FloCo","full_name":"Flow chart Image to Code","num_papers_in_archive":1},{"url":"/dataset/genotex","name":"GenoTEX","full_name":"An LLM Agent Benchmark for Automated Gene Expression Data Analysis","num_papers_in_archive":1},{"url":"/dataset/migration-bench-java-full","name":"migration-bench-java-full","full_name":"","num_papers_in_archive":1},{"url":"/dataset/migration-bench-java-selected","name":"migration-bench-java-selected","full_name":"","num_papers_in_archive":1},{"url":"/dataset/migration-bench-java-utg","name":"migration-bench-java-utg","full_name":"","num_papers_in_archive":1},{"url":"/dataset/obscurax","name":"ObscuraX","full_name":"ObscuraX","num_papers_in_archive":1},{"url":"/dataset/openapi-code-completion","name":"OpenAPI completion refined","full_name":"","num_papers_in_archive":1},{"url":"/dataset/pecc","name":"PECC","full_name":"PECC: Problem Extraction and Coding Challenges","num_papers_in_archive":1},{"url":"/dataset/projecteval","name":"ProjectEval","full_name":"","num_papers_in_archive":1},{"url":"/dataset/res-q","name":"RES-Q","full_name":"RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale","num_papers_in_archive":1},{"url":"/dataset/soeval","name":"SOEval","full_name":"","num_papers_in_archive":1},{"url":"/dataset/taco-topics-in-algorithmic-code-generation","name":"TACO-BAAI","full_name":"Topics in Algorithmic Code generation dataset","num_papers_in_archive":1},{"url":"/dataset/tfix-s-code-patch-data","name":"TFix's Code Patches Data","full_name":"","num_papers_in_archive":1},{"url":"/dataset/turbulence","name":"Turbulence","full_name":"Turbulence","num_papers_in_archive":1},{"url":"/dataset/verified-smart-contract-code-comments","name":"Verified Smart Contract Code Comments","full_name":"","num_papers_in_archive":1},{"url":"/dataset/verified-smart-contracts","name":"Verified Smart Contracts","full_name":"","num_papers_in_archive":1},{"url":"/dataset/verireason-rtl-coder-7b-reasoning-tb","name":"Verireason-RTL-Coder_7b_reasoning_tb","full_name":"VeriReason Verilog Dataset with Reasoning, Testbench, and Simulation Results","num_papers_in_archive":1},{"url":"/dataset/verireason-rtl-coder-7b-reasoning-tb-simple","name":"Verireason-RTL-Coder_7b_reasoning_tb_simple","full_name":"Simple Problems of VeriReason Verilog Dataset with Reasoning, Testbench, and Simulation Results","num_papers_in_archive":1},{"url":"/dataset/webapp1k-duo-react","name":"WebApp1k-Duo-React","full_name":"","num_papers_in_archive":1},{"url":"/dataset/webgen-bench","name":"WebGen-Bench","full_name":"","num_papers_in_archive":1}],"subtasks":[{"url":"/task/class-level-code-generation","name":"Class-level Code Generation"},{"url":"/task/code-documentation-generation","name":"Code Documentation Generation"},{"url":"/task/code-translation","name":"Code Translation"},{"url":"/task/github-issue-resolution","name":"GitHub issue resolution"},{"url":"/task/library-oriented-code-generation","name":"Library-Oriented Code Generation"}],"parent_tasks":[],"papers":{"order":"repositories listed in the archive (desc), then date (desc); the archive holds no stars","population":"papers tagged with this task that list at least one repository in the archive","shown":30,"of":745,"tagged_in_all":1697,"items":[{"url":"/paper/llama-open-and-efficient-foundation-language-1","title":"LLaMA: Open and Efficient Foundation Language Models","date":"2023-02-27","arxiv_id":"2302.13971","repositories_listed":57,"syntology":{"n":58,"n_ran":26,"n_unverified":32,"n_pointer_only":4}},{"url":"/paper/joint-face-detection-and-alignment-using","title":"Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks","date":"2016-04-11","arxiv_id":"1604.02878","repositories_listed":42,"syntology":{"n":9,"n_ran":1,"n_unverified":8,"n_pointer_only":3}},{"url":"/paper/llama-2-open-foundation-and-fine-tuned-chat","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","date":"2023-07-18","arxiv_id":"2307.09288","repositories_listed":19,"syntology":{"n":52,"n_ran":31,"n_unverified":21,"n_pointer_only":16}},{"url":"/paper/evaluating-large-language-models-trained-on","title":"Evaluating Large Language Models Trained on Code","date":"2021-07-07","arxiv_id":"2107.03374","repositories_listed":13,"syntology":{"n":39,"n_ran":6,"n_unverified":33,"n_pointer_only":2}},{"url":"/paper/gpt-4-technical-report-1","title":"GPT-4 Technical Report","date":"2023-03-15","arxiv_id":"2303.08774","repositories_listed":11,"syntology":{"n":5,"n_ran":2,"n_unverified":3,"n_pointer_only":1}},{"url":"/paper/pix2code-generating-code-from-a-graphical","title":"pix2code: Generating Code from a Graphical User Interface Screenshot","date":"2017-05-22","arxiv_id":"1705.07962","repositories_listed":11,"syntology":null},{"url":"/paper/swe-bench-can-language-models-resolve-real","title":"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?","date":"2023-10-10","arxiv_id":"2310.06770","repositories_listed":8,"syntology":{"n":5,"n_ran":0,"n_unverified":5,"n_pointer_only":0}},{"url":"/paper/a-conversational-paradigm-for-program","title":"CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis","date":"2022-03-25","arxiv_id":"2203.13474","repositories_listed":8,"syntology":{"n":10,"n_ran":10,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/santacoder-don-t-reach-for-the-stars","title":"SantaCoder: don't reach for the stars!","date":"2023-01-09","arxiv_id":"2301.03988","repositories_listed":7,"syntology":{"n":3,"n_ran":3,"n_unverified":0,"n_pointer_only":3}},{"url":"/paper/palm-scaling-language-modeling-with-pathways-1","title":"PaLM: Scaling Language Modeling with Pathways","date":"2022-04-05","arxiv_id":"2204.02311","repositories_listed":7,"syntology":{"n":37,"n_ran":30,"n_unverified":7,"n_pointer_only":0}},{"url":"/paper/codexglue-a-machine-learning-benchmark","title":"CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation","date":"2021-02-09","arxiv_id":"2102.04664","repositories_listed":7,"syntology":{"n":1,"n_ran":1,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/structvae-tree-structured-latent-variable","title":"StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing","date":"2018-06-20","arxiv_id":"1806.07832","repositories_listed":7,"syntology":null},{"url":"/paper/mixtral-of-experts","title":"Mixtral of Experts","date":"2024-01-08","arxiv_id":"2401.04088","repositories_listed":6,"syntology":{"n":5,"n_ran":5,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/mistral-7b","title":"Mistral 7B","date":"2023-10-10","arxiv_id":"2310.06825","repositories_listed":6,"syntology":{"n":11,"n_ran":9,"n_unverified":2,"n_pointer_only":1}},{"url":"/paper/a-parallel-corpus-of-python-functions-and","title":"A parallel corpus of Python functions and documentation strings for automated code documentation and code generation","date":"2017-07-07","arxiv_id":"1707.02275","repositories_listed":6,"syntology":{"n":3,"n_ran":2,"n_unverified":1,"n_pointer_only":3}},{"url":"/paper/a-syntactic-neural-model-for-general-purpose","title":"A Syntactic Neural Model for General-Purpose Code Generation","date":"2017-04-06","arxiv_id":"1704.01696","repositories_listed":6,"syntology":{"n":3,"n_ran":3,"n_unverified":0,"n_pointer_only":3}},{"url":"/paper/codet5-identifier-aware-unified-pre-trained","title":"CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation","date":"2021-09-02","arxiv_id":"2109.00859","repositories_listed":5,"syntology":{"n":11,"n_ran":1,"n_unverified":10,"n_pointer_only":0}},{"url":"/paper/content-enhanced-bert-based-text-to-sql","title":"Content Enhanced BERT-based Text-to-SQL Generation","date":"2019-10-16","arxiv_id":"1910.07179","repositories_listed":5,"syntology":null},{"url":"/paper/qwen3-technical-report","title":"Qwen3 Technical Report","date":"2025-05-14","arxiv_id":"2505.09388","repositories_listed":4,"syntology":null},{"url":"/paper/aflow-automating-agentic-workflow-generation","title":"AFlow: Automating Agentic Workflow Generation","date":"2024-10-14","arxiv_id":"2410.10762","repositories_listed":4,"syntology":null},{"url":"/paper/bigcodebench-benchmarking-code-generation","title":"BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions","date":"2024-06-22","arxiv_id":"2406.15877","repositories_listed":4,"syntology":{"n":6,"n_ran":4,"n_unverified":2,"n_pointer_only":0}},{"url":"/paper/starcoder-2-and-the-stack-v2-the-next","title":"StarCoder 2 and The Stack v2: The Next Generation","date":"2024-02-29","arxiv_id":"2402.19173","repositories_listed":4,"syntology":null},{"url":"/paper/how-is-chatgpt-s-behavior-changing-over-time","title":"How is ChatGPT's behavior changing over time?","date":"2023-07-18","arxiv_id":"2307.09009","repositories_listed":4,"syntology":{"n":6,"n_ran":2,"n_unverified":4,"n_pointer_only":5}},{"url":"/paper/wizardcoder-empowering-code-large-language","title":"WizardCoder: Empowering Code Large Language Models with Evol-Instruct","date":"2023-06-14","arxiv_id":"2306.08568","repositories_listed":4,"syntology":{"n":7,"n_ran":3,"n_unverified":4,"n_pointer_only":1}},{"url":"/paper/starcoder-may-the-source-be-with-you","title":"StarCoder: may the source be with you!","date":"2023-05-09","arxiv_id":"2305.06161","repositories_listed":4,"syntology":{"n":2,"n_ran":2,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/training-a-helpful-and-harmless-assistant","title":"Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback","date":"2022-04-12","arxiv_id":"2204.05862","repositories_listed":4,"syntology":{"n":2,"n_ran":2,"n_unverified":0,"n_pointer_only":2}},{"url":"/paper/an-empirical-cybersecurity-evaluation-of","title":"Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions","date":"2021-08-20","arxiv_id":"2108.09293","repositories_listed":4,"syntology":null},{"url":"/paper/tranx-a-transition-based-neural-abstract","title":"TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation","date":"2018-10-05","arxiv_id":"1810.02720","repositories_listed":4,"syntology":{"n":7,"n_ran":2,"n_unverified":5,"n_pointer_only":2}},{"url":"/paper/chartgalaxy-a-dataset-for-infographic-chart","title":"ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation","date":"2025-05-24","arxiv_id":"2505.18668","repositories_listed":3,"syntology":{"n":3,"n_ran":3,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/soap-enhancing-efficiency-of-generated-code","title":"EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization","date":"2024-05-24","arxiv_id":"2405.15189","repositories_listed":3,"syntology":{"n":17,"n_ran":12,"n_unverified":5,"n_pointer_only":0}}],"syntology_records":23,"syntology_note":"a paper without a record is not a recorded non-run: it may lack an arXiv id or simply be absent from the graph layer"},"description_links":{"kept":1,"unwrapped_to_text":0,"bare_urls_linked":0,"relative_images_dropped":0,"rule":"internal links are kept only when the target slug exists in the catalog"},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per-sample execution status on synthesized fixtures ('ran N of M samples'); not a correctness claim and not a ranking signal.","status_vocabulary":{"ran_honours":"ran, honoured the contract we drafted","ran_violates":"ran, violated the contract we drafted","ran_draft_wrong":"ran; our contract draft was wrong, not the code","ran_fixture":"ran; our fixture could not drive it","ran":"ran on a synthesized input","unverified":"unverified (harvested, no recorded run)"}},"not_shown":{"libraries":"the archive has no per-task library table","trend_sparklines":"the Trend column of the benchmarks table was a rendered image; it is not in the archive","social_and_latest_sorts":"stars and social signals are not in the archive"}}