{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measuring-coding-challenge-competence-with","title":"Measuring Coding Challenge Competence With APPS","arxiv_id":"2105.09938","date":"2021-05-20","proceeding":null,"authors":["Dan Hendrycks","Steven Basart","Saurav Kadavath","Mantas Mazeika","Akul Arora","Ethan Guo","Collin Burns","Samir Puranik","Horace He","Dawn Song","Jacob Steinhardt"],"abstract":"While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.","url_abs":"https://arxiv.org/abs/2105.09938v3","url_pdf":"https://arxiv.org/pdf/2105.09938v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"measuring-coding-challenge-competence-with","repo_url":"https://github.com/hendrycks/apps","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"measuring-coding-challenge-competence-with","repo_url":"https://github.com/codedotal/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"measuring-coding-challenge-competence-with","repo_url":"https://github.com/ncoop57/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[{"method_slug":"gpt-neo","method_name":"GPT-Neo"}],"datasets_introduced":[{"slug":"apps","name":"APPS","full_name":"Automated Programming Progress Standard"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-generation-on-apps","task":"Code Generation","dataset":"APPS","model":"GPT-Neo 2.7B","rank_in_archive_order":14,"of":18,"metrics":{"Competition Pass@1":"0.00%","Competition Pass@1000":"11.40%","Competition Pass@5":"0.00%","Competition Pass@any":"11.40%","Interview Pass@1":"0.57%","Interview Pass@1000":"9.83%","Interview Pass@5":"0.80%","Interview Pass@any":"9.83%","Introductory Pass@1":"3.90%","Introductory Pass@1000":"27.90%","Introductory Pass@5":"5.50%","Introductory Pass@any":"27.90%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.09938","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.09938"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hendrycks/apps","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ncoop57/gpt-code-clippy","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codedotal/gpt-code-clippy","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":9},"by_repo_kind":{"official":{"samples":11,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f05983c38acfa250","entry":"find_indentation","repo":"hendrycks/apps","repo_kind":"official","path":"eval/reindent.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/eval/reindent.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f05983c38acfa250"}},{"code_sha256_prefix":"27f23c533d62061a","entry":"replace_inline_tabs","repo":"hendrycks/apps","repo_kind":"official","path":"eval/reindent.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/eval/reindent.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27f23c533d62061a"}},{"code_sha256_prefix":"7f74025d94c22cd6","entry":"batch_bart_task","repo":"hendrycks/apps","repo_kind":"official","path":"train/dataset_lm/util.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/train/dataset_lm/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f74025d94c22cd6"}},{"code_sha256_prefix":"991b30911bfd77a2","entry":"batch_gpt_task","repo":"hendrycks/apps","repo_kind":"official","path":"train/dataset_lm/util.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/train/dataset_lm/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"991b30911bfd77a2"}},{"code_sha256_prefix":"72342b1d3d9f4231","entry":"create_split","repo":"hendrycks/apps","repo_kind":"official","path":"train/apps_create_split.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/train/apps_create_split.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72342b1d3d9f4231"}},{"code_sha256_prefix":"6b434bae76a3379f","entry":"dummy_gpt_task","repo":"hendrycks/apps","repo_kind":"official","path":"train/dataset_lm/util.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/train/dataset_lm/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b434bae76a3379f"}},{"code_sha256_prefix":"ed8331339b4de97c","entry":"get_question","repo":"hendrycks/apps","repo_kind":"official","path":"eval/testing_util.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/eval/testing_util.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed8331339b4de97c"}},{"code_sha256_prefix":"0c83bb0f43b8dc57","entry":"get_solutions","repo":"hendrycks/apps","repo_kind":"official","path":"eval/testing_util.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/eval/testing_util.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c83bb0f43b8dc57"}},{"code_sha256_prefix":"8bbb265e59e23293","entry":"get_valid_problems","repo":"hendrycks/apps","repo_kind":"official","path":"eval/testing_util.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/eval/testing_util.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8bbb265e59e23293"}},{"code_sha256_prefix":"de2e007d84fab02c","entry":"rewrite_logs","repo":"hendrycks/apps","repo_kind":"official","path":"train/CustomTensorboardCallback.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/train/CustomTensorboardCallback.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de2e007d84fab02c"}},{"code_sha256_prefix":"9c56401b1df9832c","entry":"sample_gpt_task","repo":"hendrycks/apps","repo_kind":"official","path":"train/dataset_apps/APPSBaseDataset.py","file_url":"https://github.com/hendrycks/apps/blob/HEAD/train/dataset_apps/APPSBaseDataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c56401b1df9832c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}