{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-large-language-models-trained-on","title":"Evaluating Large Language Models Trained on Code","arxiv_id":"2107.03374","date":"2021-07-07","proceeding":null,"authors":["Mark Chen","Jerry Tworek","Heewoo Jun","Qiming Yuan","Henrique Ponde de Oliveira Pinto","Jared Kaplan","Harri Edwards","Yuri Burda","Nicholas Joseph","Greg Brockman","Alex Ray","Raul Puri","Gretchen Krueger","Michael Petrov","Heidy Khlaaf","Girish Sastry","Pamela Mishkin","Brooke Chan","Scott Gray","Nick Ryder","Mikhail Pavlov","Alethea Power","Lukasz Kaiser","Mohammad Bavarian","Clemens Winter","Philippe Tillet","Felipe Petroski Such","Dave Cummings","Matthias Plappert","Fotios Chantzis","Elizabeth Barnes","Ariel Herbert-Voss","William Hebgen Guss","Alex Nichol","Alex Paino","Nikolas Tezak","Jie Tang","Igor Babuschkin","Suchir Balaji","Shantanu Jain","William Saunders","Christopher Hesse","Andrew N. Carr","Jan Leike","Josh Achiam","Vedant Misra","Evan Morikawa","Alec Radford","Matthew Knight","Miles Brundage","Mira Murati","Katie Mayer","Peter Welinder","Bob McGrew","Dario Amodei","Sam McCandlish","Ilya Sutskever","Wojciech Zaremba"],"abstract":"We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.","url_abs":"https://arxiv.org/abs/2107.03374v2","url_pdf":"https://arxiv.org/pdf/2107.03374v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/openai/human-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/2796gaurav/human-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/THUDM/CodeGeeX","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/codedotal/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/codefuse-ai/codefuse-evaluation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/fsoft-ai4code/codecapybara","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/glouppe/info8010-deep-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/microsoft/PythonProgrammingPuzzles","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/ncoop57/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/superli3/CYRMPR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/superli3/codenavi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"evaluating-large-language-models-trained-on","repo_url":"https://github.com/vhellendoorn/code-lms","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[{"slug":"humaneval","name":"HumanEval","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-generation-on-apps","task":"Code Generation","dataset":"APPS","model":"Codex 12B (Raw)","rank_in_archive_order":12,"of":18,"metrics":{"Competition Pass@1":"0.50%","Competition Pass@1000":"13.51%","Competition Pass@5":"1.00%","Competition Pass@any":"13.51%","Interview Pass@1":"1.00%","Interview Pass@1000":"13.15%","Interview Pass@5":"1.73%","Interview Pass@any":"13.15%","Introductory Pass@1":"5.60%","Introductory Pass@1000":"35.20%","Introductory Pass@5":"9.20%","Introductory Pass@any":"35.20%"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-bbh-alg","task":"Multi-task Language Understanding","dataset":"BBH-alg","model":"code-davinci-002 175B (CoT)","rank_in_archive_order":1,"of":7,"metrics":{"Average (%)":"73.9"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-bbh-nlp","task":"Multi-task Language Understanding","dataset":"BBH-nlp","model":"code-davinci-002 175B (CoT)","rank_in_archive_order":9,"of":15,"metrics":{"Average (%)":"73.5"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2107.03374","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.03374"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/human-eval","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codefuse-ai/codefuse-evaluation","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/2796gaurav/human-eval","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ncoop57/gpt-code-clippy","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/superli3/codenavi","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codedotal/gpt-code-clippy","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fsoft-ai4code/codecapybara","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/glouppe/info8010-deep-learning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/THUDM/CodeGeeX","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vhellendoorn/code-lms","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/superli3/CYRMPR","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/PythonProgrammingPuzzles","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/code-align-evals-data","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/codealign-evals-data","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran_draft_wrong":2,"ran":3,"ran_honours":1,"unverified":33},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1},"listed":{"samples":22,"ran":4,"repositories":4},"found_in_text":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"4ed7a6255ffcd3e2","entry":"chdir","repo":"openai/human-eval","repo_kind":"official","path":"human_eval/execution.py","file_url":"https://github.com/openai/human-eval/blob/HEAD/human_eval/execution.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ed7a6255ffcd3e2"}},{"code_sha256_prefix":"1214e44aeb74f364","entry":"create_tempdir","repo":"openai/human-eval","repo_kind":"official","path":"human_eval/execution.py","file_url":"https://github.com/openai/human-eval/blob/HEAD/human_eval/execution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1214e44aeb74f364"}},{"code_sha256_prefix":"e080db2eaa81fdb3","entry":"estimate_pass_at_k","repo":"2796gaurav/human-eval","repo_kind":"listed","path":"human_eval/evaluation.py","file_url":"https://github.com/2796gaurav/human-eval/blob/HEAD/human_eval/evaluation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"e080db2eaa81fdb3"}},{"code_sha256_prefix":"f05983c38acfa250","entry":"find_indentation","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"reindent.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/reindent.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f05983c38acfa250"}},{"code_sha256_prefix":"2a310ceef0a63386","entry":"pass_at_k","repo":"microsoft/PythonProgrammingPuzzles","repo_kind":"listed","path":"solvers/codex/run_codex_experiments.py","file_url":"https://github.com/microsoft/PythonProgrammingPuzzles/blob/HEAD/solvers/codex/run_codex_experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a310ceef0a63386"}},{"code_sha256_prefix":"27f23c533d62061a","entry":"replace_inline_tabs","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"reindent.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/reindent.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27f23c533d62061a"}},{"code_sha256_prefix":"9f6bcf31e58add54","entry":"add","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/add.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/add.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f6bcf31e58add54"}},{"code_sha256_prefix":"2cc1153eb2697bbf","entry":"add_elements","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/add_elements.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/add_elements.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cc1153eb2697bbf"}},{"code_sha256_prefix":"5e866b54fbf03be3","entry":"all_prefixes","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/all_prefixes.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/all_prefixes.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e866b54fbf03be3"}},{"code_sha256_prefix":"1a25d7256e5f4bb5","entry":"anti_shuffle","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/anti_shuffle.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/anti_shuffle.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a25d7256e5f4bb5"}},{"code_sha256_prefix":"c3efafd6833cfa4b","entry":"any_int","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/any_int.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/any_int.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3efafd6833cfa4b"}},{"code_sha256_prefix":"db66d9a5e3e1895b","entry":"below_threshold","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/below_threshold.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/below_threshold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db66d9a5e3e1895b"}},{"code_sha256_prefix":"92fc2114d3e87e85","entry":"below_zero","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/below_zero.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/below_zero.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92fc2114d3e87e85"}},{"code_sha256_prefix":"2ef1476740b6e3dd","entry":"bf","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/bf.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/bf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ef1476740b6e3dd"}},{"code_sha256_prefix":"f63bbaac0df6c6fe","entry":"by_length","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/by_length.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/by_length.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f63bbaac0df6c6fe"}},{"code_sha256_prefix":"d3bd061f24d302d4","entry":"can_arange","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/can_arange.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/can_arange.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3bd061f24d302d4"}},{"code_sha256_prefix":"968da8e551331977","entry":"car_race_collision","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/car_race_collision.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/car_race_collision.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"968da8e551331977"}},{"code_sha256_prefix":"63c649d1e83343f6","entry":"change_base","repo":"openai/code-align-evals-data","repo_kind":"found_in_text","path":"bad-solutions/change_base.py","file_url":"https://github.com/openai/code-align-evals-data/blob/HEAD/bad-solutions/change_base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63c649d1e83343f6"}},{"code_sha256_prefix":"87135198de7ec14b","entry":"check_correctness","repo":"openai/human-eval","repo_kind":"official","path":"human_eval/execution.py","file_url":"https://github.com/openai/human-eval/blob/HEAD/human_eval/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87135198de7ec14b"}},{"code_sha256_prefix":"5bcc0e96c2ae8311","entry":"generate_prompt","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"evaluation/apps_utils/generate_gpt_codes.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/evaluation/apps_utils/generate_gpt_codes.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bcc0e96c2ae8311"}},{"code_sha256_prefix":"73c40ea3df62ea10","entry":"generate_prompt","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"data_processing/apps.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/data_processing/apps.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73c40ea3df62ea10"}},{"code_sha256_prefix":"32636ddfa64b46bc","entry":"get_output_and_retry","repo":"my-other-github-account/llm-humaneval-benchmarks","repo_kind":"listed","path":"launch_pod.py","file_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks/blob/HEAD/launch_pod.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32636ddfa64b46bc"}},{"code_sha256_prefix":"ed8331339b4de97c","entry":"get_question","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"evaluation/apps_eval_util.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/evaluation/apps_eval_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed8331339b4de97c"}},{"code_sha256_prefix":"0c83bb0f43b8dc57","entry":"get_solutions","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"evaluation/apps_eval_util.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/evaluation/apps_eval_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c83bb0f43b8dc57"}},{"code_sha256_prefix":"8bbb265e59e23293","entry":"get_valid_problems","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"evaluation/apps_eval_util.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/evaluation/apps_eval_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8bbb265e59e23293"}},{"code_sha256_prefix":"257503e1cf9966ff","entry":"is_digit","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"data_processing/download_license_info.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/data_processing/download_license_info.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"257503e1cf9966ff"}},{"code_sha256_prefix":"8339ab6beb446371","entry":"launch_pod","repo":"my-other-github-account/llm-humaneval-benchmarks","repo_kind":"listed","path":"launch_pod.py","file_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks/blob/HEAD/launch_pod.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8339ab6beb446371"}},{"code_sha256_prefix":"d3d6eed5a7c21746","entry":"read_problems","repo":"2796gaurav/human-eval","repo_kind":"listed","path":"human_eval/data.py","file_url":"https://github.com/2796gaurav/human-eval/blob/HEAD/human_eval/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"d3d6eed5a7c21746"}},{"code_sha256_prefix":"b6404d68982b4e0b","entry":"reindent_code","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"data_processing/apps.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/data_processing/apps.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b6404d68982b4e0b"}},{"code_sha256_prefix":"53e4784adcaf6602","entry":"reliability_guard","repo":"openai/human-eval","repo_kind":"official","path":"human_eval/execution.py","file_url":"https://github.com/openai/human-eval/blob/HEAD/human_eval/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53e4784adcaf6602"}},{"code_sha256_prefix":"aa03c033d83047b0","entry":"run","repo":"microsoft/PythonProgrammingPuzzles","repo_kind":"listed","path":"solvers/codex/run_codex_experiments.py","file_url":"https://github.com/microsoft/PythonProgrammingPuzzles/blob/HEAD/solvers/codex/run_codex_experiments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa03c033d83047b0"}},{"code_sha256_prefix":"de45bd615801ae81","entry":"run","repo":"my-other-github-account/llm-humaneval-benchmarks","repo_kind":"listed","path":"benchmark_utils.py","file_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks/blob/HEAD/benchmark_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de45bd615801ae81"}},{"code_sha256_prefix":"f7666066f684b48f","entry":"run_async","repo":"my-other-github-account/llm-humaneval-benchmarks","repo_kind":"listed","path":"benchmark_utils.py","file_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks/blob/HEAD/benchmark_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7666066f684b48f"}},{"code_sha256_prefix":"4d5e8113a82b291c","entry":"run_sync","repo":"my-other-github-account/llm-humaneval-benchmarks","repo_kind":"listed","path":"benchmark_utils.py","file_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks/blob/HEAD/benchmark_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d5e8113a82b291c"}},{"code_sha256_prefix":"d27e3a9c64e09294","entry":"split_into_chunks","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"data_processing/download_license_info.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/data_processing/download_license_info.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d27e3a9c64e09294"}},{"code_sha256_prefix":"5263f985bd2e84c0","entry":"start_server","repo":"my-other-github-account/llm-humaneval-benchmarks","repo_kind":"listed","path":"benchmark_manager.py","file_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks/blob/HEAD/benchmark_manager.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5263f985bd2e84c0"}},{"code_sha256_prefix":"2e22b41539e9cf87","entry":"timeout","repo":"codedotal/gpt-code-clippy","repo_kind":"listed","path":"data_processing/download_license_info.py","file_url":"https://github.com/codedotal/gpt-code-clippy/blob/HEAD/data_processing/download_license_info.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e22b41539e9cf87"}},{"code_sha256_prefix":"3c91bdab594f03c5","entry":"unsafe_execute","repo":"openai/human-eval","repo_kind":"official","path":"human_eval/execution.py","file_url":"https://github.com/openai/human-eval/blob/HEAD/human_eval/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c91bdab594f03c5"}},{"code_sha256_prefix":"8d76e5a80e6c961c","entry":"wait_for_pod","repo":"my-other-github-account/llm-humaneval-benchmarks","repo_kind":"listed","path":"launch_pod.py","file_url":"https://github.com/my-other-github-account/llm-humaneval-benchmarks/blob/HEAD/launch_pod.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d76e5a80e6c961c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}