{"url":"/dataset/codexglue","name":"CodeXGLUE","full_name":null,"description_markdown":"CodeXGLUE is a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios:\r\n\r\n- code-code (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation)\r\n- text-code (natural language code search, text-to-code generation)\r\n- code-text (code summarization)\r\n- text-text (documentation translation)\r\n\r\nA brief summary of CodeXGLUE is provided in the figure, including tasks, datasets, language, sizes in various states, baseline systems, providers, and short definitions of each task. Datasets highlighted in BLUE are newly introduced.\r\n\r\nImage source: [https://github.com/microsoft/CodeXGLUE](https://github.com/microsoft/CodeXGLUE)","description_withheld":null,"homepage":"https://github.com/microsoft/CodeXGLUE","introduced_date":"2021-02-09","introduced_date_note":null,"introduced_by":{"paper":"/paper/codexglue-a-machine-learning-benchmark","title":"CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation","first_author":"Shuai Lu","url":null},"license":{"name":"Computational Use of Data Agreement (C-UDA) License","url":"https://github.com/microsoft/CodeXGLUE#license"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Code Generation","url":"/task/code-generation","datasets_with_task":"/datasets/task/code-generation"},{"name":"Code Search","url":"/task/code-search","datasets_with_task":"/datasets/task/code-search"},{"name":"Code Completion","url":"/task/code-completion","datasets_with_task":"/datasets/task/code-completion"},{"name":"Defect Detection","url":"/task/defect-detection","datasets_with_task":"/datasets/task/defect-detection"},{"name":"Document Translation","url":"/task/document-translation","datasets_with_task":"/datasets/task/document-translation"},{"name":"Clone Detection","url":"/task/clone-detection","datasets_with_task":"/datasets/task/clone-detection"},{"name":"Cloze Test","url":"/task/cloze-test","datasets_with_task":"/datasets/task/cloze-test"},{"name":"Code Translation","url":"/task/code-translation","datasets_with_task":"/datasets/task/code-translation"},{"name":"Text-to-Code Generation","url":"/task/text-to-code-generation","datasets_with_task":"/datasets/task/text-to-code-generation"},{"name":"Code Repair","url":"/task/code-repair","datasets_with_task":"/datasets/task/code-repair"},{"name":"Code Summarization","url":"/task/code-summarization-1","datasets_with_task":"/datasets/task/code-summarization-1"}],"languages":[],"variants":["CodeXGLUE - WebQueryTest","CodeXGLUE - PY150","CodeXGLUE - POJ-104","CodeXGLUE - Microsoft Docs","CodeXGLUE - Github Java Corpus","CodeXGLUE - Devign","CodeXGLUE - CT-maxmin","CodeXGLUE - CT-all","CodeXGLUE - CONCODE","CodeXGLUE - CodeTrans","CodeXGLUE - CodeSearchNet","CodeXGLUE - Bugs2Fix","CodeXGLUE - AdvTest","CodeXGLUE - BigCloneBench","CodeXGLUE"],"data_loaders":[],"num_papers_in_archive":205,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/code-completion-on-codexglue-github-java","task":"Code Completion","dataset_variant":"CodeXGLUE - Github Java Corpus","rows":3,"metrics":["Accuracy (token-level)","EM (line-level)","Edit Sim (line-level)"],"first_row_in_archive_order":{"model":"CodeGPT-adapted","paper":"/paper/codexglue-a-machine-learning-benchmark","metrics":{"Accuracy (token-level)":"77.13","EM (line-level)":"26.43","Edit Sim (line-level)":"63.03"},"code_links":[{"title":"microsoft/CodeXGLUE","url":"https://github.com/microsoft/CodeXGLUE"},{"title":"facebookresearch/CodeGen","url":"https://github.com/facebookresearch/CodeGen"},{"title":"sberbank-ai/fusion_brain_aij2021","url":"https://github.com/sberbank-ai/fusion_brain_aij2021"},{"title":"kilimanj4r0/code-summarization-beyond-function-level","url":"https://github.com/kilimanj4r0/code-summarization-beyond-function-level"},{"title":"yueyuel/programgen-lms-reliability","url":"https://github.com/yueyuel/programgen-lms-reliability"},{"title":"Avmb/semantic_neq_game","url":"https://github.com/Avmb/semantic_neq_game"},{"title":"deeplearnxmu/unigencoder","url":"https://github.com/deeplearnxmu/unigencoder"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/code-completion-on-codexglue-py150","task":"Code Completion","dataset_variant":"CodeXGLUE - PY150","rows":3,"metrics":["Accuracy (token-level)","EM (line-level)","Edit Sim (line-level)"],"first_row_in_archive_order":{"model":"CodeGPT-adapted","paper":"/paper/codexglue-a-machine-learning-benchmark","metrics":{"Accuracy (token-level)":"75.11","EM (line-level)":"39.65","Edit Sim (line-level)":"69.84"},"code_links":[{"title":"microsoft/CodeXGLUE","url":"https://github.com/microsoft/CodeXGLUE"},{"title":"facebookresearch/CodeGen","url":"https://github.com/facebookresearch/CodeGen"},{"title":"sberbank-ai/fusion_brain_aij2021","url":"https://github.com/sberbank-ai/fusion_brain_aij2021"},{"title":"kilimanj4r0/code-summarization-beyond-function-level","url":"https://github.com/kilimanj4r0/code-summarization-beyond-function-level"},{"title":"yueyuel/programgen-lms-reliability","url":"https://github.com/yueyuel/programgen-lms-reliability"},{"title":"Avmb/semantic_neq_game","url":"https://github.com/Avmb/semantic_neq_game"},{"title":"deeplearnxmu/unigencoder","url":"https://github.com/deeplearnxmu/unigencoder"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/code-search-on-codexglue-advtest","task":"Code Search","dataset_variant":"CodeXGLUE - AdvTest","rows":3,"metrics":["MRR"],"first_row_in_archive_order":{"model":"CodeT5+ 770M","paper":"/paper/codet5-open-code-large-language-models-for","metrics":{"MRR":"44.7"},"code_links":[{"title":"salesforce/codet5","url":"https://github.com/salesforce/codet5"},{"title":"leiluk1/codesearcher","url":"https://github.com/leiluk1/codesearcher"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/code-generation-on-codexglue-codesearchnet","task":"Code Generation","dataset_variant":"CodeXGLUE - CodeSearchNet","rows":2,"metrics":["Java/EM","Python/EM","Java/BLEU","Python/BLEU","Java/CodeBLEU","Python/CodeBLEU"],"first_row_in_archive_order":{"model":"Redcoder-ext","paper":"/paper/retrieval-augmented-code-generation-and","metrics":{"Java/BLEU":"28.98","Java/CodeBLEU":"33.18","Java/EM":"10.21","Python/BLEU":"24.43","Python/CodeBLEU":"30.21","Python/EM":"9.61"},"code_links":[{"title":"kagnlp/CodeGenerator","url":"https://github.com/kagnlp/CodeGenerator"},{"title":"rizwan09/redcoder","url":"https://github.com/rizwan09/redcoder"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/code-repair-on-codexglue-bugs2fix","task":"Code Repair","dataset_variant":"CodeXGLUE - Bugs2Fix","rows":2,"metrics":["Accuracy (medium)","Accuracy (small)","BLEU (medium)","BLEU (small)","CodeBLEU (medium)","CodeBLEU (small)"],"first_row_in_archive_order":{"model":"NSEdit","paper":"/paper/fix-bugs-with-transformer-through-a-neural-1","metrics":{"Accuracy (medium)":"13.87","Accuracy (small)":"24.04"},"code_links":[]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/code-translation-on-codexglue-codetrans","task":"Code Translation","dataset_variant":"CodeXGLUE - CodeTrans","rows":2,"metrics":["Accuracy (C#→Java)","Accuracy (Java→C#)","BLEU (C#→Java)","BLEU (Java→C#)","CodeBLEU (C#→Java)","CodeBLEU (Java→C#)"],"first_row_in_archive_order":{"model":"CodeT5","paper":"/paper/codet5-identifier-aware-unified-pre-trained","metrics":{"Accuracy (C#→Java)":"66.90","Accuracy (Java→C#)":"65.90","BLEU (C#→Java)":"79.87","BLEU (Java→C#)":"84.03"},"code_links":[{"title":"salesforce/codet5","url":"https://github.com/salesforce/codet5"},{"title":"salesforce/coderl","url":"https://github.com/salesforce/coderl"},{"title":"awsm-research/vulrepair","url":"https://github.com/awsm-research/vulrepair"},{"title":"jetbrains-research/commit_message_generation","url":"https://github.com/jetbrains-research/commit_message_generation"},{"title":"fewshotcdcs/cdcs","url":"https://github.com/fewshotcdcs/cdcs"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/text-to-code-generation-on-codexglue-concode","task":"Text-to-Code Generation","dataset_variant":"CodeXGLUE - CONCODE","rows":2,"metrics":["BLEU","CodeBLEU","EM"],"first_row_in_archive_order":{"model":"CodeT5","paper":"/paper/codet5-identifier-aware-unified-pre-trained","metrics":{"BLEU":"41.48","CodeBLEU":"44.10","EM":"22.70"},"code_links":[{"title":"salesforce/codet5","url":"https://github.com/salesforce/codet5"},{"title":"salesforce/coderl","url":"https://github.com/salesforce/coderl"},{"title":"awsm-research/vulrepair","url":"https://github.com/awsm-research/vulrepair"},{"title":"jetbrains-research/commit_message_generation","url":"https://github.com/jetbrains-research/commit_message_generation"},{"title":"fewshotcdcs/cdcs","url":"https://github.com/fewshotcdcs/cdcs"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/cloze-test-on-codexglue-ct-all","task":"Cloze Test","dataset_variant":"CodeXGLUE - CT-all","rows":1,"metrics":["Go","JS","Java","PHP","Python","Ruby"],"first_row_in_archive_order":{"model":"CodeBERT(MLM)","paper":"/paper/codexglue-a-machine-learning-benchmark","metrics":{"Go":"83.31","JS":"81.77","Java":"80.63","PHP":"85.05","Python":"87.21","Ruby":"80.17"},"code_links":[{"title":"microsoft/CodeXGLUE","url":"https://github.com/microsoft/CodeXGLUE"},{"title":"facebookresearch/CodeGen","url":"https://github.com/facebookresearch/CodeGen"},{"title":"sberbank-ai/fusion_brain_aij2021","url":"https://github.com/sberbank-ai/fusion_brain_aij2021"},{"title":"kilimanj4r0/code-summarization-beyond-function-level","url":"https://github.com/kilimanj4r0/code-summarization-beyond-function-level"},{"title":"yueyuel/programgen-lms-reliability","url":"https://github.com/yueyuel/programgen-lms-reliability"},{"title":"Avmb/semantic_neq_game","url":"https://github.com/Avmb/semantic_neq_game"},{"title":"deeplearnxmu/unigencoder","url":"https://github.com/deeplearnxmu/unigencoder"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/cloze-test-on-codexglue-ct-maxmin","task":"Cloze Test","dataset_variant":"CodeXGLUE - CT-maxmin","rows":1,"metrics":["Go","JS","Java","PHP","Python","Ruby"],"first_row_in_archive_order":{"model":"CodeBERT(MLM)","paper":"/paper/codexglue-a-machine-learning-benchmark","metrics":{"Go":"90.79","JS":"86.4","Java":"90.46","PHP":"88.21","Python":"82.2","Ruby":"86.84"},"code_links":[{"title":"microsoft/CodeXGLUE","url":"https://github.com/microsoft/CodeXGLUE"},{"title":"facebookresearch/CodeGen","url":"https://github.com/facebookresearch/CodeGen"},{"title":"sberbank-ai/fusion_brain_aij2021","url":"https://github.com/sberbank-ai/fusion_brain_aij2021"},{"title":"kilimanj4r0/code-summarization-beyond-function-level","url":"https://github.com/kilimanj4r0/code-summarization-beyond-function-level"},{"title":"yueyuel/programgen-lms-reliability","url":"https://github.com/yueyuel/programgen-lms-reliability"},{"title":"Avmb/semantic_neq_game","url":"https://github.com/Avmb/semantic_neq_game"},{"title":"deeplearnxmu/unigencoder","url":"https://github.com/deeplearnxmu/unigencoder"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/code-search-on-codexglue-webquerytest","task":"Code Search","dataset_variant":"CodeXGLUE - WebQueryTest","rows":1,"metrics":["Accuracy","F1"],"first_row_in_archive_order":{"model":"CodeBERT","paper":"/paper/codexglue-a-machine-learning-benchmark","metrics":{"Accuracy":"47.8","F1":"58.95"},"code_links":[{"title":"microsoft/CodeXGLUE","url":"https://github.com/microsoft/CodeXGLUE"},{"title":"facebookresearch/CodeGen","url":"https://github.com/facebookresearch/CodeGen"},{"title":"sberbank-ai/fusion_brain_aij2021","url":"https://github.com/sberbank-ai/fusion_brain_aij2021"},{"title":"kilimanj4r0/code-summarization-beyond-function-level","url":"https://github.com/kilimanj4r0/code-summarization-beyond-function-level"},{"title":"yueyuel/programgen-lms-reliability","url":"https://github.com/yueyuel/programgen-lms-reliability"},{"title":"Avmb/semantic_neq_game","url":"https://github.com/Avmb/semantic_neq_game"},{"title":"deeplearnxmu/unigencoder","url":"https://github.com/deeplearnxmu/unigencoder"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/codet5-open-code-large-language-models-for","title":"CodeT5+: Open Code Large Language Models for Code Understanding and Generation","date":"2023-05-13","rows_on_this_dataset":6,"code_links":2,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":4,"samples_ran":3,"samples_unverified":1,"pointer_only_for_licence":0,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/fix-bugs-with-transformer-through-a-neural-1","title":"Fix Bugs with Transformer through a Neural-Symbolic Edit Grammar","date":"2022-04-13","rows_on_this_dataset":1,"code_links":0,"syntology":null},{"paper":"/paper/gap-gen-guided-automatic-python-code","title":"GAP-Gen: Guided Automatic Python Code Generation","date":"2022-01-19","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/codet5-identifier-aware-unified-pre-trained","title":"CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation","date":"2021-09-02","rows_on_this_dataset":2,"code_links":5,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":11,"samples_ran":1,"samples_unverified":10,"pointer_only_for_licence":0,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/retrieval-augmented-code-generation-and","title":"Retrieval Augmented Code Generation and Summarization","date":"2021-08-26","rows_on_this_dataset":1,"code_links":2,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":3,"samples_ran":2,"samples_unverified":1,"pointer_only_for_licence":3,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/codexglue-a-machine-learning-benchmark","title":"CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation","date":"2021-02-09","rows_on_this_dataset":9,"code_links":7,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":1,"samples_ran":1,"samples_unverified":0,"pointer_only_for_licence":0,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}}],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":4,"samples_harvested":19,"samples_ran":7,"samples_unverified":12,"pointer_only_for_licence":3,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}