{"url":"/method/codebert","slug":"codebert","name":"CodeBERT","full_name":"CodeBERT","full_name_withheld":false,"description_markdown":"**CodeBERT** is a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. CodeBERT is developed with a [Transformer](https://paperswithcode.com/method/transformer)-based neural architecture, and is trained with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables the utilization of both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2002.08155v4","title":"CodeBERT: A Pre-Trained Model for Programming and Natural Languages","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":66,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair","date":"2025-06-05","arxiv_id":"2506.04987","n_code_links":0,"syntology":null},{"paper":null,"title":"LoRACode: LoRA Adapters for Code Embeddings","date":"2025-03-07","arxiv_id":"2503.05315","n_code_links":0,"syntology":null},{"paper":"/paper/poster-long-php-webshell-files-detection","title":"Poster: Long PHP webshell files detection based on sliding window attention","date":"2025-02-26","arxiv_id":"2502.19257","n_code_links":1,"syntology":null},{"paper":null,"title":"Less is More: On the Importance of Data Quality for Unit Test Generation","date":"2025-02-20","arxiv_id":"2502.14212","n_code_links":0,"syntology":null},{"paper":null,"title":"LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights","date":"2025-02-10","arxiv_id":"2502.07049","n_code_links":0,"syntology":null},{"paper":"/paper/repository-level-code-search-with-neural","title":"Repository-level Code Search with Neural Retrieval Methods","date":"2025-02-10","arxiv_id":"2502.07067","n_code_links":1,"syntology":null},{"paper":"/paper/how-to-select-pre-trained-code-models-for","title":"How to Select Pre-Trained Code Models for Reuse? A Learning Perspective","date":"2025-01-07","arxiv_id":"2501.03783","n_code_links":1,"syntology":null},{"paper":null,"title":"EnStack: An Ensemble Stacking Framework of Large Language Models for Enhanced Vulnerability Detection in Source Code","date":"2024-11-25","arxiv_id":"2411.16561","n_code_links":0,"syntology":null},{"paper":"/paper/codesam-source-code-representation-learning","title":"CodeSAM: Source Code Representation Learning by Infusing Self-Attention with Multi-Code-View Graphs","date":"2024-11-21","arxiv_id":"2411.14611","n_code_links":1,"syntology":null},{"paper":null,"title":"Automated Vulnerability Detection Using Deep Learning Technique","date":"2024-10-29","arxiv_id":"2410.21968","n_code_links":0,"syntology":null},{"paper":null,"title":"LecPrompt: A Prompt-based Approach for Logical Error Correction with CodeBERT","date":"2024-10-10","arxiv_id":"2410.08241","n_code_links":0,"syntology":null},{"paper":null,"title":"CLNX: Bridging Code and Natural Language for C/C++ Vulnerability-Contributing Commits Identification","date":"2024-09-11","arxiv_id":"2409.07407","n_code_links":0,"syntology":null},{"paper":null,"title":"Unlearning Trojans in Large Language Models: A Comparison Between Natural Language and Source Code","date":"2024-08-22","arxiv_id":"2408.12416","n_code_links":0,"syntology":null},{"paper":null,"title":"CodeMirage: Hallucinations in Code Generated by Large Language Models","date":"2024-08-14","arxiv_id":"2408.08333","n_code_links":0,"syntology":null},{"paper":null,"title":"Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation","date":"2024-07-29","arxiv_id":"2407.19619","n_code_links":0,"syntology":null},{"paper":"/paper/advanced-detection-of-source-code-clones-via","title":"Advanced Detection of Source Code Clones via an Ensemble of Unsupervised Similarity Measures","date":"2024-05-03","arxiv_id":"2405.02095","n_code_links":1,"syntology":null},{"paper":"/paper/software-vulnerability-prediction-in-low","title":"Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPT","date":"2024-04-26","arxiv_id":"2404.17110","n_code_links":1,"syntology":null},{"paper":"/paper/codeprompt-improving-source-code-related","title":"Enhancing Source Code Classification Effectiveness via Prompt Learning Incorporating Knowledge Features","date":"2024-01-10","arxiv_id":"2401.05544","n_code_links":1,"syntology":null},{"paper":null,"title":"Source Code is a Graph, Not a Sequence: A Cross-Lingual Perspective on Code Clone Detection","date":"2023-12-27","arxiv_id":"2312.16488","n_code_links":0,"syntology":null},{"paper":null,"title":"Naturalness of Attention: Revisiting Attention in Code Language Models","date":"2023-11-22","arxiv_id":"2311.13508","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning Defect Prediction from Unrealistic Data","date":"2023-11-02","arxiv_id":"2311.00931","n_code_links":0,"syntology":null},{"paper":"/paper/gem5pred-predictive-approaches-for-gem5","title":"Gem5Pred: Predictive Approaches For Gem5 Simulation Time","date":"2023-10-10","arxiv_id":"2310.06290","n_code_links":0,"syntology":null},{"paper":null,"title":"A Comparative Study of Filters and Deep Learning Models to predict Diabetic Retinopathy","date":"2023-09-26","arxiv_id":"2309.15216","n_code_links":0,"syntology":null},{"paper":null,"title":"XGV-BERT: Leveraging Contextualized Language Model and Graph Neural Network for Efficient Software Vulnerability Detection","date":"2023-09-26","arxiv_id":"2309.14677","n_code_links":0,"syntology":null},{"paper":null,"title":"Code quality assessment using transformers","date":"2023-09-17","arxiv_id":"2309.09264","n_code_links":0,"syntology":null},{"paper":null,"title":"Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?","date":"2023-09-14","arxiv_id":"2309.07804","n_code_links":0,"syntology":null},{"paper":"/paper/chatgpt-for-software-security-exploring-the","title":"How Does Naming Affect LLMs on Code Analysis Tasks?","date":"2023-07-24","arxiv_id":"2307.12488","n_code_links":0,"syntology":null},{"paper":null,"title":"SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?","date":"2023-07-13","arxiv_id":"2307.06616","n_code_links":0,"syntology":null},{"paper":null,"title":"FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair","date":"2023-06-21","arxiv_id":"2307.00012","n_code_links":0,"syntology":null},{"paper":null,"title":"Automatic Code Summarization via ChatGPT: How Far Are We?","date":"2023-05-22","arxiv_id":"2305.12865","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/code-search","name":"Code Search","papers":12},{"task":"/task/vulnerability-detection","name":"Vulnerability Detection","papers":10},{"task":"/task/clone-detection","name":"Clone Detection","papers":8},{"task":"/task/code-summarization-1","name":"Code Summarization","papers":8},{"task":"/task/code-generation","name":"Code Generation","papers":7},{"task":"/task/language-modelling","name":"Language Modelling","papers":7},{"task":"/task/retrieval","name":"Retrieval","papers":6},{"task":"/task/bug-fixing","name":"Bug fixing","papers":3},{"task":"/task/classification-1","name":"Classification","papers":3},{"task":"/task/code-classification","name":"Code Classification","papers":3},{"task":"/task/code-completion","name":"Code Completion","papers":3},{"task":"/task/code-documentation-generation","name":"Code Documentation Generation","papers":3},{"task":"/task/code-translation","name":"Code Translation","papers":3},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":3},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":3},{"task":"/task/language-modeling","name":"Language Modeling","papers":3},{"task":"/task/large-language-model","name":"Large Language Model","papers":3},{"task":"/task/representation-learning","name":"Representation Learning","papers":3},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":3},{"task":"/task/decoder","name":"Decoder","papers":2}],"tasks_shown":20,"n_tasks":80,"usage_by_year":[{"year":"2020","papers":1},{"year":"2021","papers":12},{"year":"2022","papers":17},{"year":"2023","papers":18},{"year":"2024","papers":11},{"year":"2025","papers":7}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/codebert"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}