{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/debertav3-improving-deberta-using-electra","title":"DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing","arxiv_id":"2111.09543","date":"2021-11-18","proceeding":null,"authors":["Pengcheng He","Jianfeng Gao","Weizhu Chen"],"abstract":"This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the \"tug-of-war\" dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.","url_abs":"https://arxiv.org/abs/2111.09543v4","url_pdf":"https://arxiv.org/pdf/2111.09543v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"debertav3-improving-deberta-using-electra","repo_url":"https://github.com/microsoft/DeBERTa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"debertav3-improving-deberta-using-electra","repo_url":"https://github.com/dashenzi721/hra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"debertav3-improving-deberta-using-electra","repo_url":"https://github.com/stareru/csqa_debertav3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"xlm-r","task_name":"XLM-R"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"deberta","method_name":"DeBERTa"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"electra","method_name":"ELECTRA"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"},{"method_slug":"xlm-r","method_name":"XLM-R"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-inference-on-mrpc","task":"Natural Language Inference","dataset":"MRPC","model":"DeBERTaV3large","rank_in_archive_order":1,"of":1,"metrics":{"Acc":"92.2"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-qnli","task":"Natural Language Inference","dataset":"QNLI","model":"DeBERTaV3large","rank_in_archive_order":8,"of":43,"metrics":{"Accuracy":"96%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"DeBERTaV3large","rank_in_archive_order":7,"of":90,"metrics":{"Accuracy":"92.7%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-swag","task":"Question Answering","dataset":"SWAG","model":"DeBERTaV3large","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"93.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.09543","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.09543"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/DeBERTa","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dashenzi721/hra","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stareru/csqa_debertav3","reach":{"status":"ok"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ac8689e2d4eb4c75","entry":"build_relative_position","repo":"microsoft/DeBERTa","repo_kind":"official","path":"DeBERTa/deberta/da_utils.py","file_url":"https://github.com/microsoft/DeBERTa/blob/HEAD/DeBERTa/deberta/da_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac8689e2d4eb4c75"}},{"code_sha256_prefix":"ce127028071b7db8","entry":"download_asset","repo":"microsoft/DeBERTa","repo_kind":"official","path":"DeBERTa/deberta/cache_utils.py","file_url":"https://github.com/microsoft/DeBERTa/blob/HEAD/DeBERTa/deberta/cache_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce127028071b7db8"}},{"code_sha256_prefix":"f7b253455c817d10","entry":"join_chunks","repo":"microsoft/DeBERTa","repo_kind":"official","path":"DeBERTa/apps/_utils.py","file_url":"https://github.com/microsoft/DeBERTa/blob/HEAD/DeBERTa/apps/_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7b253455c817d10"}},{"code_sha256_prefix":"82c76e446399db0c","entry":"load_vocab","repo":"microsoft/DeBERTa","repo_kind":"official","path":"DeBERTa/deberta/cache_utils.py","file_url":"https://github.com/microsoft/DeBERTa/blob/HEAD/DeBERTa/deberta/cache_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82c76e446399db0c"}},{"code_sha256_prefix":"792471bf746ab83e","entry":"make_log_bucket_dict","repo":"microsoft/DeBERTa","repo_kind":"official","path":"DeBERTa/deberta/da_utils.py","file_url":"https://github.com/microsoft/DeBERTa/blob/HEAD/DeBERTa/deberta/da_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"792471bf746ab83e"}},{"code_sha256_prefix":"7af87f19f9630217","entry":"make_log_bucket_position","repo":"microsoft/DeBERTa","repo_kind":"official","path":"DeBERTa/deberta/da_utils.py","file_url":"https://github.com/microsoft/DeBERTa/blob/HEAD/DeBERTa/deberta/da_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7af87f19f9630217"}},{"code_sha256_prefix":"c734e14b42bb19f0","entry":"merge_distributed","repo":"microsoft/DeBERTa","repo_kind":"official","path":"DeBERTa/apps/_utils.py","file_url":"https://github.com/microsoft/DeBERTa/blob/HEAD/DeBERTa/apps/_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c734e14b42bb19f0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}