{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deberta-decoding-enhanced-bert-with","title":"DeBERTa: Decoding-enhanced BERT with Disentangled Attention","arxiv_id":"2006.03654","date":"2020-06-05","proceeding":"ICLR 2021 1","authors":["Pengcheng He","Xiaodong Liu","Jianfeng Gao","Weizhu Chen"],"abstract":"Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, out performing the human baseline by a decent margin (90.3 versus 89.8).","url_abs":"https://arxiv.org/abs/2006.03654v6","url_pdf":"https://arxiv.org/pdf/2006.03654v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/microsoft/DeBERTa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/huggingface/transformers","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/JONGWE1/Paddle-DeBERTa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"paddle","reach":{"status":"ok"}},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/Jason-J-Choi/DeBERTa_TxtClassifier","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/huberemanuel/DeBERTa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/neuralmind-ai/coliee","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/osu-nlp-group/mind2web","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/deberta","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/deberta_v2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/Mind23-2/MindCode-161","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/MindCode-4/code-3/tree/main/deberta","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/MindCode-4/code-3/tree/main/deberta_v2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/pwc-1/Paper-8/tree/main/deberta","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"deberta-decoding-enhanced-bert-with","repo_url":"https://github.com/pwc-1/Paper-8/tree/main/deberta_v2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"linguistic-acceptability","task_name":"Linguistic Acceptability"},{"task_slug":"math-word-problem-solving","task_name":"Math Word Problem Solving"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"deberta","method_name":"DeBERTa"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roberta","method_name":"RoBERTa"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[{"slug":"deberta","name":"DeBERTa","full_name":"DeBERTa"}],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"DeBERTa-1.5B","rank_in_archive_order":3,"of":45,"metrics":{"EM":"94.1","F1":"94.5"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-swag","task":"Common Sense Reasoning","dataset":"SWAG","model":"DeBERTalarge","rank_in_archive_order":1,"of":5,"metrics":{"Test":"90.8"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"DeBERTa-1.5B","rank_in_archive_order":6,"of":82,"metrics":{"Accuracy":"95.9"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-cola-dev","task":"Linguistic Acceptability","dataset":"CoLA Dev","model":"DeBERTa (large)","rank_in_archive_order":3,"of":6,"metrics":{"Accuracy":"69.5"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-paramawps","task":"Math Word Problem Solving","dataset":"ParaMAWPS","model":"DeBERTa","rank_in_archive_order":2,"of":6,"metrics":{"Accuracy (%)":"74.1"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"DeBERTa-1.5B","rank_in_archive_order":6,"of":20,"metrics":{"Accuracy":"97.2","F1":"94.9"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli","task":"Natural Language Inference","dataset":"MultiNLI","model":"DeBERTa (large)","rank_in_archive_order":7,"of":67,"metrics":{"Matched":"91.1","Mismatched":"91.1"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-qnli","task":"Natural Language Inference","dataset":"QNLI","model":"DeBERTa (large)","rank_in_archive_order":10,"of":43,"metrics":{"Accuracy":"95.3%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"DeBERTa-1.5B","rank_in_archive_order":5,"of":90,"metrics":{"Accuracy":"93.2%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-wnli","task":"Natural Language Inference","dataset":"WNLI","model":"DeBERTa","rank_in_archive_order":2,"of":23,"metrics":{"Accuracy":"94.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"DeBERTa-1.5B","rank_in_archive_order":9,"of":65,"metrics":{"Accuracy":"90.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"DeBERTa-Ensemble","rank_in_archive_order":5,"of":60,"metrics":{"Accuracy":"98.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"DeBERTa-1.5B","rank_in_archive_order":7,"of":60,"metrics":{"Accuracy":"96.8"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"DeBERTa-1.5B","rank_in_archive_order":4,"of":30,"metrics":{"EM":"63.7","F1":"88.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quora-question-pairs","task":"Question Answering","dataset":"Quora Question Pairs","model":"DeBERTa (large)","rank_in_archive_order":2,"of":19,"metrics":{"Accuracy":"92.3%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad20","task":"Question Answering","dataset":"SQuAD2.0","model":"DeBERTalarge","rank_in_archive_order":73,"of":286,"metrics":{"EM":"88.0","F1":"90.7"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"DeBERTalarge","rank_in_archive_order":5,"of":24,"metrics":{"Accuracy":"86.8"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-textual-similarity-on-sts-benchmark","task":"Semantic Textual Similarity","dataset":"STS Benchmark","model":"DeBERTa (large)","rank_in_archive_order":64,"of":66,"metrics":{"Accuracy":"92.5"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"DeBERTa++","rank_in_archive_order":11,"of":89,"metrics":{"Accuracy":"93"},"uses_additional_data":false},{"leaderboard":"/sota/sentiment-analysis-on-sst-2-binary","task":"Sentiment Analysis","dataset":"SST-2 Binary classification","model":"DeBERTa (large)","rank_in_archive_order":14,"of":87,"metrics":{"Accuracy":"96.5"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"DeBERTa-Ensemble","rank_in_archive_order":4,"of":37,"metrics":{"Accuracy":"77.5"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"DeBERTa-1.5B","rank_in_archive_order":9,"of":37,"metrics":{"Accuracy":"76.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.03654","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.03654"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/transformers","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/DeBERTa","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-3/tree/main/deberta_v2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/deberta","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jason-J-Choi/DeBERTa_TxtClassifier","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JONGWE1/Paddle-DeBERTa","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huberemanuel/DeBERTa","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/osu-nlp-group/mind2web","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-8/tree/main/deberta","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-8/tree/main/deberta_v2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neuralmind-ai/coliee","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-3/tree/main/deberta","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mind23-2/MindCode-161","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/deberta_v2","reach":null}],"summary":{"ran_honours":2,"ran_fixture":1,"ran_violates":1,"unverified":9},"by_repo_kind":{"named_in_paper":{"samples":8,"ran":2,"repositories":1},"listed":{"samples":4,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"13eb8311c1ce7813","entry":"build_relative_position","repo":"huberemanuel/DeBERTa","repo_kind":"listed","path":"DeBERTa/deberta/disentangled_attention.py","file_url":"https://github.com/huberemanuel/DeBERTa/blob/HEAD/DeBERTa/deberta/disentangled_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13eb8311c1ce7813"}},{"code_sha256_prefix":"ea37553c09194184","entry":"c2p_dynamic_expand","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ea37553c09194184"}},{"code_sha256_prefix":"ae6281c3878be6f2","entry":"p2c_dynamic_expand","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae6281c3878be6f2"}},{"code_sha256_prefix":"e2a1ad92582b1c32","entry":"uneven_size_corrected","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2a1ad92582b1c32"}},{"code_sha256_prefix":"19acc433093c6b7f","entry":"DisentangledSelfAttention","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19acc433093c6b7f"}},{"code_sha256_prefix":"11b6e1335dfd930b","entry":"DisentangledSelfAttention","repo":"huberemanuel/DeBERTa","repo_kind":"listed","path":"DeBERTa/deberta/disentangled_attention.py","file_url":"https://github.com/huberemanuel/DeBERTa/blob/HEAD/DeBERTa/deberta/disentangled_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11b6e1335dfd930b"}},{"code_sha256_prefix":"d58dca4065cc811b","entry":"DisentangledSelfAttention","repo":"Jason-J-Choi/DeBERTa_TxtClassifier","repo_kind":"listed","path":"DeBERTa/DeBERTa/deberta/disentangled_attention.py","file_url":"https://github.com/Jason-J-Choi/DeBERTa_TxtClassifier/blob/HEAD/DeBERTa/DeBERTa/deberta/disentangled_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d58dca4065cc811b"}},{"code_sha256_prefix":"329f69a2123fce0a","entry":"build_relative_position","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"329f69a2123fce0a"}},{"code_sha256_prefix":"f786b01bf4a40368","entry":"build_relative_position","repo":"Jason-J-Choi/DeBERTa_TxtClassifier","repo_kind":"listed","path":"DeBERTa/DeBERTa/deberta/disentangled_attention.py","file_url":"https://github.com/Jason-J-Choi/DeBERTa_TxtClassifier/blob/HEAD/DeBERTa/DeBERTa/deberta/disentangled_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f786b01bf4a40368"}},{"code_sha256_prefix":"ae7fe256df7fb4a0","entry":"build_rpos","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae7fe256df7fb4a0"}},{"code_sha256_prefix":"f396ca17c8d05eb5","entry":"compute_attention_span","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f396ca17c8d05eb5"}},{"code_sha256_prefix":"39afa351a5c3eb0f","entry":"pos_dynamic_expand","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39afa351a5c3eb0f"}},{"code_sha256_prefix":"3baad94f5c62f90f","entry":"scaled_size_sqrt","repo":"huggingface/transformers","repo_kind":"named_in_paper","path":"src/transformers/models/deberta/modeling_deberta.py","file_url":"https://github.com/huggingface/transformers/blob/HEAD/src/transformers/models/deberta/modeling_deberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3baad94f5c62f90f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}