{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynabert-dynamic-bert-with-adaptive-width-and","title":"DynaBERT: Dynamic BERT with Adaptive Width and Depth","arxiv_id":"2004.04037","date":"2020-04-08","proceeding":"NeurIPS 2020 12","authors":["Lu Hou","Zhiqi Huang","Lifeng Shang","Xin Jiang","Xiao Chen","Qun Liu"],"abstract":"The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive. To alleviate this problem, one approach is to compress them for specific tasks before deployment. However, recent works on BERT compression usually compress the large BERT model to a fixed smaller size. They can not fully satisfy the requirements of different edge devices with various hardware performances. In this paper, we propose a novel dynamic BERT model (abbreviated as DynaBERT), which can flexibly adjust the size and latency by selecting adaptive width and depth. The training process of DynaBERT includes first training a width-adaptive BERT and then allowing both adaptive width and depth, by distilling knowledge from the full-sized model to small sub-networks. Network rewiring is also used to keep the more important attention heads and neurons shared by more sub-networks. Comprehensive experiments under various efficiency constraints demonstrate that our proposed dynamic BERT (or RoBERTa) at its largest size has comparable performance as BERT-base (or RoBERTa-base), while at smaller widths and depths consistently outperforms existing BERT compression methods. Code is available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/DynaBERT.","url_abs":"https://arxiv.org/abs/2004.04037v2","url_pdf":"https://arxiv.org/pdf/2004.04037v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynabert-dynamic-bert-with-adaptive-width-and","repo_url":"https://github.com/huawei-noah/Pretrained-Language-Model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"dynabert-dynamic-bert-with-adaptive-width-and","repo_url":"https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/DynaBERT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"dynabert-dynamic-bert-with-adaptive-width-and","repo_url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/model_compression/ofa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"dynabert","method_name":"DynaBERT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roberta","method_name":"RoBERTa"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[{"slug":"dynabert","name":"DynaBERT","full_name":"DynaBERT"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.04037","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.04037"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huawei-noah/Pretrained-Language-Model","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/model_compression/ofa","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/DynaBERT","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"ran_fixture":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"0cc816edf95e5320","entry":"DynaLinear","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"official","path":"DynaBERT/transformers/modeling_bert.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/DynaBERT/transformers/modeling_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0cc816edf95e5320"}},{"code_sha256_prefix":"35a997654c159c2b","entry":"convert_examples_to_features_test","repo":"huawei-noah/pretrained-language-model","repo_kind":"official","path":"DynaBERT/eval_glue.py","file_url":"https://github.com/huawei-noah/pretrained-language-model/blob/HEAD/DynaBERT/eval_glue.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35a997654c159c2b"}},{"code_sha256_prefix":"4ac7efa4e6383d65","entry":"gelu","repo":"huawei-noah/pretrained-language-model","repo_kind":"official","path":"DynaBERT/transformers/modeling_bert.py","file_url":"https://github.com/huawei-noah/pretrained-language-model/blob/HEAD/DynaBERT/transformers/modeling_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ac7efa4e6383d65"}},{"code_sha256_prefix":"25a3e8171033da6c","entry":"round_to_nearest","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"official","path":"DynaBERT/transformers/modeling_bert.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/DynaBERT/transformers/modeling_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25a3e8171033da6c"}},{"code_sha256_prefix":"32fee35165e8a309","entry":"soft_cross_entropy","repo":"huawei-noah/pretrained-language-model","repo_kind":"official","path":"DynaBERT/run_glue.py","file_url":"https://github.com/huawei-noah/pretrained-language-model/blob/HEAD/DynaBERT/run_glue.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32fee35165e8a309"}},{"code_sha256_prefix":"99a4d7ac6391ba1e","entry":"get_tensor_data","repo":"huawei-noah/pretrained-language-model","repo_kind":"official","path":"DynaBERT/eval_glue.py","file_url":"https://github.com/huawei-noah/pretrained-language-model/blob/HEAD/DynaBERT/eval_glue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99a4d7ac6391ba1e"}},{"code_sha256_prefix":"58ca2fa000be80dc","entry":"load_tf_weights_in_bert","repo":"huawei-noah/pretrained-language-model","repo_kind":"official","path":"DynaBERT/transformers/modeling_bert.py","file_url":"https://github.com/huawei-noah/pretrained-language-model/blob/HEAD/DynaBERT/transformers/modeling_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58ca2fa000be80dc"}},{"code_sha256_prefix":"745504b7e11ff510","entry":"round_to_nearest","repo":"huawei-noah/pretrained-language-model","repo_kind":"official","path":"DynaBERT/transformers/modeling_bert.py","file_url":"https://github.com/huawei-noah/pretrained-language-model/blob/HEAD/DynaBERT/transformers/modeling_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"745504b7e11ff510"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}