{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-stability-of-fine-tuning-bert","title":"On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines","arxiv_id":"2006.04884","date":"2020-06-08","proceeding":"ICLR 2021 1","authors":["Marius Mosbach","Maksym Andriushchenko","Dietrich Klakow"],"abstract":"Fine-tuning pre-trained transformer-based language models such as BERT has become a common practice dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature (Devlin et al., 2019; Lee et al., 2020; Dodge et al., 2020) identified two potential reasons for the observed instability: catastrophic forgetting and small size of the fine-tuning datasets. In this paper, we show that both hypotheses fail to explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, fine-tuned on commonly used datasets from the GLUE benchmark, and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization where fine-tuned models with the same training loss exhibit noticeably different test performance. Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches. Code to reproduce our results is available online: https://github.com/uds-lsv/bert-stable-fine-tuning.","url_abs":"https://arxiv.org/abs/2006.04884v3","url_pdf":"https://arxiv.org/pdf/2006.04884v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-stability-of-fine-tuning-bert","repo_url":"https://github.com/uds-lsv/bert-stable-fine-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"on-the-stability-of-fine-tuning-bert","repo_url":"https://github.com/ruanchaves/eplm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"misconceptions","task_name":"Misconceptions"}],"methods":[{"method_slug":"albert","method_name":"ALBERT"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"lamb","method_name":"LAMB"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roberta","method_name":"RoBERTa"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.04884","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.04884"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/uds-lsv/bert-stable-fine-tuning","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/scitail","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruanchaves/eplm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"ran_honours":3,"unverified":13},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1},"listed":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/tokenization_bert.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"9bc31d06256e1691","entry":"gelu","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/modeling_tf_bert.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/modeling_tf_bert.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9bc31d06256e1691"}},{"code_sha256_prefix":"77601724cad03f95","entry":"gelu_new","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77601724cad03f95"}},{"code_sha256_prefix":"e7fbc7a74a3457c7","entry":"load_vocab","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/tokenization_bert.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7fbc7a74a3457c7"}},{"code_sha256_prefix":"060527bb34794b59","entry":"mish","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/modeling_bert.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"060527bb34794b59"}},{"code_sha256_prefix":"86d13b9f67274f19","entry":"swish","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/modeling_tf_bert.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/modeling_tf_bert.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86d13b9f67274f19"}},{"code_sha256_prefix":"33aeaf51efd2501f","entry":"add_benchmark_result","repo":"ruanchaves/eplm","repo_kind":"listed","path":"leaderboard/manage_data.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/leaderboard/manage_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33aeaf51efd2501f"}},{"code_sha256_prefix":"a6f1e11c54df3de5","entry":"add_dataset","repo":"ruanchaves/eplm","repo_kind":"listed","path":"leaderboard/manage_data.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/leaderboard/manage_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6f1e11c54df3de5"}},{"code_sha256_prefix":"d87861334682866a","entry":"check_dependency","repo":"ruanchaves/eplm","repo_kind":"listed","path":"leaderboard/run_app.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/leaderboard/run_app.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d87861334682866a"}},{"code_sha256_prefix":"d772ba25f4de22b0","entry":"convert_pytorch_checkpoint_to_tf","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/convert_bert_pytorch_checkpoint_to_original_tf.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/convert_bert_pytorch_checkpoint_to_original_tf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d772ba25f4de22b0"}},{"code_sha256_prefix":"ef9ba0bf81d816aa","entry":"convert_to_completion_format","repo":"ruanchaves/eplm","repo_kind":"listed","path":"src/napolab/napolab.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/src/napolab/napolab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef9ba0bf81d816aa"}},{"code_sha256_prefix":"02907c137bc813d1","entry":"export_napolab_benchmark","repo":"ruanchaves/eplm","repo_kind":"listed","path":"src/napolab/napolab.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/src/napolab/napolab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02907c137bc813d1"}},{"code_sha256_prefix":"653c7bc811a7f863","entry":"extract_data_from_json","repo":"ruanchaves/eplm","repo_kind":"listed","path":"leaderboard/extract_portuguese_leaderboard.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/leaderboard/extract_portuguese_leaderboard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"653c7bc811a7f863"}},{"code_sha256_prefix":"6e1f4e02403886e3","entry":"find_json_files","repo":"ruanchaves/eplm","repo_kind":"listed","path":"leaderboard/extract_portuguese_leaderboard.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/leaderboard/extract_portuguese_leaderboard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e1f4e02403886e3"}},{"code_sha256_prefix":"83257a2b015273fa","entry":"gelu_new","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/modeling_tf_bert.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/modeling_tf_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83257a2b015273fa"}},{"code_sha256_prefix":"e0c0f0759208e777","entry":"is_valid_json_file","repo":"ruanchaves/eplm","repo_kind":"listed","path":"leaderboard/extract_portuguese_leaderboard.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/leaderboard/extract_portuguese_leaderboard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0c0f0759208e777"}},{"code_sha256_prefix":"8db96d079a008116","entry":"load_napolab_benchmark","repo":"ruanchaves/eplm","repo_kind":"listed","path":"src/napolab/napolab.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/src/napolab/napolab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8db96d079a008116"}},{"code_sha256_prefix":"0766f48ad6e01b56","entry":"load_tf_weights_in_bert","repo":"uds-lsv/bert-stable-fine-tuning","repo_kind":"official","path":"src/transformers/modeling_bert.py","file_url":"https://github.com/uds-lsv/bert-stable-fine-tuning/blob/HEAD/src/transformers/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0766f48ad6e01b56"}},{"code_sha256_prefix":"03c4a8263ac6e7a1","entry":"validate_yaml_structure","repo":"ruanchaves/eplm","repo_kind":"listed","path":"leaderboard/manage_data.py","file_url":"https://github.com/ruanchaves/eplm/blob/HEAD/leaderboard/manage_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03c4a8263ac6e7a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}