{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-mode-connectivity-for-pre-trained","title":"Exploring Mode Connectivity for Pre-trained Language Models","arxiv_id":"2210.14102","date":"2022-10-25","proceeding":null,"authors":["Yujia Qin","Cheng Qian","Jing Yi","Weize Chen","Yankai Lin","Xu Han","Zhiyuan Liu","Maosong Sun","Jie zhou"],"abstract":"Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found. Although plenty of works have studied how to effectively and efficiently adapt PLMs to high-performance minima, little is known about the connection of various minima reached under different adaptation configurations. In this paper, we investigate the geometric connections of different minima through the lens of mode connectivity, which measures whether two minima can be connected with a low-loss path. We conduct empirical analyses to investigate three questions: (1) how could hyperparameters, specific tuning methods, and training data affect PLM's mode connectivity? (2) How does mode connectivity change during pre-training? (3) How does the PLM's task knowledge change along the path connecting two minima? In general, exploring the mode connectivity of PLMs conduces to understanding the geometric connection of different minima, which may help us fathom the inner workings of PLM downstream adaptation.","url_abs":"https://arxiv.org/abs/2210.14102v1","url_pdf":"https://arxiv.org/pdf/2210.14102v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-mode-connectivity-for-pre-trained","repo_url":"https://github.com/thunlp/mode-connectivity-plm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.14102","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.14102"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/mode-connectivity-plm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3f56b459ed961ce0","entry":"curve_find_finetune","repo":"thunlp/mode-connectivity-plm","repo_kind":"official","path":"T5_model/modeling_t5_PET_curve_find_finetune.py","file_url":"https://github.com/thunlp/mode-connectivity-plm/blob/HEAD/T5_model/modeling_t5_PET_curve_find_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f56b459ed961ce0"}},{"code_sha256_prefix":"be4dc5f3750cbf91","entry":"get_group_layer_name","repo":"thunlp/mode-connectivity-plm","repo_kind":"official","path":"T5_model/modeling_t5_layer_weight_find_finetune.py","file_url":"https://github.com/thunlp/mode-connectivity-plm/blob/HEAD/T5_model/modeling_t5_layer_weight_find_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be4dc5f3750cbf91"}},{"code_sha256_prefix":"d73f71b6a098b459","entry":"layer_weight_find_finetune","repo":"thunlp/mode-connectivity-plm","repo_kind":"official","path":"T5_model/modeling_t5_layer_weight_find_finetune.py","file_url":"https://github.com/thunlp/mode-connectivity-plm/blob/HEAD/T5_model/modeling_t5_layer_weight_find_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d73f71b6a098b459"}},{"code_sha256_prefix":"136abfa3e8ce028d","entry":"load_tf_weights_in_t5","repo":"thunlp/mode-connectivity-plm","repo_kind":"official","path":"T5_model/modeling_t5.py","file_url":"https://github.com/thunlp/mode-connectivity-plm/blob/HEAD/T5_model/modeling_t5.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"136abfa3e8ce028d"}},{"code_sha256_prefix":"aeda0735304f6bf2","entry":"write_result","repo":"thunlp/mode-connectivity-plm","repo_kind":"official","path":"tune_hps_singletask_PET_curve_find_finetune.py","file_url":"https://github.com/thunlp/mode-connectivity-plm/blob/HEAD/tune_hps_singletask_PET_curve_find_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aeda0735304f6bf2"}},{"code_sha256_prefix":"660d07f1033edbdb","entry":"write_result","repo":"thunlp/mode-connectivity-plm","repo_kind":"official","path":"tune_hps_singletask_layer_weight_find_finetune.py","file_url":"https://github.com/thunlp/mode-connectivity-plm/blob/HEAD/tune_hps_singletask_layer_weight_find_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"660d07f1033edbdb"}},{"code_sha256_prefix":"cd2afd80ad6aabcd","entry":"write_result","repo":"thunlp/mode-connectivity-plm","repo_kind":"official","path":"RoBERTa_model/itp_pretrain_adapter.py","file_url":"https://github.com/thunlp/mode-connectivity-plm/blob/HEAD/RoBERTa_model/itp_pretrain_adapter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd2afd80ad6aabcd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}