{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/to-stay-or-not-to-stay-in-the-pre-train-basin","title":"To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning","arxiv_id":"2303.03374","date":"2023-03-06","proceeding":"NeurIPS 2023 11","authors":["Ildus Sadrtdinov","Dmitrii Pozdeev","Dmitry Vetrov","Ekaterina Lobacheva"],"abstract":"Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of models fine-tuned from a single pre-trained checkpoint are often used in practice. Such models end up in the same basin of the loss landscape, which we call the pre-train basin, and thus have limited diversity. In this work, we show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, however, leaving the basin results in losing the benefits of transfer learning and in degradation of the ensemble quality. Based on the analysis of existing exploration methods, we propose a more effective modification of the Snapshot Ensembles (SSE) for transfer learning setup, StarSSE, which results in stronger ensembles and uniform model soups.","url_abs":"https://arxiv.org/abs/2303.03374v3","url_pdf":"https://arxiv.org/pdf/2303.03374v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"to-stay-or-not-to-stay-in-the-pre-train-basin","repo_url":"https://github.com/isadrtdinov/ens-for-transfer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"snapshot-ensembles","method_name":"Snapshot Ensembles"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2303.03374","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.03374"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/isadrtdinov/ens-for-transfer","reach":null}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"144bded7463f1200","entry":"Regularization","repo":"isadrtdinov/ens-for-transfer","repo_kind":"official","path":"models/reg/regularization.py","file_url":"https://github.com/isadrtdinov/ens-for-transfer/blob/HEAD/models/reg/regularization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"144bded7463f1200"}},{"code_sha256_prefix":"d35768eb9556827d","entry":"VolumeRegularization","repo":"isadrtdinov/ens-for-transfer","repo_kind":"official","path":"models/reg/regularization.py","file_url":"https://github.com/isadrtdinov/ens-for-transfer/blob/HEAD/models/reg/regularization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d35768eb9556827d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}