{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-balancing-helps-low-data-training-and","title":"Model Balancing Helps Low-data Training and Fine-tuning","arxiv_id":"2410.12178","date":"2024-10-16","proceeding":null,"authors":["Zihang Liu","Yuanzhe Hu","Tianyu Pang","Yefan Zhou","Pu Ren","Yaoqing Yang"],"abstract":"Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets. Studies on these foundation models underscore the importance of low-data training and fine-tuning. This topic, well-known in natural language processing (NLP), has also gained increasing attention in the emerging field of scientific machine learning (SciML). To address the limitations of low-data training and fine-tuning, we draw inspiration from Heavy-Tailed Self-Regularization (HT-SR) theory, analyzing the shape of empirical spectral densities (ESDs) and revealing an imbalance in training quality across different model layers. To mitigate this issue, we adapt a recently proposed layer-wise learning rate scheduler, TempBalance, which effectively balances training quality across layers and enhances low-data training and fine-tuning for both NLP and SciML tasks. Notably, TempBalance demonstrates increasing performance gains as the amount of available tuning data decreases. Comparative analyses further highlight the effectiveness of TempBalance and its adaptability as an \"add-on\" method for improving model performance.","url_abs":"https://arxiv.org/abs/2410.12178v1","url_pdf":"https://arxiv.org/pdf/2410.12178v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-balancing-helps-low-data-training-and","repo_url":"https://github.com/zihanghliu/modelbalancing","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.12178","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.12178"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zihanghliu/modelbalancing","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":10},"by_repo_kind":{"official":{"samples":12,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"73b1d8b8ee56ade1","entry":"inverse_metrics","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"sciml/pdebench/models/metrics.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/sciml/pdebench/models/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73b1d8b8ee56ade1"}},{"code_sha256_prefix":"866c2ba03b2619cf","entry":"metric_func","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"sciml/pdebench/models/metrics.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/sciml/pdebench/models/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"866c2ba03b2619cf"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/src/transformers/activations_tf.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"9a09cbf63a4ddff4","entry":"hertz_to_mel","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/src/transformers/audio_utils.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/src/transformers/audio_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a09cbf63a4ddff4"}},{"code_sha256_prefix":"7e7c601df5b22499","entry":"mel_filter_bank","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/src/transformers/audio_utils.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/src/transformers/audio_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e7c601df5b22499"}},{"code_sha256_prefix":"2650530d2f0dd9b2","entry":"mel_to_hertz","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/src/transformers/audio_utils.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/src/transformers/audio_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2650530d2f0dd9b2"}},{"code_sha256_prefix":"c62bfade3b1b51b2","entry":"metrics","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"sciml/pdebench/models/metrics.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/sciml/pdebench/models/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c62bfade3b1b51b2"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/src/transformers/activations_tf.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"zihanghliu/modelbalancing","repo_kind":"official","path":"transformers/src/transformers/activations_tf.py","file_url":"https://github.com/zihanghliu/modelbalancing/blob/HEAD/transformers/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}