{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hift-a-hierarchical-full-parameter-fine","title":"HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy","arxiv_id":"2401.15207","date":"2024-01-26","proceeding":null,"authors":["Yongkang Liu","Yiqun Zhang","Qian Li","Tong Liu","Shi Feng","Daling Wang","Yifei Zhang","Hinrich Schütze"],"abstract":"Full-parameter fine-tuning has become the go-to choice for adapting language models (LMs) to downstream tasks due to its excellent performance. As LMs grow in size, fine-tuning the full parameters of LMs requires a prohibitively large amount of GPU memory. Existing approaches utilize zeroth-order optimizer to conserve GPU memory, which can potentially compromise the performance of LMs as non-zero order optimizers tend to converge more readily on most downstream tasks. In this paper, we propose a novel optimizer-independent end-to-end hierarchical fine-tuning strategy, HiFT, which only updates a subset of parameters at each training step. HiFT can significantly reduce the amount of gradients and optimizer state parameters residing in GPU memory at the same time, thereby reducing GPU memory usage. Our results demonstrate that: (1) HiFT achieves comparable performance to parameter-efficient fine-tuning and standard full parameter fine-tuning. (2) HiFT supports various optimizers including AdamW, AdaGrad, SGD, etc. (3) HiFT can save more than 60\\% GPU memory compared with standard full-parameter fine-tuning for 7B model. (4) HiFT enables full-parameter fine-tuning of a 7B model on single 48G A6000 with a precision of 32 using the AdamW optimizer, without using any memory saving techniques.","url_abs":"https://arxiv.org/abs/2401.15207v3","url_pdf":"https://arxiv.org/pdf/2401.15207v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hift-a-hierarchical-full-parameter-fine","repo_url":"https://github.com/misonsky/HiFT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.15207","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.15207"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/misonsky/HiFT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e663bce882e8bf09","entry":"move_device","repo":"misonsky/HiFT","repo_kind":"official","path":"hift/optimizers/replace_operation.py","file_url":"https://github.com/misonsky/HiFT/blob/HEAD/hift/optimizers/replace_operation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e663bce882e8bf09"}},{"code_sha256_prefix":"e71d5072840e250a","entry":"unscale_and_clip_grads","repo":"misonsky/HiFT","repo_kind":"official","path":"hift/optimizers/replace_operation.py","file_url":"https://github.com/misonsky/HiFT/blob/HEAD/hift/optimizers/replace_operation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e71d5072840e250a"}},{"code_sha256_prefix":"1d5caf3fed4fa072","entry":"GetCallBack","repo":"misonsky/HiFT","repo_kind":"official","path":"hift/registerCallBack.py","file_url":"https://github.com/misonsky/HiFT/blob/HEAD/hift/registerCallBack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1d5caf3fed4fa072"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}