{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-sparse-fine-tuning-to-large-language","title":"Scaling Sparse Fine-Tuning to Large Language Models","arxiv_id":"2401.16405","date":"2024-01-29","proceeding":null,"authors":["Alan Ansell","Ivan Vulić","Hannah Sterz","Anna Korhonen","Edoardo M. Ponti"],"abstract":"Large Language Models (LLMs) are difficult to fully fine-tune (e.g., with instructions or human feedback) due to their sheer number of parameters. A family of parameter-efficient sparse fine-tuning methods have proven promising in terms of performance but their memory requirements increase proportionally to the size of the LLMs. In this work, we scale sparse fine-tuning to state-of-the-art LLMs like LLaMA 2 7B and 13B. We propose SpIEL, a novel sparse fine-tuning method which, for a desired density level, maintains an array of parameter indices and the deltas of these parameters relative to their pretrained values. It iterates over: (a) updating the active deltas, (b) pruning indices (based on the change of magnitude of their deltas) and (c) regrowth of indices. For regrowth, we explore two criteria based on either the accumulated gradients of a few candidate parameters or their approximate momenta estimated using the efficient SM3 optimizer. We experiment with instruction-tuning of LLMs on standard dataset mixtures, finding that SpIEL is often superior to popular parameter-efficient fine-tuning methods like LoRA (low-rank adaptation) in terms of performance and comparable in terms of run time. We additionally show that SpIEL is compatible with both quantization and efficient optimizers, to facilitate scaling to ever-larger model sizes. We release the code for SpIEL at https://github.com/AlanAnsell/peft and for the instruction-tuning experiments at https://github.com/ducdauge/sft-llm.","url_abs":"https://arxiv.org/abs/2401.16405v2","url_pdf":"https://arxiv.org/pdf/2401.16405v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-sparse-fine-tuning-to-large-language","repo_url":"https://github.com/alanansell/peft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"scaling-sparse-fine-tuning-to-large-language","repo_url":"https://github.com/ducdauge/sft-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"sft","method_name":"SFT"},{"method_slug":"sm3","method_name":"SM3"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.16405","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.16405"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ducdauge/sft-llm","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alanansell/peft","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":4,"ran":1,"unverified":8},"by_repo_kind":{"official":{"samples":14,"ran":6,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"32380a050ca033d4","entry":"b2mb","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"32380a050ca033d4"}},{"code_sha256_prefix":"1cb0eeb16cf04ee3","entry":"encode_with_messages_format","repo":"ducdauge/sft-llm","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/finetune/finetune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cb0eeb16cf04ee3"}},{"code_sha256_prefix":"88b8bf1223d522a4","entry":"encode_with_prompt_completion_format","repo":"ducdauge/sft-llm","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/finetune/finetune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88b8bf1223d522a4"}},{"code_sha256_prefix":"e080db2eaa81fdb3","entry":"estimate_pass_at_k","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/codex_humaneval/evaluation.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/codex_humaneval/evaluation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e080db2eaa81fdb3"}},{"code_sha256_prefix":"62004a6900e0e332","entry":"generate_completions","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62004a6900e0e332"}},{"code_sha256_prefix":"e2284f55bbde4f81","entry":"get_closest_label","repo":"alanansell/peft","repo_kind":"official","path":"examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py","file_url":"https://github.com/alanansell/peft/blob/HEAD/examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2284f55bbde4f81"}},{"code_sha256_prefix":"01f1ac304787218d","entry":"levenshtein_distance","repo":"alanansell/peft","repo_kind":"official","path":"examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py","file_url":"https://github.com/alanansell/peft/blob/HEAD/examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01f1ac304787218d"}},{"code_sha256_prefix":"12d4fef547070f7c","entry":"check_correctness","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/codex_humaneval/execution.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/codex_humaneval/execution.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12d4fef547070f7c"}},{"code_sha256_prefix":"e51c9713fadad16f","entry":"create_prompt_with_llama2_chat_format","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/templates.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/templates.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e51c9713fadad16f"}},{"code_sha256_prefix":"697d92a5d51eff3e","entry":"create_prompt_with_tulu_chat_format","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/templates.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/templates.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"697d92a5d51eff3e"}},{"code_sha256_prefix":"dcad9ac62979ca49","entry":"dispatch_openai_chat_requesets","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/dispatch_openai_requests.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/dispatch_openai_requests.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcad9ac62979ca49"}},{"code_sha256_prefix":"71e4259026911444","entry":"dispatch_openai_prompt_requesets","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/dispatch_openai_requests.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/dispatch_openai_requests.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71e4259026911444"}},{"code_sha256_prefix":"d134c88af0236263","entry":"get_next_word_predictions","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d134c88af0236263"}},{"code_sha256_prefix":"776a712a353130e7","entry":"read_problems","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/codex_humaneval/data.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/codex_humaneval/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"776a712a353130e7"}},{"code_sha256_prefix":"e86d5bc4e595d32c","entry":"score_completions","repo":"ducdauge/sft-llm","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/ducdauge/sft-llm/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e86d5bc4e595d32c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}