{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parameter-efficient-fine-tuning-of-state","title":"Parameter-Efficient Fine-Tuning of State Space Models","arxiv_id":"2410.09016","date":"2024-10-11","proceeding":null,"authors":["Kevin Galim","Wonjun Kang","Yuchen Zeng","Hyung Il Koo","Kangwook Lee"],"abstract":"Deep State Space Models (SSMs), such as Mamba (Gu & Dao, 2024), have emerged as powerful tools for language modeling, offering high performance with efficient inference and linear scaling in sequence length. However, the application of parameter-efficient fine-tuning (PEFT) methods to SSM-based models remains largely unexplored. This paper aims to systematically study two key questions: (i) How do existing PEFT methods perform on SSM-based models? (ii) Which modules are most effective for fine-tuning? We conduct an empirical benchmark of four basic PEFT methods on SSM-based models. Our findings reveal that prompt-based methods (e.g., prefix-tuning) are no longer effective, an empirical result further supported by theoretical analysis. In contrast, LoRA remains effective for SSM-based models. We further investigate the optimal application of LoRA within these models, demonstrating both theoretically and experimentally that applying LoRA to linear projection matrices without modifying SSM modules yields the best results, as LoRA is not effective at tuning SSM modules. To further improve performance, we introduce LoRA with Selective Dimension tuning (SDLoRA), which selectively updates certain channels and states on SSM modules while applying LoRA to linear projection matrices. Extensive experimental results show that this approach outperforms standard LoRA.","url_abs":"https://arxiv.org/abs/2410.09016v1","url_pdf":"https://arxiv.org/pdf/2410.09016v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parameter-efficient-fine-tuning-of-state","repo_url":"https://github.com/furiosa-ai/ssm-peft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"state-space-models","task_name":"State Space Models"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"mamba","method_name":"Mamba"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.09016","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.09016"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/furiosa-ai/ssm-peft","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"ba7b868fec181b57","entry":"selective_scan_ref","repo":"furiosa-ai/ssm-peft","repo_kind":"official","path":"mamba-peft/mamba_ssm_peft/ops/selective_scan_interface.py","file_url":"https://github.com/furiosa-ai/ssm-peft/blob/HEAD/mamba-peft/mamba_ssm_peft/ops/selective_scan_interface.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba7b868fec181b57"}},{"code_sha256_prefix":"2ce8fcf5bedd19ec","entry":"layer_norm_ref","repo":"furiosa-ai/ssm-peft","repo_kind":"official","path":"mamba-peft/mamba_ssm_peft/ops/triton/layernorm.py","file_url":"https://github.com/furiosa-ai/ssm-peft/blob/HEAD/mamba-peft/mamba_ssm_peft/ops/triton/layernorm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ce8fcf5bedd19ec"}},{"code_sha256_prefix":"ab802ae3195c6316","entry":"rms_norm_ref","repo":"furiosa-ai/ssm-peft","repo_kind":"official","path":"mamba-peft/mamba_ssm_peft/ops/triton/layernorm.py","file_url":"https://github.com/furiosa-ai/ssm-peft/blob/HEAD/mamba-peft/mamba_ssm_peft/ops/triton/layernorm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab802ae3195c6316"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}