{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crash-clustering-removing-and-sharing-enhance","title":"CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model","arxiv_id":"2310.15477","date":"2023-10-24","proceeding":null,"authors":["Kaiyan Zhang","Ning Ding","Biqing Qi","Xuekai Zhu","Xinwei Long","BoWen Zhou"],"abstract":"Instruction tuning has recently been recognized as an effective way of aligning Large Language Models (LLMs) to enhance their generalization ability across various tasks. However, when tuning publicly accessible, centralized LLMs with private instruction data, privacy concerns are inevitable. While direct transfer of parameterized modules between models is a plausible approach to address this, its implications and effectiveness need further exploration. This paper focuses on Offsite-Tuning (OFT), a representative technique that transfers transformer blocks between centralized LLMs and downstream emulators. Given the limited understanding of the underlying mechanism of OFT, we perform an empirical analysis on LLMs from the perspectives of representation and functional similarity. Interestingly, our findings reveal a unique modular structure within the layers of LLMs that appears to emerge as the model size expands. Simultaneously, we note subtle but potentially significant changes in representation and intermediate predictions across the layers. Inspired by these observations, we propose CRaSh, involving Clustering, Removing, and Sharing, a training-free strategy to derive improved emulators from LLMs. CRaSh significantly boosts performance of OFT with billions of parameters. Furthermore, we investigate the optimal solutions yielded by fine-tuning with and without full model through the lens of loss landscape. Our findings demonstrate a linear connectivity among these optima falling over the same basin, thereby highlighting the effectiveness of CRaSh and OFT. The source code is publicly available at https://github.com/TsinghuaC3I/CRaSh.","url_abs":"https://arxiv.org/abs/2310.15477v1","url_pdf":"https://arxiv.org/pdf/2310.15477v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crash-clustering-removing-and-sharing-enhance","repo_url":"https://github.com/tsinghuac3i/crash","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.15477","atlas_url":"https://app.syntology.ai/?focus=2310.15477","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.15477"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/TsinghuaC3I/CRaSh","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"82116195b9969a71","entry":"SVDLinearModule","repo":"TsinghuaC3I/CRaSh","repo_kind":"official","path":"src/linear_decompose.py","file_url":"https://github.com/TsinghuaC3I/CRaSh/blob/HEAD/src/linear_decompose.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82116195b9969a71"}},{"code_sha256_prefix":"d610598796bd4532","entry":"get_target_module","repo":"TsinghuaC3I/CRaSh","repo_kind":"official","path":"src/linear_decompose.py","file_url":"https://github.com/TsinghuaC3I/CRaSh/blob/HEAD/src/linear_decompose.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d610598796bd4532"}},{"code_sha256_prefix":"a74d50aadd79e8ac","entry":"get_target_modules_from_model_type","repo":"TsinghuaC3I/CRaSh","repo_kind":"official","path":"src/linear_decompose.py","file_url":"https://github.com/TsinghuaC3I/CRaSh/blob/HEAD/src/linear_decompose.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a74d50aadd79e8ac"}},{"code_sha256_prefix":"75b98da7d6e78d7d","entry":"find_linear_and_replace_with_svd_module","repo":"TsinghuaC3I/CRaSh","repo_kind":"official","path":"src/linear_decompose.py","file_url":"https://github.com/TsinghuaC3I/CRaSh/blob/HEAD/src/linear_decompose.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"75b98da7d6e78d7d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}