{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/customizing-language-models-with-instance","title":"Customizing Language Models with Instance-wise LoRA for Sequential Recommendation","arxiv_id":"2408.10159","date":"2024-08-19","proceeding":null,"authors":["Xiaoyu Kong","Jiancan Wu","An Zhang","Leheng Sheng","Hui Lin","Xiang Wang","Xiangnan He"],"abstract":"Sequential recommendation systems predict the next interaction item based on users' past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge comprehension and reasoning, recent approaches are eager to apply LLMs to sequential recommendation. A common paradigm is converting user behavior sequences into instruction data, and fine-tuning the LLM with parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaption (LoRA). However, the uniform application of LoRA across diverse user behaviors is insufficient to capture individual variability, resulting in negative transfer between disparate sequences. To address these challenges, we propose Instance-wise LoRA (iLoRA). We innovatively treat the sequential recommendation task as a form of multi-task learning, integrating LoRA with the Mixture of Experts (MoE) framework. This approach encourages different experts to capture various aspects of user behavior. Additionally, we introduce a sequence representation guided gate function that generates customized expert participation weights for each user sequence, which allows dynamic parameter adjustment for instance-wise recommendations. In sequential recommendation, iLoRA achieves an average relative improvement of 11.4\\% over basic LoRA in the hit ratio metric, with less than a 1\\% relative increase in trainable parameters. Extensive experiments on three benchmark datasets demonstrate the effectiveness of iLoRA, highlighting its superior performance compared to existing methods in mitigating negative transfer and improving recommendation accuracy. Our data and code are available at https://github.com/AkaliKong/iLoRA.","url_abs":"https://arxiv.org/abs/2408.10159v4","url_pdf":"https://arxiv.org/pdf/2408.10159v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"customizing-language-models-with-instance","repo_url":"https://github.com/akalikong/ilora","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"sequential-recommendation","task_name":"Sequential Recommendation"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.10159","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.10159"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/AkaliKong/iLoRA","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/akalikong/ilora","reach":null}],"summary":{"ran":7,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"0f7faa1a9bdea8d3","entry":"Dense","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f7faa1a9bdea8d3"}},{"code_sha256_prefix":"7f027e7b0075c867","entry":"Drop","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f027e7b0075c867"}},{"code_sha256_prefix":"bd31da13616cefc1","entry":"MLP","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd31da13616cefc1"}},{"code_sha256_prefix":"8ae2a6c35d209940","entry":"MLP_noise","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ae2a6c35d209940"}},{"code_sha256_prefix":"cab304c6fe8e399f","entry":"MoeLoraLayer","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cab304c6fe8e399f"}},{"code_sha256_prefix":"752e59eb5fe15c43","entry":"Noise","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"752e59eb5fe15c43"}},{"code_sha256_prefix":"66b65be6c96f7959","entry":"extract_axis_1","repo":"AkaliKong/iLoRA","repo_kind":"official","path":"recommender/A_SASRec_final_bce_llm.py","file_url":"https://github.com/AkaliKong/iLoRA/blob/HEAD/recommender/A_SASRec_final_bce_llm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66b65be6c96f7959"}},{"code_sha256_prefix":"559c0fdcb9d02e48","entry":"topK","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"559c0fdcb9d02e48"}},{"code_sha256_prefix":"9d19ea5c94bfb203","entry":"Conv2d","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d19ea5c94bfb203"}},{"code_sha256_prefix":"fdb24ec07f306248","entry":"Embedding","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdb24ec07f306248"}},{"code_sha256_prefix":"9d07ddd15e3baf47","entry":"Linear","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d07ddd15e3baf47"}},{"code_sha256_prefix":"18f6c92d1729ae16","entry":"MoeLoraModel","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18f6c92d1729ae16"}},{"code_sha256_prefix":"3ab2cfc21d501f78","entry":"mark_only_lora_as_trainable","repo":"akalikong/ilora","repo_kind":"official","path":"model/peft/tuners/moelora.py","file_url":"https://github.com/akalikong/ilora/blob/HEAD/model/peft/tuners/moelora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ab2cfc21d501f78"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}