{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scalable-language-model-with-generalized","title":"Scalable Language Model with Generalized Continual Learning","arxiv_id":"2404.07470","date":"2024-04-11","proceeding":null,"authors":["Bohao Peng","Zhuotao Tian","Shu Liu","MingChang Yang","Jiaya Jia"],"abstract":"Continual learning has gained increasing importance as it facilitates the acquisition and refinement of scalable knowledge and skills in language models. However, existing methods typically encounter strict limitations and challenges in real-world scenarios, such as reliance on experience replay, optimization constraints, and inference task-ID. In this study, we introduce the Scalable Language Model (SLM) to overcome these limitations within a more challenging and generalized setting, representing a significant advancement toward practical applications for continual learning. Specifically, we propose the Joint Adaptive Re-Parameterization (JARe), integrated with Dynamic Task-related Knowledge Retrieval (DTKR), to enable adaptive adjustment of language models based on specific downstream tasks. This approach leverages the task distribution within the vector space, aiming to achieve a smooth and effortless continual learning process. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting. Moreover, while prior research primarily focused on a single task type such as classification, our study goes beyond, with the large language model, i.e., LLaMA-2, to explore the effects across diverse domains and task types, such that a single language model can be decently scaled to broader applications.","url_abs":"https://arxiv.org/abs/2404.07470v1","url_pdf":"https://arxiv.org/pdf/2404.07470v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scalable-language-model-with-generalized","repo_url":"https://github.com/pbihao/slm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.07470","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.07470"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pbihao/slm","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran":5,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"pbihao/slm","repo_kind":"official","path":"SLM-llama/models/llama.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-llama/models/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"3e833a99fe0661d6","entry":"accuracy_score","repo":"pbihao/slm","repo_kind":"official","path":"SLM-llama/score.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-llama/score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e833a99fe0661d6"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"pbihao/slm","repo_kind":"official","path":"SLM-llama/models/llama.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-llama/models/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"5bd1a404bf01d36d","entry":"attention_forward","repo":"pbihao/slm","repo_kind":"official","path":"SLM-bert/models/bert.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-bert/models/bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bd1a404bf01d36d"}},{"code_sha256_prefix":"6c1dbafc5286268c","entry":"generate_orthogonal_matrix","repo":"pbihao/slm","repo_kind":"official","path":"SLM-bert/models/retriever.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-bert/models/retriever.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c1dbafc5286268c"}},{"code_sha256_prefix":"50645962b9c6b6ee","entry":"layer_forward","repo":"pbihao/slm","repo_kind":"official","path":"SLM-bert/models/bert.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-bert/models/bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50645962b9c6b6ee"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"pbihao/slm","repo_kind":"official","path":"SLM-llama/models/llama.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-llama/models/llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"26276704c1c3fe54","entry":"self_output_forward","repo":"pbihao/slm","repo_kind":"official","path":"SLM-bert/models/bert.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-bert/models/bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26276704c1c3fe54"}},{"code_sha256_prefix":"ac03805e23103eb0","entry":"get_dataset_by_name","repo":"pbihao/slm","repo_kind":"official","path":"SLM-bert/dataset/utils.py","file_url":"https://github.com/pbihao/slm/blob/HEAD/SLM-bert/dataset/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac03805e23103eb0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}