{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/specialized-language-models-with-cheap","title":"Need a Small Specialized Language Model? Plan Early!","arxiv_id":"2402.01093","date":"2024-02-02","proceeding":null,"authors":["David Grangier","Angelos Katharopoulos","Pierre Ablin","Awni Hannun"],"abstract":"Large language models are versatile tools but are not suitable for small inference budgets. Small models have more efficient inference, but their lower capacity means that their performance can be good only if one limits their scope to a specialized domain. This paper explores how to get good specialized small language models using a large, generic, pretraining set and a limited amount of specialized data. We consider two scenarios, depending on whether (i) one can afford pretraining a model for each specialization task, or (ii) one wants to cheaply adapt a single pretrained model for each task. In the first scenario, we propose an effective solution based on importance sampling: we resample the pretraining set to imitate the specialization data and train a small model on it. In the second scenario, we propose a novel architecture, projected networks (PN). PN is a large network whose parameters can be linearly projected into a small network for specialization. For both scenarios, we demonstrate the empirical effectiveness of our solutions across various domains, training set sizes, and training budgets.","url_abs":"https://arxiv.org/abs/2402.01093v2","url_pdf":"https://arxiv.org/pdf/2402.01093v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-the-pile","task":"Language Modelling","dataset":"The Pile","model":"Larger Transformer 771M (fine-tuned)","rank_in_archive_order":28,"of":39,"metrics":{"Test perplexity":"10"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-the-pile","task":"Language Modelling","dataset":"The Pile","model":"Smaller Transformer 126M (fine-tuned)","rank_in_archive_order":33,"of":39,"metrics":{"Test perplexity":"12"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-the-pile","task":"Language Modelling","dataset":"The Pile","model":"Larger Transformer 771M (pre-trained)","rank_in_archive_order":37,"of":39,"metrics":{"Test perplexity":"28.1"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-the-pile","task":"Language Modelling","dataset":"The Pile","model":"Smaller Transformer 126M (pre-trained)","rank_in_archive_order":39,"of":39,"metrics":{"Test perplexity":"33"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.01093","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}