{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-laws-with-vocabulary-larger-models","title":"Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies","arxiv_id":"2407.13623","date":"2024-07-18","proceeding":null,"authors":["Chaofan Tao","Qian Liu","Longxu Dou","Niklas Muennighoff","Zhongwei Wan","Ping Luo","Min Lin","Ngai Wong"],"abstract":"Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the conclusion that the optimal vocabulary size depends on the compute budget, with larger models requiring larger vocabularies. Most LLMs, however, use insufficient vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2-70B should have been at least 216K, 7 times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs. Our work highlights the importance of jointly considering tokenization and model scaling for efficient pre-training. The code and demo are available at https://github.com/sail-sg/scaling-with-vocab and https://hf.co/spaces/sail/scaling-with-vocab-demo.","url_abs":"https://arxiv.org/abs/2407.13623v3","url_pdf":"https://arxiv.org/pdf/2407.13623v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-laws-with-vocabulary-larger-models","repo_url":"https://github.com/sail-sg/scaling-with-vocab","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"arc","task_name":"ARC"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.13623","atlas_url":"https://app.syntology.ai/?focus=2407.13623","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.13623"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/scaling-with-vocab","reach":null}],"summary":{"ran_fixture":1,"ran_violates":2,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"c8ff0a0bc420dacd","entry":"LSE","repo":"sail-sg/scaling-with-vocab","repo_kind":"official","path":"approach1_isoflops.py","file_url":"https://github.com/sail-sg/scaling-with-vocab/blob/HEAD/approach1_isoflops.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8ff0a0bc420dacd"}},{"code_sha256_prefix":"fe98406b87b34d45","entry":"LSE","repo":"sail-sg/scaling-with-vocab","repo_kind":"official","path":"approach2_derivative.py","file_url":"https://github.com/sail-sg/scaling-with-vocab/blob/HEAD/approach2_derivative.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe98406b87b34d45"}},{"code_sha256_prefix":"8031b85f41bc755f","entry":"LSE_Nnv_H","repo":"sail-sg/scaling-with-vocab","repo_kind":"official","path":"approach1_isoflops.py","file_url":"https://github.com/sail-sg/scaling-with-vocab/blob/HEAD/approach1_isoflops.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8031b85f41bc755f"}},{"code_sha256_prefix":"efa2370bd36a4ee6","entry":"dF_dV","repo":"sail-sg/scaling-with-vocab","repo_kind":"official","path":"approach2_derivative.py","file_url":"https://github.com/sail-sg/scaling-with-vocab/blob/HEAD/approach2_derivative.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efa2370bd36a4ee6"}},{"code_sha256_prefix":"bccf520fe6c78f34","entry":"objective_function_Nnv","repo":"sail-sg/scaling-with-vocab","repo_kind":"official","path":"approach2_derivative.py","file_url":"https://github.com/sail-sg/scaling-with-vocab/blob/HEAD/approach2_derivative.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bccf520fe6c78f34"}},{"code_sha256_prefix":"fd48483acd7fe2c1","entry":"dl_dv","repo":"sail-sg/scaling-with-vocab","repo_kind":"official","path":"approach3_isoloss.py","file_url":"https://github.com/sail-sg/scaling-with-vocab/blob/HEAD/approach3_isoloss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd48483acd7fe2c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}