{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transcending-scaling-laws-with-0-1-extra","title":"Transcending Scaling Laws with 0.1% Extra Compute","arxiv_id":"2210.11399","date":"2022-10-20","proceeding":null,"authors":["Yi Tay","Jason Wei","Hyung Won Chung","Vinh Q. Tran","David R. So","Siamak Shakeri","Xavier Garcia","Huaixiu Steven Zheng","Jinfeng Rao","Aakanksha Chowdhery","Denny Zhou","Donald Metzler","Slav Petrov","Neil Houlsby","Quoc V. Le","Mostafa Dehghani"],"abstract":"Scaling language models improves performance but comes with significant computational costs. This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. The key idea is to continue training a state-of-the-art large language model (e.g., PaLM) on a few more steps with UL2's mixture-of-denoiser objective. We show that, with almost negligible extra computational costs and no new sources of data, we are able to substantially improve the scaling properties of large language models on downstream metrics. In this paper, we continue training PaLM with UL2R, introducing a new set of models at 8B, 62B, and 540B scale which we call U-PaLM. Impressively, at 540B scale, we show an approximately 2x computational savings rate where U-PaLM achieves the same performance as the final PaLM 540B model at around half its computational budget (i.e., saving $\\sim$4.4 million TPUv4 hours). We further show that this improved scaling curve leads to 'emergent abilities' on challenging BIG-Bench tasks -- for instance, U-PaLM does much better than PaLM on some tasks or demonstrates better quality at much smaller scale (62B as opposed to 540B). Overall, we show that U-PaLM outperforms PaLM on many few-shot setups, i.e., English NLP tasks (e.g., commonsense reasoning, question answering), reasoning tasks with chain-of-thought (e.g., GSM8K), multilingual tasks (MGSM, TydiQA), MMLU and challenging BIG-Bench tasks. Finally, we provide qualitative examples showing the new capabilities of U-PaLM for single and multi-span infilling.","url_abs":"https://arxiv.org/abs/2210.11399v2","url_pdf":"https://arxiv.org/pdf/2210.11399v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"cross-lingual-question-answering","task_name":"Cross-Lingual Question Answering"},{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"palm","method_name":"PaLM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"U-PaLM","rank_in_archive_order":119,"of":164,"metrics":{"Accuracy":"58.5","Parameters (Billion)":"540"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-question-answering-on-tydiqa","task":"Cross-Lingual Question Answering","dataset":"TyDiQA-GoldP","model":"U-PaLM 62B (fine-tuned)","rank_in_archive_order":2,"of":11,"metrics":{"EM":"78.4","F1":"88.5"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-question-answering-on-tydiqa","task":"Cross-Lingual Question Answering","dataset":"TyDiQA-GoldP","model":"U-PaLM-540B (CoT)","rank_in_archive_order":6,"of":11,"metrics":{"EM":"54.6"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-mgsm","task":"Multi-task Language Understanding","dataset":"MGSM","model":"U-PaLM 540B (CoT)","rank_in_archive_order":7,"of":12,"metrics":{"Average (%)":"49.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-strategyqa","task":"Question Answering","dataset":"StrategyQA","model":"U-PaLM 540B","rank_in_archive_order":4,"of":12,"metrics":{"Accuracy":"76.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-strategyqa","task":"Question Answering","dataset":"StrategyQA","model":"PaLM 540B","rank_in_archive_order":5,"of":12,"metrics":{"Accuracy":"76.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-strategyqa","task":"Question Answering","dataset":"StrategyQA","model":"Minerva 540B","rank_in_archive_order":6,"of":12,"metrics":{"Accuracy":"61.9"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2210.11399","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}