{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reasoning-to-learn-from-latent-thoughts","title":"Reasoning to Learn from Latent Thoughts","arxiv_id":"2503.18866","date":"2025-03-24","proceeding":null,"authors":["Yangjun Ruan","Neil Band","Chris J. Maddison","Tatsunori Hashimoto"],"abstract":"Compute scaling for language model (LM) pretraining has outpaced the growth of human-written texts, leading to concerns that data will become the bottleneck to LM scaling. To continue scaling pretraining in this data-constrained regime, we propose that explicitly modeling and inferring the latent thoughts that underlie the text generation process can significantly improve pretraining data efficiency. Intuitively, our approach views web text as the compressed final outcome of a verbose human thought process and that the latent thoughts contain important contextual knowledge and reasoning steps that are critical to data-efficient learning. We empirically demonstrate the effectiveness of our approach through data-constrained continued pretraining for math. We first show that synthetic data approaches to inferring latent thoughts significantly improve data efficiency, outperforming training on the same amount of raw data (5.7\\% $\\rightarrow$ 25.4\\% on MATH). Furthermore, we demonstrate latent thought inference without a strong teacher, where an LM bootstraps its own performance by using an EM algorithm to iteratively improve the capability of the trained LM and the quality of thought-augmented pretraining data. We show that a 1B LM can bootstrap its performance across at least three iterations and significantly outperform baselines trained on raw data, with increasing gains from additional inference compute when performing the E-step. The gains from inference scaling and EM iterations suggest new opportunities for scaling data-constrained pretraining.","url_abs":"https://arxiv.org/abs/2503.18866v1","url_pdf":"https://arxiv.org/pdf/2503.18866v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reasoning-to-learn-from-latent-thoughts","repo_url":"https://github.com/ryoungj/BoLT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.18866","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.18866"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ryoungj/BoLT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":16},"by_repo_kind":{"listed":{"samples":20,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e0ebc4027f1bd03b","entry":"dataclass_from_dict","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/args.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/args.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0ebc4027f1bd03b"}},{"code_sha256_prefix":"116b2f19f9a6148c","entry":"dataclass_to_dict","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/args.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/args.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"116b2f19f9a6148c"}},{"code_sha256_prefix":"15be2e56e073a228","entry":"flatten_dict","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/args.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/args.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15be2e56e073a228"}},{"code_sha256_prefix":"bfdcca22e67a0a67","entry":"get_empty_buffer_state","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/data.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfdcca22e67a0a67"}},{"code_sha256_prefix":"151d95dbffc50b17","entry":"consolidate_checkpoints","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/checkpoint.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/checkpoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"151d95dbffc50b17"}},{"code_sha256_prefix":"9caf44ff8e00b1dc","entry":"dist_max","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/distributed.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9caf44ff8e00b1dc"}},{"code_sha256_prefix":"d09c7072d82024f4","entry":"dist_mean","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/distributed.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d09c7072d82024f4"}},{"code_sha256_prefix":"c653762aea4a44dc","entry":"distribute_data_to_rank","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/data.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c653762aea4a44dc"}},{"code_sha256_prefix":"a74ec567207165c9","entry":"find_and_sanitize_chunks","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/data.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a74ec567207165c9"}},{"code_sha256_prefix":"d9d11e6287ad4384","entry":"get_device_mesh","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/distributed.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9d11e6287ad4384"}},{"code_sha256_prefix":"81d8582d1fab7bda","entry":"get_num_params","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/metrics.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81d8582d1fab7bda"}},{"code_sha256_prefix":"7acd931f5e29ebf3","entry":"load_dataset_from_hf","repo":"ryoungj/BoLT","repo_kind":"listed","path":"apps/main/finetune.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/apps/main/finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7acd931f5e29ebf3"}},{"code_sha256_prefix":"e2d321ba9c5bbf1d","entry":"load_from_checkpoint","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/checkpoint.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/checkpoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2d321ba9c5bbf1d"}},{"code_sha256_prefix":"22fbaa2caf605450","entry":"log_stats","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/probe.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/probe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22fbaa2caf605450"}},{"code_sha256_prefix":"784760add5d1bc21","entry":"lr_cosine","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/optim.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"784760add5d1bc21"}},{"code_sha256_prefix":"90403d20388a5636","entry":"lr_inv_sqrt","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/optim.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90403d20388a5636"}},{"code_sha256_prefix":"4e2454a79a8338d0","entry":"lr_linear","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/optim.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e2454a79a8338d0"}},{"code_sha256_prefix":"30e68c9cadfddd32","entry":"matmul","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/float8.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/float8.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30e68c9cadfddd32"}},{"code_sha256_prefix":"aaa4b37e564944fe","entry":"named_replace","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/float8.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/float8.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aaa4b37e564944fe"}},{"code_sha256_prefix":"2848341791f9a450","entry":"scale","repo":"ryoungj/BoLT","repo_kind":"listed","path":"lingua/float8.py","file_url":"https://github.com/ryoungj/BoLT/blob/HEAD/lingua/float8.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2848341791f9a450"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}