{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/synthetic-continued-pretraining","title":"Synthetic continued pretraining","arxiv_id":"2409.07431","date":"2024-09-11","proceeding":null,"authors":["Zitong Yang","Neil Band","Shuangping Li","Emmanuel Candès","Tatsunori Hashimoto"],"abstract":"Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it. This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities. Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them. If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation. To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can \"rearrange\" knowledge to enable more data-efficient learning.","url_abs":"https://arxiv.org/abs/2409.07431v2","url_pdf":"https://arxiv.org/pdf/2409.07431v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"synthetic-continued-pretraining","repo_url":"https://github.com/zitongyang/synthetic_continued_pretraining","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.07431","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.07431"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zitongyang/synthetic_continued_pretraining","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d07d04439cd1d44f","entry":"jload","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/io_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/io_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d07d04439cd1d44f"}},{"code_sha256_prefix":"e48ecbe8e5f76edf","entry":"rerank_with_backoff","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"inference/retry_wrapper.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/inference/retry_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e48ecbe8e5f76edf"}},{"code_sha256_prefix":"9a7cf5feb8b5a38e","entry":"alleq","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/python_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/python_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a7cf5feb8b5a38e"}},{"code_sha256_prefix":"a6ce1f529ec84c5d","entry":"format_name","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/prompt_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/prompt_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6ce1f529ec84c5d"}},{"code_sha256_prefix":"8f53b333e679e9b4","entry":"gptqa","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"inference/devapi.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/inference/devapi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8f53b333e679e9b4"}},{"code_sha256_prefix":"d6d25a00486cb0a3","entry":"jload_list","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/io_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/io_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d6d25a00486cb0a3"}},{"code_sha256_prefix":"0cc00af6d58514f4","entry":"read","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/python_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/python_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0cc00af6d58514f4"}},{"code_sha256_prefix":"6d5f6f5924625234","entry":"round_dict_values","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/io_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/io_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d5f6f5924625234"}},{"code_sha256_prefix":"3115fb31c4023373","entry":"second_last_character","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/prompt_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/prompt_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3115fb31c4023373"}},{"code_sha256_prefix":"b96d28b695619968","entry":"uncapitalize_first","repo":"zitongyang/synthetic_continued_pretraining","repo_kind":"official","path":"utils/prompt_utils.py","file_url":"https://github.com/zitongyang/synthetic_continued_pretraining/blob/HEAD/utils/prompt_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b96d28b695619968"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}