{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-training-of-language-models-to-fill","title":"Efficient Training of Language Models to Fill in the Middle","arxiv_id":"2207.14255","date":"2022-07-28","proceeding":null,"authors":["Mohammad Bavarian","Heewoo Jun","Nikolas Tezak","John Schulman","Christine McLeavey","Jerry Tworek","Mark Chen"],"abstract":"We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data augmentation has garnered much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not harm the original left-to-right generative capability, as measured by perplexity and sampling evaluations across a wide range of scales. Given the usefulness, simplicity, and efficiency of training models to fill-in-the-middle (FIM), we suggest that future autoregressive language models be trained with FIM by default. To this end, we run a series of ablations on key hyperparameters, such as the data transformation frequency, the structure of the transformation, and the method of selecting the infill span. We use these ablations to prescribe strong default settings and best practices to train FIM models. We have released our best infilling model trained with best practices in our API, and release our infilling benchmarks to aid future research.","url_abs":"https://arxiv.org/abs/2207.14255v1","url_pdf":"https://arxiv.org/pdf/2207.14255v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-training-of-language-models-to-fill","repo_url":"https://github.com/openai/human-eval-infilling","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"efficient-training-of-language-models-to-fill","repo_url":"https://github.com/loubnabnl/santacoder-finetuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"efficient-training-of-language-models-to-fill","repo_url":"https://github.com/qwenlm/qwen2.5-coder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"efficient-training-of-language-models-to-fill","repo_url":"https://github.com/sensellm/fim-se","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2207.14255","atlas_url":"https://app.syntology.ai/?focus=2207.14255","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.14255"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sensellm/fim-se","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openai/human-eval-infilling","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/loubnabnl/santacoder-finetuning","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qwenlm/qwen2.5-coder","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"af79b5e511f124e0","entry":"estimate_pass_at_k","repo":"openai/human-eval-infilling","repo_kind":"official","path":"human_eval_infilling/evaluation.py","file_url":"https://github.com/openai/human-eval-infilling/blob/HEAD/human_eval_infilling/evaluation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af79b5e511f124e0"}},{"code_sha256_prefix":"46b65cfaf05bc714","entry":"get_fim_token_ids","repo":"loubnabnl/santacoder-finetuning","repo_kind":"listed","path":"fim.py","file_url":"https://github.com/loubnabnl/santacoder-finetuning/blob/HEAD/fim.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46b65cfaf05bc714"}},{"code_sha256_prefix":"6245ccab460ba92b","entry":"permute","repo":"loubnabnl/santacoder-finetuning","repo_kind":"listed","path":"fim.py","file_url":"https://github.com/loubnabnl/santacoder-finetuning/blob/HEAD/fim.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6245ccab460ba92b"}},{"code_sha256_prefix":"c3e6a38615ffe825","entry":"read_problems","repo":"openai/human-eval-infilling","repo_kind":"official","path":"human_eval_infilling/data.py","file_url":"https://github.com/openai/human-eval-infilling/blob/HEAD/human_eval_infilling/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3e6a38615ffe825"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}