{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analysing-the-impact-of-sequence-composition","title":"Analysing The Impact of Sequence Composition on Language Model Pre-Training","arxiv_id":"2402.13991","date":"2024-02-21","proceeding":null,"authors":["Yu Zhao","Yuanbin Qu","Konrad Staniszewski","Szymon Tworkowski","Wei Liu","Piotr Miłoś","Yuxiang Wu","Pasquale Minervini"],"abstract":"Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and efficiency. However, to this day, the influence of the pre-training sequence composition strategy on the generalisation properties of the model remains under-explored. In this work, we find that applying causal masking can lead to the inclusion of distracting information from previous documents during pre-training, which negatively impacts the performance of the models on language modelling and downstream tasks. In intra-document causal masking, the likelihood of each token is only conditioned on the previous tokens in the same document, eliminating potential distracting information from previous documents and significantly improving performance. Furthermore, we find that concatenating related documents can reduce some potential distractions during pre-training, and our proposed efficient retrieval-based sequence construction method, BM25Chunk, can improve in-context learning (+11.6\\%), knowledge memorisation (+9.8\\%), and context utilisation (+7.2\\%) abilities of language models without sacrificing efficiency.","url_abs":"https://arxiv.org/abs/2402.13991v1","url_pdf":"https://arxiv.org/pdf/2402.13991v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"analysing-the-impact-of-sequence-composition","repo_url":"https://github.com/yuzhaouoe/pretraining-data-packing","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.13991","atlas_url":"https://app.syntology.ai/?focus=2402.13991","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.13991"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yuzhaouoe/pretraining-data-packing","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran":16,"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":21,"ran":21,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"llama_modeling.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/llama_modeling.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"llama_modeling.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/llama_modeling.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"874a9519ebeb3c53","entry":"data_collator","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"packing_dataset.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/packing_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"874a9519ebeb3c53"}},{"code_sha256_prefix":"eb90b0d7f1e2c692","entry":"estimate_total_batch_num","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"save_offline_dataset.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/save_offline_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb90b0d7f1e2c692"}},{"code_sha256_prefix":"d9243c078ceacce8","entry":"eval_generation_em","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"evaluation/eval_utils.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/evaluation/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9243c078ceacce8"}},{"code_sha256_prefix":"efa3643614e9c5f5","entry":"exact_match_score","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"evaluation/eval_utils.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/evaluation/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efa3643614e9c5f5"}},{"code_sha256_prefix":"bb142953e9e84a14","entry":"get_batch_dir_and_path","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"analysis/burstiness.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/analysis/burstiness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb142953e9e84a14"}},{"code_sha256_prefix":"1bc433883369de22","entry":"get_fragment_lens","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"packing_dataset.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/packing_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bc433883369de22"}},{"code_sha256_prefix":"da105da73281cce5","entry":"get_offset_path","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da105da73281cce5"}},{"code_sha256_prefix":"2a6a0d112f2ef000","entry":"get_worker_id_and_iter_indices","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"packing_dataset.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/packing_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a6a0d112f2ef000"}},{"code_sha256_prefix":"813da56adc0dd787","entry":"load_json","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"evaluation/fewshot.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/evaluation/fewshot.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"813da56adc0dd787"}},{"code_sha256_prefix":"ad0a5228eb3d81f4","entry":"load_jsonl","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"evaluation/fewshot.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/evaluation/fewshot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad0a5228eb3d81f4"}},{"code_sha256_prefix":"11cde6e6bca04ca4","entry":"load_offset","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11cde6e6bca04ca4"}},{"code_sha256_prefix":"b678e0cd3681e4cd","entry":"load_pickle","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b678e0cd3681e4cd"}},{"code_sha256_prefix":"6829ceb72569fa12","entry":"metric_max_over_ground_truths","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"evaluation/eval_utils.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/evaluation/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6829ceb72569fa12"}},{"code_sha256_prefix":"9afac6122e0fd74e","entry":"normalise_pred","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"evaluation/fewshot.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/evaluation/fewshot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9afac6122e0fd74e"}},{"code_sha256_prefix":"137756758ba95a6a","entry":"process_batch_attn","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"analysis/distraction.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/analysis/distraction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"137756758ba95a6a"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"llama_modeling.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/llama_modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"efbd9b87adf49321","entry":"save_data_collator","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"save_offline_dataset.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/save_offline_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efbd9b87adf49321"}},{"code_sha256_prefix":"b494dc24033bda53","entry":"tokenize_one_item","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"analysis/distraction.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/analysis/distraction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b494dc24033bda53"}},{"code_sha256_prefix":"92ed24e518eb48dc","entry":"zipfian_fit","repo":"yuzhaouoe/pretraining-data-packing","repo_kind":"official","path":"analysis/burstiness.py","file_url":"https://github.com/yuzhaouoe/pretraining-data-packing/blob/HEAD/analysis/burstiness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92ed24e518eb48dc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}