{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/meta-chunking-learning-efficient-text","title":"Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perception","arxiv_id":"2410.12788","date":"2024-10-16","proceeding":null,"authors":["Jihao Zhao","Zhiyuan Ji","Yuchen Feng","Pengnian Qi","Simin Niu","Bo Tang","Feiyu Xiong","Zhiyu Li"],"abstract":"While Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm for boosting large language models (LLMs) in knowledge-intensive tasks, it often overlooks the crucial aspect of text chunking within its workflow. This paper proposes the Meta-Chunking framework, which specifically enhances chunking quality through a dual strategy that identifies optimal segmentation points and preserves global information. Initially, breaking limitations of similarity-based chunking, we design two adaptive chunking techniques based on uncertainty, namely Perplexity Chunking and Margin Sampling Chunking, by utilizing the logical perception capabilities of LLMs. Given the inherent complexity across different texts, we integrate meta-chunk with dynamic merging, striking a balance between fine-grained and coarse-grained text chunking. Furthermore, we establish the global information compensation mechanism, encompassing a two-stage hierarchical summary generation process and a three-stage text chunk rewriting procedure focused on missing reflection, refinement, and completion. These components collectively strengthen the semantic integrity and contextual coherence of chunks. Extensive experiments demonstrate that Meta-Chunking effectively addresses challenges of the chunking task within the RAG system, providing LLMs with more logically coherent text chunks. Additionally, our methodology validates the feasibility of implementing high-quality chunking tasks with smaller-scale models, thereby eliminating the reliance on robust instruction-following capabilities.","url_abs":"https://arxiv.org/abs/2410.12788v3","url_pdf":"https://arxiv.org/pdf/2410.12788v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"meta-chunking-learning-efficient-text","repo_url":"https://github.com/IAAR-Shanghai/Meta-Chunking","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"chunking","task_name":"Chunking"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"multi-hop-question-answering","task_name":"Multi-hop Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"text-segmentation","task_name":"Text Segmentation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.12788","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.12788"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/IAAR-Shanghai/Meta-Chunking","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":4},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b9bb5562919b3c10","entry":"add_ids","repo":"IAAR-Shanghai/Meta-Chunking","repo_kind":"official","path":"meta_chunking/LongBench/LumberChunker.py","file_url":"https://github.com/IAAR-Shanghai/Meta-Chunking/blob/HEAD/meta_chunking/LongBench/LumberChunker.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"none","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9bb5562919b3c10"}},{"code_sha256_prefix":"a4d6434de692eabe","entry":"count_words","repo":"IAAR-Shanghai/Meta-Chunking","repo_kind":"official","path":"meta_chunking/LongBench/LumberChunker.py","file_url":"https://github.com/IAAR-Shanghai/Meta-Chunking/blob/HEAD/meta_chunking/LongBench/LumberChunker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4d6434de692eabe"}},{"code_sha256_prefix":"d9695cd69058903d","entry":"find_minima","repo":"IAAR-Shanghai/Meta-Chunking","repo_kind":"official","path":"meta_chunking/CRUD/chunk_rag.py","file_url":"https://github.com/IAAR-Shanghai/Meta-Chunking/blob/HEAD/meta_chunking/CRUD/chunk_rag.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9695cd69058903d"}},{"code_sha256_prefix":"6e1958db7ccbedb0","entry":"find_minima_dynamic","repo":"IAAR-Shanghai/Meta-Chunking","repo_kind":"official","path":"meta_chunking/LongBench/chunk_rag.py","file_url":"https://github.com/IAAR-Shanghai/Meta-Chunking/blob/HEAD/meta_chunking/LongBench/chunk_rag.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e1958db7ccbedb0"}},{"code_sha256_prefix":"e5efaeeb2a1dd2ee","entry":"split_text_by_punctuation","repo":"IAAR-Shanghai/Meta-Chunking","repo_kind":"official","path":"meta_chunking/CRUD/chunk_rag.py","file_url":"https://github.com/IAAR-Shanghai/Meta-Chunking/blob/HEAD/meta_chunking/CRUD/chunk_rag.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5efaeeb2a1dd2ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}