Papers › Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling

Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling

18 Oct 2023arXiv:2310.11772archive 2025-07-28

Hai Yu, Chong Deng, Qinglin Zhang, Jiaqing Liu, Qian Chen, Wen Wang

Topic segmentation is critical for obtaining structured documents and improving downstream tasks such as information retrieval. Due to its ability of automatically exploring clues of topic shift from abundant labeled data, recent supervised neural models have greatly promoted the development of long document topic segmentation, but leaving the deeper relationship between coherence and topic segmentation underexplored. Therefore, this paper enhances the ability of supervised models to capture coherence from both logical structure and semantic similarity perspectives to further improve the topic segmentation performance, proposing Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). Specifically, the TSSP task is proposed to force the model to comprehend structural information by learning the original relations between adjacent sentences in a disarrayed document, which is constructed by jointly disrupting the original document at topic and sentence levels. Moreover, we utilize inter- and intra-topic information to construct contrastive samples and design the CSSL objective to ensure that the sentences representations in the same topic have higher similarity, while those in different topics are less similar. Extensive experiments show that the Longformer with our approach significantly outperforms old state-of-the-art (SOTA) methods. Our approach improve F₁ of old SOTA by 3.42 (73.74 -> 77.16) and reduces Pₖ by 1.11 points (15.0 -> 13.89) on WIKI-727K and achieves an average relative reduction of 4.3% on Pₖ on WikiSection. The average relative Pₖ drop of 8.38% on two out-of-domain datasets also demonstrates the robustness of our approach.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

alibaba-damo-academy/spokennlp officialmentioned in papertfApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to BERT for Joint Intent Classification and Slot Filling (arXiv:1902.10909; that paper's own run record: 26 ran (of which 0 constructed an object rather than computing a result; 18 with no instrument failure: 1 honoured, 0 violated, 17 with no contract checked; 8 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Information RetrievalSegmentationSemantic SimilaritySemantic Textual SimilaritySentence

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

AdamWAttentionAttention DropoutDense ConnectionsDropoutLayer NormalizationLinear LayerLinear Warmup With Linear DecayLongformerMulti-Head AttentionResidual ConnectionSoftmaxWeight DecayWordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections