Papers › CDLM: Cross-Document Language Modeling

CDLM: Cross-Document Language Modeling

2 Jan 2021Findings (EMNLP) 2021 11arXiv:2101.00406archive 2025-07-28

Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters, Arie Cattan, Ido Dagan

We introduce a new pretraining approach geared for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective. First, instead of considering documents in isolation, we pretrain over sets of multiple related documents, encouraging the model to learn cross-document relationships. Second, we improve over recent long-range transformers by introducing dynamic global attention that has access to the entire input to predict masked tokens. We release CDLM (Cross-Document Language Model), a new general language model for multi-document setting that can be easily applied to downstream tasks. Our extensive analysis shows that both ideas are essential for the success of CDLM, and work in synergy to set new state-of-the-art results for several multi-text tasks. Code and models are available at https://github.com/aviclu/CDLM.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

aviclu/CD-LM officialmentioned in papermentioned on GitHubpytorch report

aviclu/cdlm officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Citation Recommendation Coreference Resolution Cross-Document Language Modeling Entity Cross-Document Coreference Resolution Event Coreference Resolution Event Cross-Document Coreference Resolution Language Modeling Language Modelling Masked Language Modeling Question Answering

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Cross-Document Language Modeling	MultiNews test	CD-LM	Perplexity	1.76	#1 of 3	Archive leaderboard	report
Cross-Document Language Modeling	MultiNews test	Rand CD-LM	Perplexity	1.93	#2 of 3	Archive leaderboard	report
Cross-Document Language Modeling	MultiNews test	Longformer	Perplexity	2.34	#3 of 3	Archive leaderboard	report
Cross-Document Language Modeling	MultiNews val	CD-LM	Perplexity	1.69	#1 of 3	Archive leaderboard	report
Cross-Document Language Modeling	MultiNews val	Rand CD-LM	Perplexity	1.88	#2 of 3	Archive leaderboard	report
Cross-Document Language Modeling	MultiNews val	Longformer	Perplexity	2.03	#3 of 3	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AdamW Attention Attention Dropout Dense Connections Dropout Layer Normalization Linear Layer Linear Warmup With Linear Decay Longformer Multi-Head Attention Residual Connection Softmax Weight Decay WordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections