Papers › Adapting Pretrained Text-to-Text Models for Long Text Sequences

Adapting Pretrained Text-to-Text Models for Long Text Sequences

21 Sep 2022arXiv:2209.10052archive 2025-07-28

Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, Wen-tau Yih

We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through a comprehensive study along three axes of the pretraining pipeline -- model architecture, optimization objective, and pretraining corpus, we propose an effective recipe to build long-context models from existing short-context models. Specifically, we replace the full attention in transformers with pooling-augmented blockwise attention, and pretrain the model with a masked-span prediction task with spans of varying length. In terms of the pretraining corpus, we find that using randomly concatenated short-documents from a large open-domain corpus results in better performance than using existing long document corpora which are typically limited in their domain coverage. With these findings, we build a long-context model that achieves competitive performance on long-text QA tasks and establishes the new state of the art on five long-text summarization datasets, often outperforming previous methods with larger model sizes. Our code has been released at https://github.com/facebookresearch/bart_ls.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

facebookresearch/bart_ls officialmentioned in papermentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Long-range modelingQuestion AnsweringText Summarization

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Long-range modeling SCROLLS BART-LS Avg. 39.76 #4 of 13 Archive leaderboard report
Long-range modeling SCROLLS BART-LS CNLI 87.1 #4 of 13 Archive leaderboard report
Long-range modeling SCROLLS BART-LS GovRep 59.4 / 29.8 / 30.8 #4 of 13 Archive leaderboard report
Long-range modeling SCROLLS BART-LS Nrtv 26.2 #4 of 13 Archive leaderboard report
Long-range modeling SCROLLS BART-LS QALT EM-T/H 37.8 / 34.0 #4 of 13 Archive leaderboard report
Long-range modeling SCROLLS BART-LS QMSum 35.1 / 11.0 / 22.0 #4 of 13 Archive leaderboard report
Long-range modeling SCROLLS BART-LS Qspr 48.7 #4 of 13 Archive leaderboard report
Long-range modeling SCROLLS BART-LS SumScr 37.7 / 10.2 / 21.5 #4 of 13 Archive leaderboard report
Text Summarization Arxiv HEP-TH citation graph BART-LS ROUGE-1 50.2 #2 of 28 Archive leaderboard report
Text Summarization BookSum BART-LS ROUGE 38.5 #2 of 3 Archive leaderboard report
Text Summarization GovReport BART-LS ROUGE-1 62.0 #2 of 2 Archive leaderboard report
Text Summarization Pubmed BART-LS ROUGE-1 50.3 #3 of 29 Archive leaderboard report
Text Summarization QMSum BART-LS ROUGE-1 37.9 #1 of 1 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections