{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adapting-pretrained-text-to-text-models-for","title":"Adapting Pretrained Text-to-Text Models for Long Text Sequences","arxiv_id":"2209.10052","date":"2022-09-21","proceeding":null,"authors":["Wenhan Xiong","Anchit Gupta","Shubham Toshniwal","Yashar Mehdad","Wen-tau Yih"],"abstract":"We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through a comprehensive study along three axes of the pretraining pipeline -- model architecture, optimization objective, and pretraining corpus, we propose an effective recipe to build long-context models from existing short-context models. Specifically, we replace the full attention in transformers with pooling-augmented blockwise attention, and pretrain the model with a masked-span prediction task with spans of varying length. In terms of the pretraining corpus, we find that using randomly concatenated short-documents from a large open-domain corpus results in better performance than using existing long document corpora which are typically limited in their domain coverage. With these findings, we build a long-context model that achieves competitive performance on long-text QA tasks and establishes the new state of the art on five long-text summarization datasets, often outperforming previous methods with larger model sizes. Our code has been released at https://github.com/facebookresearch/bart_ls.","url_abs":"https://arxiv.org/abs/2209.10052v2","url_pdf":"https://arxiv.org/pdf/2209.10052v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adapting-pretrained-text-to-text-models-for","repo_url":"https://github.com/facebookresearch/bart_ls","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"long-range-modeling","task_name":"Long-range modeling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/long-range-modeling-on-scrolls","task":"Long-range modeling","dataset":"SCROLLS","model":"BART-LS","rank_in_archive_order":4,"of":13,"metrics":{"Avg.":"39.76","CNLI":"87.1","GovRep":"59.4 / 29.8 / 30.8","Nrtv":"26.2","QALT EM-T/H":"37.8 / 34.0","QMSum":"35.1 / 11.0 / 22.0","Qspr":"48.7","SumScr":"37.7 / 10.2 / 21.5"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-arxiv","task":"Text Summarization","dataset":"Arxiv HEP-TH citation graph","model":"BART-LS","rank_in_archive_order":2,"of":28,"metrics":{"ROUGE-1":"50.2"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-booksum","task":"Text Summarization","dataset":"BookSum","model":"BART-LS","rank_in_archive_order":2,"of":3,"metrics":{"ROUGE":"38.5"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-govreport","task":"Text Summarization","dataset":"GovReport","model":"BART-LS","rank_in_archive_order":2,"of":2,"metrics":{"ROUGE-1":"62.0"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-pubmed-1","task":"Text Summarization","dataset":"Pubmed","model":"BART-LS","rank_in_archive_order":3,"of":29,"metrics":{"ROUGE-1":"50.3"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-qmsum","task":"Text Summarization","dataset":"QMSum","model":"BART-LS","rank_in_archive_order":1,"of":1,"metrics":{"ROUGE-1":"37.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.10052","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}