{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/locost-state-space-models-for-long-document","title":"LOCOST: State-Space Models for Long Document Abstractive Summarization","arxiv_id":"2401.17919","date":"2024-01-31","proceeding":null,"authors":["Florian Le Bronnec","Song Duong","Mathieu Ravaut","Alexandre Allauzen","Nancy F. Chen","Vincent Guigue","Alberto Lumbreras","Laure Soulier","Patrick Gallinari"],"abstract":"State-space models are a low-complexity alternative to transformers for encoding long sequences and capturing long-term dependencies. We propose LOCOST: an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. With a computational complexity of $O(L \\log L)$, this architecture can handle significantly longer sequences than state-of-the-art models that are based on sparse attention patterns. We evaluate our model on a series of long document abstractive summarization tasks. The model reaches a performance level that is 93-96% comparable to the top-performing sparse transformers of the same size while saving up to 50% memory during training and up to 87% during inference. Additionally, LOCOST effectively handles input texts exceeding 600K tokens at inference time, setting new state-of-the-art results on full-book summarization and opening new perspectives for long input processing.","url_abs":"https://arxiv.org/abs/2401.17919v3","url_pdf":"https://arxiv.org/pdf/2401.17919v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"locost-state-space-models-for-long-document","repo_url":"https://github.com/flbbb/locost-summarization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":null,"task_name":"Book summarization"},{"task_slug":"conditional-text-generation","task_name":"Conditional Text Generation"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"state-space-models","task_name":"State Space Models"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.17919","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}