{"url":"/method/compressive-transformer","slug":"compressive-transformer","name":"Compressive Transformer","full_name":"Compressive Transformer","full_name_withheld":false,"description_markdown":"The **Compressive Transformer** is an extension to the [Transformer](https://paperswithcode.com/method/transformer) which maps past hidden activations (memories) to a smaller set of compressed representations (compressed memories). The Compressive Transformer uses the same attention mechanism over its set of memories and compressed memories, learning to query both its short-term granular memory and longer-term coarse memory. It builds on the ideas of [Transformer-XL](https://paperswithcode.com/method/transformer-xl) which maintains a memory of past activations at each layer to preserve a longer history of context. The Transformer-XL discards past activations when they become sufficiently old (controlled by the size of the memory). The key principle of the Compressive Transformer is to compress these old memories, instead of discarding them, and store them in an additional [compressed memory](https://paperswithcode.com/method/compressed-memory).\r\n\r\nAt each time step $t$, we discard the oldest compressed memories (FIFO) and then the oldest $n$ states from ordinary memory are compressed and shifted to the new slot in compressed memory. During training, the compressive memory component is optimized separately from the main language model (separate training loop).","description_state":"present","introduced_year":null,"introduced_by":{"title":"Compressive Transformers for Long-Range Sequence Modelling","paper":"/paper/compressive-transformers-for-long-range-1","first_author":"Jack W. Rae","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/compressive-transformers-for-long-range-1"},"source":{"url":"https://arxiv.org/abs/1911.05507v1","title":"Compressive Transformers for Long-Range Sequence Modelling","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/lucidrains/compressive-transformer-pytorch/blob/a73d146ee7bc9e47cc6c525f645a5bbb583537ad/compressive_transformer_pytorch/compressive_transformer_pytorch.py#L291","code_snippet_url_on_a_code_host":true,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":3,"archive_num_papers":3,"papers_newest_first":[{"paper":"/paper/collection-space-navigator-an-interactive","title":"Collection Space Navigator: An Interactive Visualization Interface for Multidimensional Datasets","date":"2023-05-11","arxiv_id":"2305.06809","n_code_links":2,"syntology":null},{"paper":null,"title":"DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence","date":"2021-10-10","arxiv_id":"2110.04821","n_code_links":0,"syntology":null},{"paper":"/paper/compressive-transformers-for-long-range-1","title":"Compressive Transformers for Long-Range Sequence Modelling","date":"2019-11-13","arxiv_id":"1911.05507","n_code_links":6,"syntology":{"ran":3,"of":11,"unverified":8,"pointer_only":0}}],"papers_shown":3,"tasks":[{"task":"/task/data-visualization","name":"Data Visualization","papers":1},{"task":"/task/dimensionality-reduction","name":"Dimensionality Reduction","papers":1},{"task":"/task/embeddings-evaluation","name":"Embeddings Evaluation","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/sentence","name":"Sentence","papers":1}],"tasks_shown":5,"n_tasks":5,"usage_by_year":[{"year":"2019","papers":1},{"year":"2021","papers":1},{"year":"2023","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/compressive-transformer"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}