Papers › Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator

Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator

24 May 2023arXiv:2305.15099archive 2025-07-28

Ziwei He, Meng Yang, Minwei Feng, Jingcheng Yin, Xinbing Wang, Jingwen Leng, Zhouhan Lin

The transformer model is known to be computationally demanding, and prohibitively costly for long sequences, as the self-attention module uses a quadratic time and space complexity with respect to sequence length. Many researchers have focused on designing new forms of self-attention or introducing new parameters to overcome this limitation, however a large portion of them prohibits the model to inherit weights from large pretrained models. In this work, the transformer's inefficiency has been taken care of from another perspective. We propose Fourier Transformer, a simple yet effective approach by progressively removing redundancies in hidden sequence using the ready-made Fast Fourier Transform (FFT) operator to perform Discrete Cosine Transformation (DCT). Fourier Transformer is able to significantly reduce computational costs while retain the ability to inherit from various large pretrained models. Experiments show that our model achieves state-of-the-art performances among all transformer-based models on the long-range modeling benchmark LRA with significant improvement in both speed and space. For generative seq-to-seq tasks including CNN/DailyMail and ELI5, by inheriting the BART weights our model outperforms the standard BART and other efficient models. Our code is publicly available at https://github.com/LUMIA-Group/FourierTransformer

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

lumia-group/fouriertransformer officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Abstractive Text Summarization Document Summarization Long-range modeling Open-Domain Question Answering

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Abstractive Text Summarization	CNN / Daily Mail	Fourier Transformer	ROUGE-1	44.76	#8 of 53	Archive leaderboard	report
Abstractive Text Summarization	CNN / Daily Mail	Fourier Transformer	ROUGE-2	21.55	#8 of 53	Archive leaderboard	report
Abstractive Text Summarization	CNN / Daily Mail	Fourier Transformer	ROUGE-L	41.34	#8 of 53	Archive leaderboard	report
Document Summarization	CNN / Daily Mail	Fourier Transformer	ROUGE-1	44.76	#3 of 26	Archive leaderboard	report
Document Summarization	CNN / Daily Mail	Fourier Transformer	ROUGE-2	21.55	#3 of 26	Archive leaderboard	report
Document Summarization	CNN / Daily Mail	Fourier Transformer	ROUGE-L	41.34	#3 of 26	Archive leaderboard	report
Open-Domain Question Answering	ELI5	Fourier Transformer	Rouge-L	26.9	#1 of 6	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BART BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection SPEED Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections