{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/freeformer-frequency-enhanced-transformer-for","title":"FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecasting","arxiv_id":"2501.13989","date":"2025-01-23","proceeding":null,"authors":["Wenzhen Yue","Yong liu","Xianghua Ying","Bowei Xing","Ruohao Guo","Ji Shi"],"abstract":"This paper presents \\textbf{FreEformer}, a simple yet effective model that leverages a \\textbf{Fre}quency \\textbf{E}nhanced Trans\\textbf{former} for multivariate time series forecasting. Our work is based on the assumption that the frequency spectrum provides a global perspective on the composition of series across various frequencies and is highly suitable for robust representation learning. Specifically, we first convert time series into the complex frequency domain using the Discrete Fourier Transform (DFT). The Transformer architecture is then applied to the frequency spectra to capture cross-variate dependencies, with the real and imaginary parts processed independently. However, we observe that the vanilla attention matrix exhibits a low-rank characteristic, thus limiting representation diversity. This could be attributed to the inherent sparsity of the frequency domain and the strong-value-focused nature of Softmax in vanilla attention. To address this, we enhance the vanilla attention mechanism by introducing an additional learnable matrix to the original attention matrix, followed by row-wise L1 normalization. Theoretical analysis~demonstrates that this enhanced attention mechanism improves both feature diversity and gradient flow. Extensive experiments demonstrate that FreEformer consistently outperforms state-of-the-art models on eighteen real-world benchmarks covering electricity, traffic, weather, healthcare and finance. Notably, the enhanced attention mechanism also consistently improves the performance of state-of-the-art Transformer-based forecasters.","url_abs":"https://arxiv.org/abs/2501.13989v1","url_pdf":"https://arxiv.org/pdf/2501.13989v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"freeformer-frequency-enhanced-transformer-for","repo_url":"https://github.com/jackyue1994/FreEformer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"multivariate-time-series-forecasting","task_name":"Multivariate Time Series Forecasting"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"time-series-1","task_name":"Time Series"},{"task_slug":"time-series-forecasting","task_name":"Time Series Forecasting"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.13989","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.13989"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jackyue1994/FreEformer","reach":null}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"9bc6c1a9a4e99258","entry":"AttentionLayer","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9bc6c1a9a4e99258"}},{"code_sha256_prefix":"a9e6623f1781245e","entry":"CudaCKA","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9e6623f1781245e"}},{"code_sha256_prefix":"ca4789ac538fe313","entry":"EncoderLayer","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca4789ac538fe313"}},{"code_sha256_prefix":"2fa52f9908622dba","entry":"Encoder_ori","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fa52f9908622dba"}},{"code_sha256_prefix":"d91bc87d34339026","entry":"FullAttention_ablation","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d91bc87d34339026"}},{"code_sha256_prefix":"f3b0dc9186deaf87","entry":"RevIN","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3b0dc9186deaf87"}},{"code_sha256_prefix":"87351e0f476389c8","entry":"dynamic_projection","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87351e0f476389c8"}},{"code_sha256_prefix":"985ac3aa5ccdb250","entry":"Model","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"985ac3aa5ccdb250"}},{"code_sha256_prefix":"4c834ad81d5d62cd","entry":"plot_mat","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c834ad81d5d62cd"}},{"code_sha256_prefix":"b10eef45e3000da8","entry":"write_into_xls","repo":"jackyue1994/FreEformer","repo_kind":"official","path":"model/FrePatchTST3_fre_all.py","file_url":"https://github.com/jackyue1994/FreEformer/blob/HEAD/model/FrePatchTST3_fre_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b10eef45e3000da8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}