{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/flowformer-linearizing-transformers-with","title":"Flowformer: Linearizing Transformers with Conservation Flows","arxiv_id":"2202.06258","date":"2022-02-13","proceeding":null,"authors":["Haixu Wu","Jialong Wu","Jiehui Xu","Jianmin Wang","Mingsheng Long"],"abstract":"Transformers based on the attention mechanism have achieved impressive success in various areas. However, the attention mechanism has a quadratic complexity, significantly impeding Transformers from dealing with numerous tokens and scaling up to bigger models. Previous methods mainly utilize the similarity decomposition and the associativity of matrix multiplication to devise linear-time attention mechanisms. They avoid degeneration of attention to a trivial distribution by reintroducing inductive biases such as the locality, thereby at the expense of model generality and expressiveness. In this paper, we linearize Transformers free from specific inductive biases based on the flow network theory. We cast attention as the information flow aggregated from the sources (values) to the sinks (results) through the learned flow capacities (attentions). Within this framework, we apply the property of flow conservation into attention and propose the Flow-Attention mechanism of linear complexity. By respectively conserving the incoming flow of sinks for source competition and the outgoing flow of sources for sink allocation, Flow-Attention inherently generates informative attentions without using specific inductive biases. Empowered by the Flow-Attention, Flowformer yields strong performance in linear time for wide areas, including long sequence, time series, vision, natural language, and reinforcement learning. The code and settings are available at this repository: https://github.com/thuml/Flowformer.","url_abs":"https://arxiv.org/abs/2202.06258v2","url_pdf":"https://arxiv.org/pdf/2202.06258v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"flowformer-linearizing-transformers-with","repo_url":"https://github.com/thuml/Flowformer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"time-series-1","task_name":"Time Series"},{"task_slug":"time-series","task_name":"Time Series Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/d4rl-on-d4rl","task":"D4RL","dataset":"D4RL","model":"Flowformer","rank_in_archive_order":4,"of":9,"metrics":{"Average Reward":"73.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.06258","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.06258"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thuml/Flowformer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"908619be92da9903","entry":"batch_index_select","repo":"thuml/Flowformer","repo_kind":"official","path":"Flowformer_CV/utils.py","file_url":"https://github.com/thuml/Flowformer/blob/HEAD/Flowformer_CV/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"908619be92da9903"}},{"code_sha256_prefix":"7337f1f5ff01dcd0","entry":"get_lr","repo":"thuml/Flowformer","repo_kind":"official","path":"Flowformer_RL/utils.py","file_url":"https://github.com/thuml/Flowformer/blob/HEAD/Flowformer_RL/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7337f1f5ff01dcd0"}},{"code_sha256_prefix":"42958997018c3581","entry":"make_dir","repo":"thuml/Flowformer","repo_kind":"official","path":"Flowformer_RL/utils.py","file_url":"https://github.com/thuml/Flowformer/blob/HEAD/Flowformer_RL/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42958997018c3581"}},{"code_sha256_prefix":"09d8970cc61c5886","entry":"build_dataset","repo":"thuml/Flowformer","repo_kind":"official","path":"Flowformer_CV/datasets.py","file_url":"https://github.com/thuml/Flowformer/blob/HEAD/Flowformer_CV/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09d8970cc61c5886"}},{"code_sha256_prefix":"984b0cb17eae025e","entry":"build_transform","repo":"thuml/Flowformer","repo_kind":"official","path":"Flowformer_CV/datasets.py","file_url":"https://github.com/thuml/Flowformer/blob/HEAD/Flowformer_CV/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"984b0cb17eae025e"}},{"code_sha256_prefix":"48ee0364b1d8b847","entry":"get_pos_encoder","repo":"thuml/Flowformer","repo_kind":"official","path":"Flowformer_TimeSeries/models/Flowformer.py","file_url":"https://github.com/thuml/Flowformer/blob/HEAD/Flowformer_TimeSeries/models/Flowformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48ee0364b1d8b847"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}