{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continuum-attention-for-neural-operators","title":"Continuum Attention for Neural Operators","arxiv_id":"2406.06486","date":"2024-06-10","proceeding":null,"authors":["Edoardo Calvello","Nikola B. Kovachki","Matthew E. Levine","Andrew M. Stuart"],"abstract":"Transformers, and the attention mechanism in particular, have become ubiquitous in machine learning. Their success in modeling nonlocal, long-range correlations has led to their widespread adoption in natural language processing, computer vision, and time-series problems. Neural operators, which map spaces of functions into spaces of functions, are necessarily both nonlinear and nonlocal if they are universal; it is thus natural to ask whether the attention mechanism can be used in the design of neural operators. Motivated by this, we study transformers in the function space setting. We formulate attention as a map between infinite dimensional function spaces and prove that the attention mechanism as implemented in practice is a Monte Carlo or finite difference approximation of this operator. The function space formulation allows for the design of transformer neural operators, a class of architectures designed to learn mappings between function spaces, for which we prove a universal approximation result. The prohibitive cost of applying the attention operator to functions defined on multi-dimensional domains leads to the need for more efficient attention-based architectures. For this reason we also introduce a function space generalization of the patching strategy from computer vision, and introduce a class of associated neural operators. Numerical results, on an array of operator learning problems, demonstrate the promise of our approaches to function space formulations of attention and their use in neural operators.","url_abs":"https://arxiv.org/abs/2406.06486v1","url_pdf":"https://arxiv.org/pdf/2406.06486v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continuum-attention-for-neural-operators","repo_url":"https://github.com/EdoardoCalvello/TransformerNeuralOperators","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"operator-learning","task_name":"Operator learning"}],"methods":[{"method_slug":"patching","method_name":"Patching"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.06486","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06486"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EdoardoCalvello/TransformerNeuralOperators","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"42bfa77d54601cab","entry":"dict_combiner","repo":"EdoardoCalvello/TransformerNeuralOperators","repo_kind":"official","path":"utils.py","file_url":"https://github.com/EdoardoCalvello/TransformerNeuralOperators/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42bfa77d54601cab"}},{"code_sha256_prefix":"efb4e55b145a90d2","entry":"patch_coords","repo":"EdoardoCalvello/TransformerNeuralOperators","repo_kind":"official","path":"utils.py","file_url":"https://github.com/EdoardoCalvello/TransformerNeuralOperators/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efb4e55b145a90d2"}},{"code_sha256_prefix":"df12635e293bf3c4","entry":"subsample_and_flatten","repo":"EdoardoCalvello/TransformerNeuralOperators","repo_kind":"official","path":"utils.py","file_url":"https://github.com/EdoardoCalvello/TransformerNeuralOperators/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df12635e293bf3c4"}},{"code_sha256_prefix":"4e5bb2fe4cfdf281","entry":"load_dyn_sys_class","repo":"EdoardoCalvello/TransformerNeuralOperators","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/EdoardoCalvello/TransformerNeuralOperators/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e5bb2fe4cfdf281"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}