{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analysis-of-mean-field-models-arising-from","title":"Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization","arxiv_id":"2501.03096","date":"2025-01-06","proceeding":null,"authors":["Martin Burger","Samira Kabri","Yury Korolev","Tim Roith","Lukas Weigand"],"abstract":"The aim of this paper is to provide a mathematical analysis of transformer architectures using a self-attention mechanism with layer normalization. In particular, observed patterns in such architectures resembling either clusters or uniform distributions pose a number of challenging mathematical questions. We focus on a special case that admits a gradient flow formulation in the spaces of probability measures on the unit sphere under a special metric, which allows us to give at least partial answers in a rigorous way. The arising mathematical problems resemble those recently studied in aggregation equations, but with additional challenges emerging from restricting the dynamics to the sphere and the particular form of the interaction energy. We provide a rigorous framework for studying the gradient flow, which also suggests a possible metric geometry to study the general case (i.e. one that is not described by a gradient flow). We further analyze the stationary points of the induced self-attention dynamics. The latter are related to stationary points of the interaction energy in the Wasserstein geometry, and we further discuss energy minimizers and maximizers in different parameter settings.","url_abs":"https://arxiv.org/abs/2501.03096v2","url_pdf":"https://arxiv.org/pdf/2501.03096v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"analysis-of-mean-field-models-arising-from","repo_url":"https://github.com/timroith/transformerdynamics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2501.03096","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.03096"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/timroith/transformerdynamics","reach":null}],"summary":{"ran_fixture":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"05f6382df0fd6fd1","entry":"cart2sph","repo":"timroith/transformerdynamics","repo_kind":"official","path":"tfdy/coordinates.py","file_url":"https://github.com/timroith/transformerdynamics/blob/HEAD/tfdy/coordinates.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05f6382df0fd6fd1"}},{"code_sha256_prefix":"ea14f2bdb57c7ef1","entry":"init_sph_normal","repo":"timroith/transformerdynamics","repo_kind":"official","path":"tfdy/coordinates.py","file_url":"https://github.com/timroith/transformerdynamics/blob/HEAD/tfdy/coordinates.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea14f2bdb57c7ef1"}},{"code_sha256_prefix":"bd6a94ac4e2cf2ca","entry":"sph2cart","repo":"timroith/transformerdynamics","repo_kind":"official","path":"tfdy/coordinates.py","file_url":"https://github.com/timroith/transformerdynamics/blob/HEAD/tfdy/coordinates.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd6a94ac4e2cf2ca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}