{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/momentum-stiefel-optimizer-with-applications","title":"Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport","arxiv_id":"2205.14173","date":"2022-05-27","proceeding":null,"authors":["Lingkai Kong","Yuqing Wang","Molei Tao"],"abstract":"The problem of optimization on Stiefel manifold, i.e., minimizing functions of (not necessarily square) matrices that satisfy orthogonality constraints, has been extensively studied. Yet, a new approach is proposed based on, for the first time, an interplay between thoughtfully designed continuous and discrete dynamics. It leads to a gradient-based optimizer with intrinsically added momentum. This method exactly preserves the manifold structure but does not require additional operation to keep momentum in the changing (co)tangent space, and thus has low computational cost and pleasant accuracy. Its generalization to adaptive learning rates is also demonstrated. Notable performances are observed in practical tasks. For instance, we found that placing orthogonal constraints on attention heads of trained-from-scratch Vision Transformer [Dosovitskiy et al. 2022] could markedly improve its performance, when our optimizer is used, and it is better that each head is made orthogonal within itself but not necessarily to other heads. This optimizer also makes the useful notion of Projection Robust Wasserstein Distance [Paty & Cuturi 2019; Lin et al. 2020] for high-dim. optimal transport even more effective.","url_abs":"https://arxiv.org/abs/2205.14173v3","url_pdf":"https://arxiv.org/pdf/2205.14173v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"momentum-stiefel-optimizer-with-applications","repo_url":"https://github.com/konglk1203/variationalstiefeloptimizer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.14173","atlas_url":"https://app.syntology.ai/?focus=2205.14173","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.14173"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/konglk1203/variationalstiefeloptimizer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_violates":1},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"11f75e7ba9b99252","entry":"bures_wasserstein","repo":"konglk1203/variationalstiefeloptimizer","repo_kind":"official","path":"ProjectionRobustWasserstein/utils.py","file_url":"https://github.com/konglk1203/variationalstiefeloptimizer/blob/HEAD/ProjectionRobustWasserstein/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11f75e7ba9b99252"}},{"code_sha256_prefix":"8c58d2935bdccfe6","entry":"euclidean_proj_simplex","repo":"konglk1203/variationalstiefeloptimizer","repo_kind":"official","path":"ProjectionRobustWasserstein/utils.py","file_url":"https://github.com/konglk1203/variationalstiefeloptimizer/blob/HEAD/ProjectionRobustWasserstein/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c58d2935bdccfe6"}},{"code_sha256_prefix":"9b40536aafe9e6c8","entry":"matrix_root","repo":"konglk1203/variationalstiefeloptimizer","repo_kind":"official","path":"utils_StiefelOptimizers.py","file_url":"https://github.com/konglk1203/variationalstiefeloptimizer/blob/HEAD/utils_StiefelOptimizers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b40536aafe9e6c8"}},{"code_sha256_prefix":"f27a35e212b683b4","entry":"matrix_root_inv","repo":"konglk1203/variationalstiefeloptimizer","repo_kind":"official","path":"utils_StiefelOptimizers.py","file_url":"https://github.com/konglk1203/variationalstiefeloptimizer/blob/HEAD/utils_StiefelOptimizers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f27a35e212b683b4"}},{"code_sha256_prefix":"d53df6acf4c93e3b","entry":"matrix_square_root","repo":"konglk1203/variationalstiefeloptimizer","repo_kind":"official","path":"utils_StiefelOptimizers.py","file_url":"https://github.com/konglk1203/variationalstiefeloptimizer/blob/HEAD/utils_StiefelOptimizers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d53df6acf4c93e3b"}},{"code_sha256_prefix":"6ba8cee9f5daea41","entry":"pair","repo":"konglk1203/variationalstiefeloptimizer","repo_kind":"official","path":"ViT/ViT.py","file_url":"https://github.com/konglk1203/variationalstiefeloptimizer/blob/HEAD/ViT/ViT.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ba8cee9f5daea41"}},{"code_sha256_prefix":"4e8159ee568761a8","entry":"sample_simplex","repo":"konglk1203/variationalstiefeloptimizer","repo_kind":"official","path":"ProjectionRobustWasserstein/utils.py","file_url":"https://github.com/konglk1203/variationalstiefeloptimizer/blob/HEAD/ProjectionRobustWasserstein/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e8159ee568761a8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}