{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lipsformer-introducing-lipschitz-continuity","title":"LipsFormer: Introducing Lipschitz Continuity to Vision Transformers","arxiv_id":"2304.09856","date":"2023-04-19","proceeding":null,"authors":["Xianbiao Qi","Jianan Wang","Yihao Chen","Yukai Shi","Lei Zhang"],"abstract":"We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models. In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation, and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability. In LipsFormer, we replace unstable Transformer component modules with Lipschitz continuous counterparts: CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot-product attention, and weighted residual shortcut. We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer. Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization. As a result, on the ImageNet 1K dataset, LipsFormer-Swin-Tiny based on Swin Transformer training for 300 epochs can obtain 82.7\\% without any learning rate warmup. Moreover, LipsFormer-CSwin-Tiny, based on CSwin, training for 300 epochs achieves a top-1 accuracy of 83.5\\% with 4.7G FLOPs and 24M parameters. The code will be released at \\url{https://github.com/IDEA-Research/LipsFormer}.","url_abs":"https://arxiv.org/abs/2304.09856v1","url_pdf":"https://arxiv.org/pdf/2304.09856v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lipsformer-introducing-lipschitz-continuity","repo_url":"https://github.com/idea-research/lipsformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"stochastic-depth","method_name":"Stochastic Depth"},{"method_slug":"swin-transformer","method_name":"Swin Transformer"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2304.09856","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.09856"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/idea-research/lipsformer","reach":null}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bff3ab4aec8724e7","entry":"CenterNorm","repo":"idea-research/lipsformer","repo_kind":"official","path":"models/lipsformer_swin.py","file_url":"https://github.com/idea-research/lipsformer/blob/HEAD/models/lipsformer_swin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bff3ab4aec8724e7"}},{"code_sha256_prefix":"d9360a58bc57b6fa","entry":"PatchMerging","repo":"idea-research/lipsformer","repo_kind":"official","path":"models/lipsformer_swin.py","file_url":"https://github.com/idea-research/lipsformer/blob/HEAD/models/lipsformer_swin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9360a58bc57b6fa"}},{"code_sha256_prefix":"1b5b3985105da5c3","entry":"ScaleLayer","repo":"idea-research/lipsformer","repo_kind":"official","path":"models/lipsformer_swin.py","file_url":"https://github.com/idea-research/lipsformer/blob/HEAD/models/lipsformer_swin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b5b3985105da5c3"}},{"code_sha256_prefix":"1f9b66b45a3405ea","entry":"WindowAttention","repo":"idea-research/lipsformer","repo_kind":"official","path":"models/lipsformer_swin.py","file_url":"https://github.com/idea-research/lipsformer/blob/HEAD/models/lipsformer_swin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f9b66b45a3405ea"}},{"code_sha256_prefix":"585a6cfefd2fda73","entry":"BasicLayer","repo":"idea-research/lipsformer","repo_kind":"official","path":"models/lipsformer_swin.py","file_url":"https://github.com/idea-research/lipsformer/blob/HEAD/models/lipsformer_swin.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"585a6cfefd2fda73"}},{"code_sha256_prefix":"8c334ecdf4b974b0","entry":"LipsFormerSwin","repo":"idea-research/lipsformer","repo_kind":"official","path":"models/lipsformer_swin.py","file_url":"https://github.com/idea-research/lipsformer/blob/HEAD/models/lipsformer_swin.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c334ecdf4b974b0"}},{"code_sha256_prefix":"8a37e13d93c3dbf9","entry":"SwinTransformerBlock","repo":"idea-research/lipsformer","repo_kind":"official","path":"models/lipsformer_swin.py","file_url":"https://github.com/idea-research/lipsformer/blob/HEAD/models/lipsformer_swin.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a37e13d93c3dbf9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}