{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/preventing-gradient-attenuation-in-lipschitz","title":"Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks","arxiv_id":"1911.00937","date":"2019-11-03","proceeding":"NeurIPS 2019 12","authors":["Qiyang Li","Saminul Haque","Cem Anil","James Lucas","Roger Grosse","Jörn-Henrik Jacobsen"],"abstract":"Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuristic approaches such as the gradient penalty have seen much practical success, it is challenging to achieve similar practical performance while provably enforcing a Lipschitz constraint. In principle, one can design Lipschitz constrained architectures using the composition property of Lipschitz functions, but Anil et al. recently identified a key obstacle to this approach: gradient norm attenuation. They showed how to circumvent this problem in the case of fully connected networks by designing each layer to be gradient norm preserving. We extend their approach to train scalable, expressive, provably Lipschitz convolutional networks. In particular, we present the Block Convolution Orthogonal Parameterization (BCOP), an expressive parameterization of orthogonal convolution operations. We show that even though the space of orthogonal convolutions is disconnected, the largest connected component of BCOP with 2n channels can represent arbitrary BCOP convolutions over n channels. Our BCOP parameterization allows us to train large convolutional networks with provable Lipschitz bounds. Empirically, we find that it is competitive with existing approaches to provable adversarial robustness and Wasserstein distance estimation.","url_abs":"https://arxiv.org/abs/1911.00937v2","url_pdf":"https://arxiv.org/pdf/1911.00937v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"preventing-gradient-attenuation-in-lipschitz","repo_url":"https://github.com/ColinQiyangLi/LConvNet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1911.00937","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.00937"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ColinQiyangLi/LConvNet","reach":null}],"summary":{"ran_fixture":4,"ran_draft_wrong":4,"ran":1,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c6d2da76d5fbb78a","entry":"bjorck_orthonormalize","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6d2da76d5fbb78a"}},{"code_sha256_prefix":"67faa492a8df483a","entry":"block_orth","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67faa492a8df483a"}},{"code_sha256_prefix":"0e56ee61b3f7a88d","entry":"conv2d_cyclic_pad","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e56ee61b3f7a88d"}},{"code_sha256_prefix":"8c6d22a2cf2995bc","entry":"conv_singular_values_numpy","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c6d22a2cf2995bc"}},{"code_sha256_prefix":"a03b76e7e8b7145d","entry":"cyclic_pad_2d","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a03b76e7e8b7145d"}},{"code_sha256_prefix":"2bdb69b2570e1bd8","entry":"dict_to_tensor","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bdb69b2570e1bd8"}},{"code_sha256_prefix":"20d0aab322225a36","entry":"matrix_conv","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20d0aab322225a36"}},{"code_sha256_prefix":"6e1165eb18d80570","entry":"orthogonal_matrix","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e1165eb18d80570"}},{"code_sha256_prefix":"264dacf0aae30538","entry":"power_iteration","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"264dacf0aae30538"}},{"code_sha256_prefix":"246c26b7c160c814","entry":"symmetric_projection","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"246c26b7c160c814"}},{"code_sha256_prefix":"568d86e4f1f7b544","entry":"BCOP","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"568d86e4f1f7b544"}},{"code_sha256_prefix":"b3df67d1d29dafde","entry":"LipschitzModuleL2","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3df67d1d29dafde"}},{"code_sha256_prefix":"33d734483d94ad69","entry":"StreamlinedModule","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33d734483d94ad69"}},{"code_sha256_prefix":"5a14b8c3f447240f","entry":"convolution_orthogonal_generator_projs","repo":"ColinQiyangLi/LConvNet","repo_kind":"official","path":"lconvnet/layers/bcop.py","file_url":"https://github.com/ColinQiyangLi/LConvNet/blob/HEAD/lconvnet/layers/bcop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a14b8c3f447240f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}