{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improved-analysis-of-clipping-algorithms-for","title":"Improved Analysis of Clipping Algorithms for Non-convex Optimization","arxiv_id":"2010.02519","date":"2020-10-05","proceeding":"NeurIPS 2020 12","authors":["Bohang Zhang","Jikai Jin","Cong Fang","LiWei Wang"],"abstract":"Gradient clipping is commonly used in training deep neural networks partly due to its practicability in relieving the exploding gradient problem. Recently, \\citet{zhang2019gradient} show that clipped (stochastic) Gradient Descent (GD) converges faster than vanilla GD/SGD via introducing a new assumption called $(L_0, L_1)$-smoothness, which characterizes the violent fluctuation of gradients typically encountered in deep neural networks. However, their iteration complexities on the problem-dependent parameters are rather pessimistic, and theoretical justification of clipping combined with other crucial techniques, e.g. momentum acceleration, are still lacking. In this paper, we bridge the gap by presenting a general framework to study the clipping algorithms, which also takes momentum methods into consideration. We provide convergence analysis of the framework in both deterministic and stochastic setting, and demonstrate the tightness of our results by comparing them with existing lower bounds. Our results imply that the efficiency of clipping methods will not degenerate even in highly non-smooth regions of the landscape. Experiments confirm the superiority of clipping-based methods in deep learning tasks.","url_abs":"https://arxiv.org/abs/2010.02519v2","url_pdf":"https://arxiv.org/pdf/2010.02519v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improved-analysis-of-clipping-algorithms-for","repo_url":"https://github.com/zbh2047/clipping-algorithms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2010.02519","atlas_url":"https://app.syntology.ai/?focus=2010.02519","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.02519"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zbh2047/clipping-algorithms","reach":null}],"summary":{"ran":1,"ran_draft_wrong":3,"ran_fixture":4,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"2f2b5b01c14d1c34","entry":"SGDClip","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"algorithm.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f2b5b01c14d1c34"}},{"code_sha256_prefix":"96d4db57e5c2248f","entry":"batchify","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_lstm.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_lstm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96d4db57e5c2248f"}},{"code_sha256_prefix":"f1095e1951238c8e","entry":"get_batch","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_lstm.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_lstm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1095e1951238c8e"}},{"code_sha256_prefix":"8762f81d32460bc1","entry":"parse_algo","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_mnist_cifar.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_mnist_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8762f81d32460bc1"}},{"code_sha256_prefix":"a01e88267842be7f","entry":"parse_dataset","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_mnist_cifar.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_mnist_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a01e88267842be7f"}},{"code_sha256_prefix":"ec8a09099cf40ac2","entry":"parse_model","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_mnist_cifar.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_mnist_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec8a09099cf40ac2"}},{"code_sha256_prefix":"ec9329606c156d1d","entry":"repackage_hidden","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_lstm.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_lstm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec9329606c156d1d"}},{"code_sha256_prefix":"64af453cacc5f615","entry":"sum_of_square_grad","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"algorithm.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64af453cacc5f615"}},{"code_sha256_prefix":"3d552685e82f026d","entry":"update_momentum","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"algorithm.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d552685e82f026d"}},{"code_sha256_prefix":"9ab5f783e4f918a3","entry":"AlgorithmBase","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"algorithm.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/algorithm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ab5f783e4f918a3"}},{"code_sha256_prefix":"9b8289076669fe4f","entry":"accuracy","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_imagenet.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b8289076669fe4f"}},{"code_sha256_prefix":"5118f5ac5157ed29","entry":"validate","repo":"zbh2047/clipping-algorithms","repo_kind":"official","path":"main_imagenet.py","file_url":"https://github.com/zbh2047/clipping-algorithms/blob/HEAD/main_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5118f5ac5157ed29"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}