{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gradient-centralization-a-new-optimization","title":"Gradient Centralization: A New Optimization Technique for Deep Neural Networks","arxiv_id":"2004.01461","date":"2020-04-03","proceeding":"ECCV 2020 8","authors":["Hongwei Yong","Jianqiang Huang","Xian-Sheng Hua","Lei Zhang"],"abstract":"Optimization techniques are of great importance to effectively and efficiently train a deep neural network (DNN). It has been shown that using the first and second order statistics (e.g., mean and variance) to perform Z-score standardization on network activations or weight vectors, such as batch normalization (BN) and weight standardization (WS), can improve the training performance. Different from these existing methods that mostly operate on activations or weights, we present a new optimization technique, namely gradient centralization (GC), which operates directly on gradients by centralizing the gradient vectors to have zero mean. GC can be viewed as a projected gradient descent method with a constrained loss function. We show that GC can regularize both the weight space and output feature space so that it can boost the generalization performance of DNNs. Moreover, GC improves the Lipschitzness of the loss function and its gradient so that the training process becomes more efficient and stable. GC is very simple to implement and can be easily embedded into existing gradient based DNN optimizers with only one line of code. It can also be directly used to fine-tune the pre-trained DNNs. Our experiments on various applications, including general image classification, fine-grained image classification, detection and segmentation, demonstrate that GC can consistently improve the performance of DNN learning. The code of GC can be found at https://github.com/Yonghongwei/Gradient-Centralization.","url_abs":"https://arxiv.org/abs/2004.01461v2","url_pdf":"https://arxiv.org/pdf/2004.01461v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/Yonghongwei/Gradient-Centralization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/HamadYA/GhostFaceNets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/IssamLaradji/sps","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/mhassann22/GCSAM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/mnikitin/Gradient-Centralization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mxnet","reach":{"status":"ok"}},{"paper_slug":"gradient-centralization-a-new-optimization","repo_url":"https://github.com/keras-team/keras-io/blob/master/examples/vision/ipynb/gradient_centralization.ipynb","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"fine-grained-image-classification","task_name":"Fine-Grained Image Classification"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"weight-standardization","method_name":"Weight Standardization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.01461","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.01461"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mnikitin/Gradient-Centralization","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mhassann22/GCSAM","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/IssamLaradji/sps","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keras-team/keras-io/blob/master/examples/vision/ipynb/gradient_centralization.ipynb","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HamadYA/GhostFaceNets","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Yonghongwei/Gradient-Centralization","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran_honours":1,"unverified":27},"by_repo_kind":{"listed":{"samples":29,"ran":2,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"27a3d56a09e0e7cf","entry":"centralized_gradient","repo":"mhassann22/GCSAM","repo_kind":"listed","path":"cifar10_gcsam_resnet50.py","file_url":"https://github.com/mhassann22/GCSAM/blob/HEAD/cifar10_gcsam_resnet50.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27a3d56a09e0e7cf"}},{"code_sha256_prefix":"bd1c72a9d8556505","entry":"to_4d","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"augment.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/augment.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd1c72a9d8556505"}},{"code_sha256_prefix":"619357c6cd6c6727","entry":"activation","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"backbones/ghost_model.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/backbones/ghost_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"619357c6cd6c6727"}},{"code_sha256_prefix":"8f86e7549e63b17d","entry":"adadelta","repo":"Rishit-dagli/Gradient-Centralization-TensorFlow","repo_kind":"listed","path":"gctf/optimizers.py","file_url":"https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/HEAD/gctf/optimizers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8f86e7549e63b17d"}},{"code_sha256_prefix":"a02310673d3edff8","entry":"adagrad","repo":"Rishit-dagli/Gradient-Centralization-TensorFlow","repo_kind":"listed","path":"gctf/optimizers.py","file_url":"https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/HEAD/gctf/optimizers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a02310673d3edff8"}},{"code_sha256_prefix":"719582a083732fb7","entry":"add_l2_regularizer_2_model","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"GhostFaceNets.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/GhostFaceNets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"719582a083732fb7"}},{"code_sha256_prefix":"e8ddfad281bed049","entry":"buildin_models","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"GhostFaceNets.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/GhostFaceNets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8ddfad281bed049"}},{"code_sha256_prefix":"6dbeec44cfb81d9f","entry":"buildin_models","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"GhostFaceNets_with_Bias.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/GhostFaceNets_with_Bias.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6dbeec44cfb81d9f"}},{"code_sha256_prefix":"c3af5327e1b4e163","entry":"calculate_roc","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"evals.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/evals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3af5327e1b4e163"}},{"code_sha256_prefix":"62a2acf5cdbc743f","entry":"centralized_gradient","repo":"lessw2020/Ranger-Deep-Learning-Optimizer","repo_kind":"listed","path":"ranger/ranger2020.py","file_url":"https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer/blob/HEAD/ranger/ranger2020.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62a2acf5cdbc743f"}},{"code_sha256_prefix":"fd031cd7977c1eee","entry":"centralized_gradients_for_optimizer","repo":"Rishit-dagli/Gradient-Centralization-TensorFlow","repo_kind":"listed","path":"gctf/centralized_gradients.py","file_url":"https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/HEAD/gctf/centralized_gradients.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd031cd7977c1eee"}},{"code_sha256_prefix":"4799f292d886421d","entry":"distiller_loss_cosine","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"losses.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4799f292d886421d"}},{"code_sha256_prefix":"82eb5716083ae566","entry":"distiller_loss_euclidean","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"losses.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82eb5716083ae566"}},{"code_sha256_prefix":"95b7e550f0df874b","entry":"face_align_landmark","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"IJB_evals.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/IJB_evals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95b7e550f0df874b"}},{"code_sha256_prefix":"b10fb8ac53c53a97","entry":"from_4d","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"augment.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/augment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b10fb8ac53c53a97"}},{"code_sha256_prefix":"52780979cc1c6e50","entry":"get_centralized_gradients","repo":"Rishit-dagli/Gradient-Centralization-TensorFlow","repo_kind":"listed","path":"gctf/centralized_gradients.py","file_url":"https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/HEAD/gctf/centralized_gradients.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52780979cc1c6e50"}},{"code_sha256_prefix":"2dcab8fa77a49fd8","entry":"ghost_module","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"backbones/ghost_model.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/backbones/ghost_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dcab8fa77a49fd8"}},{"code_sha256_prefix":"b76d78b83d84097c","entry":"half_split_weighted_cosine_similarity","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"evals.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/evals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b76d78b83d84097c"}},{"code_sha256_prefix":"75f14459448a4cef","entry":"half_split_weighted_cosine_similarity_11","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"evals.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/evals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75f14459448a4cef"}},{"code_sha256_prefix":"0d40d2078622e4a1","entry":"keras_model_interf","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"IJB_evals.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/IJB_evals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d40d2078622e4a1"}},{"code_sha256_prefix":"aa04aa5fb992d2bc","entry":"plot_tpr_far","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"eval_folder.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/eval_folder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa04aa5fb992d2bc"}},{"code_sha256_prefix":"b0dc513e1e285780","entry":"random_cutout_or_cutout_mask","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"data.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0dc513e1e285780"}},{"code_sha256_prefix":"c5ac51f2fe4a6729","entry":"read_IJB_meta_columns_to_int","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"IJB_evals.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/IJB_evals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5ac51f2fe4a6729"}},{"code_sha256_prefix":"d1106a8957c8d037","entry":"replace_ReLU_with_PReLU","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"GhostFaceNets.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/GhostFaceNets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1106a8957c8d037"}},{"code_sha256_prefix":"5de5a42ab87d80ec","entry":"replace_add_with_stochastic_depth","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"models.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5de5a42ab87d80ec"}},{"code_sha256_prefix":"ed575936f30efada","entry":"se_module","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"backbones/ghost_model.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/backbones/ghost_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed575936f30efada"}},{"code_sha256_prefix":"36a0054d36a367be","entry":"teacher_model_interf_wrapper","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"data_distiller.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/data_distiller.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36a0054d36a367be"}},{"code_sha256_prefix":"95d626160eefd668","entry":"tf_imread","repo":"HamadYA/GhostFaceNets","repo_kind":"listed","path":"data.py","file_url":"https://github.com/HamadYA/GhostFaceNets/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95d626160eefd668"}},{"code_sha256_prefix":"df2465eb334d9036","entry":"update_optimizer","repo":"Rishit-dagli/Gradient-Centralization-TensorFlow","repo_kind":"listed","path":"gctf/optimizers.py","file_url":"https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/HEAD/gctf/optimizers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df2465eb334d9036"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}