{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gradnorm-gradient-normalization-for-adaptive","title":"GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks","arxiv_id":"1711.02257","date":"2017-11-07","proceeding":"ICML 2018 7","authors":["Zhao Chen","Vijay Badrinarayanan","Chen-Yu Lee","Andrew Rabinovich"],"abstract":"Deep multitask networks, in which one neural network produces multiple\npredictive outputs, can offer better speed and performance than their\nsingle-task counterparts but are challenging to train properly. We present a\ngradient normalization (GradNorm) algorithm that automatically balances\ntraining in deep multitask models by dynamically tuning gradient magnitudes. We\nshow that for various network architectures, for both regression and\nclassification tasks, and on both synthetic and real datasets, GradNorm\nimproves accuracy and reduces overfitting across multiple tasks when compared\nto single-task networks, static baselines, and other adaptive multitask loss\nbalancing techniques. GradNorm also matches or surpasses the performance of\nexhaustive grid search methods, despite only involving a single asymmetry\nhyperparameter $\\alpha$. Thus, what was once a tedious search process that\nincurred exponentially more compute for each task added can now be accomplished\nwithin a few training runs, irrespective of the number of tasks. Ultimately, we\nwill demonstrate that gradient manipulation affords us great control over the\ntraining dynamics of multitask networks and may be one of the keys to unlocking\nthe potential of multitask learning.","url_abs":"http://arxiv.org/abs/1711.02257v4","url_pdf":"http://arxiv.org/pdf/1711.02257v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gradnorm-gradient-normalization-for-adaptive","repo_url":"https://github.com/VICO-UoE/KD4MTL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gradnorm-gradient-normalization-for-adaptive","repo_url":"https://github.com/choltz95/MTGP-NN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"gradnorm-gradient-normalization-for-adaptive","repo_url":"https://github.com/hav4ik/Hydra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gradnorm-gradient-normalization-for-adaptive","repo_url":"https://github.com/AvivNavon/AuxiLearn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.02257","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}