{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/drmad-distilling-reverse-mode-automatic","title":"DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks","arxiv_id":"1601.00917","date":"2016-01-05","proceeding":null,"authors":["Jie Fu","Hongyin Luo","Jiashi Feng","Kian Hsiang Low","Tat-Seng Chua"],"abstract":"The performance of deep neural networks is well-known to be sensitive to the\nsetting of their hyperparameters. Recent advances in reverse-mode automatic\ndifferentiation allow for optimizing hyperparameters with gradients. The\nstandard way of computing these gradients involves a forward and backward pass\nof computations. However, the backward pass usually needs to consume\nunaffordable memory to store all the intermediate variables to exactly reverse\nthe forward training procedure. In this work we propose a simple but effective\nmethod, DrMAD, to distill the knowledge of the forward pass into a shortcut\npath, through which we approximately reverse the training trajectory.\nExperiments on several image benchmark datasets show that DrMAD is at least 45\ntimes faster and consumes 100 times less memory compared to state-of-the-art\nmethods for optimizing hyperparameters with minimal compromise to its\neffectiveness. To the best of our knowledge, DrMAD is the first research\nattempt to make it practical to automatically tune thousands of hyperparameters\nof deep neural networks. The code can be downloaded from\nhttps://github.com/bigaidream-projects/drmad","url_abs":"http://arxiv.org/abs/1601.00917v5","url_pdf":"http://arxiv.org/pdf/1601.00917v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"drmad-distilling-reverse-mode-automatic","repo_url":"https://github.com/bigaidream-projects/drmad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1601.00917","atlas_url":"https://app.syntology.ai/?focus=1601.00917","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}