{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/shade-information-based-regularization-for","title":"SHADE: Information-Based Regularization for Deep Learning","arxiv_id":"1805.05814","date":"2018-05-14","proceeding":null,"authors":["Michael Blot","Thomas Robert","Nicolas Thome","Matthieu Cord"],"abstract":"Regularization is a big issue for training deep neural networks. In this\npaper, we propose a new information-theory-based regularization scheme named\nSHADE for SHAnnon DEcay. The originality of the approach is to define a prior\nbased on conditional entropy, which explicitly decouples the learning of\ninvariant representations in the regularizer and the learning of correlations\nbetween inputs and labels in the data fitting term. Our second contribution is\nto derive a stochastic version of the regularizer compatible with deep\nlearning, resulting in a tractable training scheme. We empirically validate the\nefficiency of our approach to improve classification performances compared to\nstandard regularization schemes on several standard architectures.","url_abs":"http://arxiv.org/abs/1805.05814v1","url_pdf":"http://arxiv.org/pdf/1805.05814v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"shade-information-based-regularization-for","repo_url":"https://github.com/ThomasRobertFr/deep-learning-figures","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"classification","task_name":"General Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.05814","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}