{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparseout-controlling-sparsity-in-deep","title":"Sparseout: Controlling Sparsity in Deep Networks","arxiv_id":"1904.08050","date":"2019-04-17","proceeding":null,"authors":["Najeeb Khan","Ian Stavness"],"abstract":"Dropout is commonly used to help reduce overfitting in deep neural networks.\nSparsity is a potentially important property of neural networks, but is not\nexplicitly controlled by Dropout-based regularization. In this work, we propose\nSparseout a simple and efficient variant of Dropout that can be used to control\nthe sparsity of the activations in a neural network. We theoretically prove\nthat Sparseout is equivalent to an $L_q$ penalty on the features of a\ngeneralized linear model and that Dropout is a special case of Sparseout for\nneural networks. We empirically demonstrate that Sparseout is computationally\ninexpensive and is able to control the desired level of sparsity in the\nactivations. We evaluated Sparseout on image classification and language\nmodelling tasks to see the effect of sparsity on these tasks. We found that\nsparsity of the activations is favorable for language modelling performance\nwhile image classification benefits from denser activations. Sparseout provides\na way to investigate sparsity in state-of-the-art deep learning models. Source\ncode for Sparseout could be found at\n\\url{https://github.com/najeebkhan/sparseout}.","url_abs":"http://arxiv.org/abs/1904.08050v1","url_pdf":"http://arxiv.org/pdf/1904.08050v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparseout-controlling-sparsity-in-deep","repo_url":"https://github.com/najeebkhan/sparseout","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}