{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-flip-side-of-the-reweighted-coin-duality","title":"The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization","arxiv_id":"2106.07769","date":"2021-06-14","proceeding":"NeurIPS 2021 12","authors":["Daniel LeJeune","Hamid Javadi","Richard G. Baraniuk"],"abstract":"Among the most successful methods for sparsifying deep (neural) networks are those that adaptively mask the network weights throughout training. By examining this masking, or dropout, in the linear case, we uncover a duality between such adaptive methods and regularization through the so-called \"$\\eta$-trick\" that casts both as iteratively reweighted optimizations. We show that any dropout strategy that adapts to the weights in a monotonic way corresponds to an effective subquadratic regularization penalty, and therefore leads to sparse solutions. We obtain the effective penalties for several popular sparsification strategies, which are remarkably similar to classical penalties commonly used in sparse optimization. Considering variational dropout as a case study, we demonstrate similar empirical behavior between the adaptive dropout method and classical methods on the task of deep network sparsification, validating our theory.","url_abs":"https://arxiv.org/abs/2106.07769v3","url_pdf":"https://arxiv.org/pdf/2106.07769v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-flip-side-of-the-reweighted-coin-duality","repo_url":"https://github.com/dlej/adaptive-dropout","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"adaptive-dropout","method_name":"Adaptive Dropout"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"variational-dropout","method_name":"Variational Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.07769","atlas_url":"https://app.syntology.ai/?focus=2106.07769","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.07769"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dlej/adaptive-dropout","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b8e95809ca2c17c9","entry":"sigmoid","repo":"dlej/adaptive-dropout","repo_kind":"official","path":"dropout/penalties.py","file_url":"https://github.com/dlej/adaptive-dropout/blob/HEAD/dropout/penalties.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8e95809ca2c17c9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}