{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-16761","title":"Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets","arxiv_id":"2607.16761","date":"2026-07-18","proceeding":null,"authors":["Javier Maass","Lénaïc Chizat"],"abstract":"Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leaves the forward pass unchanged and instead masks the gradients. In particular, the noise induced by RaM in the parameter updates is unbiased, so standard explanations for the effectiveness of dropout, such as the penalization effect or the prevention of co-adaptation between neurons, do not apply to RaM. In this work, we show that the difference between the two methods disappears for ResNets in the large depth and width asymptotics: in the complete feature learning regime, they both converge to the same large-scale limiting dynamics. This asymptotic equivalence holds for several variants of dropout and RaM, including layerwise dropout as used in stochastic-depth ResNets, albeit at slower quantitative rates. In fact, we also show that several of these variants collapse to the same limit asymptotically.","url_abs":"https://arxiv.org/abs/2607.16761","url_pdf":"https://arxiv.org/pdf/2607.16761","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.16761","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.16761"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/xavimaass/dropout_and_ram_2026","reach":null}],"summary":{"ran":11,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"fc7c880afd73a5d0","entry":"cross_entropy_from_logits","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/losses.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc7c880afd73a5d0"}},{"code_sha256_prefix":"a99dc95f2d0d489b","entry":"init_params","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/model.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a99dc95f2d0d489b"}},{"code_sha256_prefix":"830b55776d61e942","entry":"lr_scales_standard","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/training.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"830b55776d61e942"}},{"code_sha256_prefix":"c12a1585279f96f6","entry":"make_metrics_fn","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/training.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c12a1585279f96f6"}},{"code_sha256_prefix":"81ffb998d09ea849","entry":"quadratic_mean_error","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/losses.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81ffb998d09ea849"}},{"code_sha256_prefix":"db36ceba25a2140a","entry":"quadratic_mean_error_derivative","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/losses.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db36ceba25a2140a"}},{"code_sha256_prefix":"bb3e244596afa241","entry":"relu","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/activations.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb3e244596afa241"}},{"code_sha256_prefix":"a40f5d87095f7973","entry":"relu_derivative","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/activations.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a40f5d87095f7973"}},{"code_sha256_prefix":"ae65958e8b3a4289","entry":"sample_internal_dropout_mask","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/dropout_masks.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/dropout_masks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae65958e8b3a4289"}},{"code_sha256_prefix":"f8d1df13d5fcd3b6","entry":"sample_zero_mask","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/dropout_masks.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/dropout_masks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8d1df13d5fcd3b6"}},{"code_sha256_prefix":"d3fcb3822b7cde96","entry":"tanh","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/activations.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3fcb3822b7cde96"}},{"code_sha256_prefix":"b3273e4c36bfb863","entry":"make_dataset_mnist","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3273e4c36bfb863"}},{"code_sha256_prefix":"dfb15926269d7334","entry":"sample_dropout_mask","repo":"xavimaass/dropout_and_ram_2026","repo_kind":"found_in_text","path":"src/jax_resnet/dropout_masks.py","file_url":"https://github.com/xavimaass/dropout_and_ram_2026/blob/HEAD/src/jax_resnet/dropout_masks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfb15926269d7334"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}