{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/subsampling-is-not-magic-why-large-batch","title":"Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation","arxiv_id":"2402.03990","date":"2024-02-06","proceeding":null,"authors":["Ossi Räisä","Joonas Jälkö","Antti Honkela"],"abstract":"We study how the batch size affects the total gradient variance in differentially private stochastic gradient descent (DP-SGD), seeking a theoretical explanation for the usefulness of large batch sizes. As DP-SGD is the basis of modern DP deep learning, its properties have been widely studied, and recent works have empirically found large batch sizes to be beneficial. However, theoretical explanations of this benefit are currently heuristic at best. We first observe that the total gradient variance in DP-SGD can be decomposed into subsampling-induced and noise-induced variances. We then prove that in the limit of an infinite number of iterations, the effective noise-induced variance is invariant to the batch size. The remaining subsampling-induced variance decreases with larger batch sizes, so large batches reduce the effective total gradient variance. We confirm numerically that the asymptotic regime is relevant in practical settings when the batch size is not small, and find that outside the asymptotic regime, the total gradient variance decreases even more with large batch sizes. We also find a sufficient condition that implies that large batch sizes similarly reduce effective DP noise variance for one iteration of DP-SGD.","url_abs":"https://arxiv.org/abs/2402.03990v3","url_pdf":"https://arxiv.org/pdf/2402.03990v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"subsampling-is-not-magic-why-large-batch","repo_url":"https://github.com/dpbayes/subsampling-is-not-magic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.03990","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.03990"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/DPBayes/subsampling-is-not-magic","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dpbayes/subsampling-is-not-magic","reach":{"status":"ok"}}],"summary":{"ran_violates":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"f90b392ae85c587e","entry":"derf","repo":"DPBayes/subsampling-is-not-magic","repo_kind":"official","path":"src/plot_a_vs_b.py","file_url":"https://github.com/DPBayes/subsampling-is-not-magic/blob/HEAD/src/plot_a_vs_b.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f90b392ae85c587e"}},{"code_sha256_prefix":"795e223528fcff8d","entry":"find_no_subsampling_sigma","repo":"DPBayes/subsampling-is-not-magic","repo_kind":"official","path":"src/plot_a_vs_b.py","file_url":"https://github.com/DPBayes/subsampling-is-not-magic/blob/HEAD/src/plot_a_vs_b.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"795e223528fcff8d"}},{"code_sha256_prefix":"7333d176de493fb5","entry":"single_step_subsampled_delta","repo":"DPBayes/subsampling-is-not-magic","repo_kind":"official","path":"src/plot_a_vs_b.py","file_url":"https://github.com/DPBayes/subsampling-is-not-magic/blob/HEAD/src/plot_a_vs_b.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7333d176de493fb5"}},{"code_sha256_prefix":"124a2f3b2c681b57","entry":"wrap_defaultdict","repo":"dpbayes/subsampling-is-not-magic","repo_kind":"official","path":"src/plot_convergence.py","file_url":"https://github.com/dpbayes/subsampling-is-not-magic/blob/HEAD/src/plot_convergence.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"124a2f3b2c681b57"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}