{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-benefits-of-reusing-batches-for-gradient","title":"The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents","arxiv_id":"2402.03220","date":"2024-02-05","proceeding":null,"authors":["Yatin Dandi","Emanuele Troiani","Luca Arnaboldi","Luca Pesce","Lenka Zdeborová","Florent Krzakala"],"abstract":"We investigate the training dynamics of two-layer neural networks when learning multi-index target functions. We focus on multi-pass gradient descent (GD) that reuses the batches multiple times and show that it significantly changes the conclusion about which functions are learnable compared to single-pass gradient descent. In particular, multi-pass GD with finite stepsize is found to overcome the limitations of gradient flow and single-pass GD given by the information exponent (Ben Arous et al., 2021) and leap exponent (Abbe et al., 2023) of the target function. We show that upon re-using batches, the network achieves in just two time steps an overlap with the target subspace even for functions not satisfying the staircase property (Abbe et al., 2021). We characterize the (broad) class of functions efficiently learned in finite time. The proof of our results is based on the analysis of the Dynamical Mean-Field Theory (DMFT). We further provide a closed-form description of the dynamical process of the low-dimensional projections of the weights, and numerical experiments illustrating the theory.","url_abs":"https://arxiv.org/abs/2402.03220v3","url_pdf":"https://arxiv.org/pdf/2402.03220v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-benefits-of-reusing-batches-for-gradient","repo_url":"https://github.com/idephics/benefit-reusing-batch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.03220","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.03220"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/IdePHICS/benefit-reusing-batch","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/idephics/benefit-reusing-batch","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran":4,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"ee4033ae07851e27","entry":"loss","repo":"IdePHICS/benefit-reusing-batch","repo_kind":"official","path":"dmft.py","file_url":"https://github.com/IdePHICS/benefit-reusing-batch/blob/HEAD/dmft.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee4033ae07851e27"}},{"code_sha256_prefix":"7f257a6a2b02c83f","entry":"loss_func","repo":"idephics/benefit-reusing-batch","repo_kind":"official","path":"simulations_PyTorch.py","file_url":"https://github.com/idephics/benefit-reusing-batch/blob/HEAD/simulations_PyTorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f257a6a2b02c83f"}},{"code_sha256_prefix":"93200779916e345f","entry":"net_student","repo":"idephics/benefit-reusing-batch","repo_kind":"official","path":"simulation.py","file_url":"https://github.com/idephics/benefit-reusing-batch/blob/HEAD/simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93200779916e345f"}},{"code_sha256_prefix":"f62072dd733dd928","entry":"net_student_derivative","repo":"idephics/benefit-reusing-batch","repo_kind":"official","path":"simulation.py","file_url":"https://github.com/idephics/benefit-reusing-batch/blob/HEAD/simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f62072dd733dd928"}},{"code_sha256_prefix":"c187b45d37555154","entry":"net_teacher","repo":"idephics/benefit-reusing-batch","repo_kind":"official","path":"simulation.py","file_url":"https://github.com/idephics/benefit-reusing-batch/blob/HEAD/simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c187b45d37555154"}},{"code_sha256_prefix":"a1a941200969dd90","entry":"student","repo":"IdePHICS/benefit-reusing-batch","repo_kind":"official","path":"dmft.py","file_url":"https://github.com/IdePHICS/benefit-reusing-batch/blob/HEAD/dmft.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1a941200969dd90"}},{"code_sha256_prefix":"ffbf5a1105a678ef","entry":"teacher","repo":"IdePHICS/benefit-reusing-batch","repo_kind":"official","path":"dmft.py","file_url":"https://github.com/IdePHICS/benefit-reusing-batch/blob/HEAD/dmft.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffbf5a1105a678ef"}},{"code_sha256_prefix":"a8d365268bb9ddbb","entry":"set_weight","repo":"idephics/benefit-reusing-batch","repo_kind":"official","path":"simulations_PyTorch.py","file_url":"https://github.com/idephics/benefit-reusing-batch/blob/HEAD/simulations_PyTorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8d365268bb9ddbb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}