{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-implicit-bias-of-depth-how-incremental","title":"The Implicit Bias of Depth: How Incremental Learning Drives Generalization","arxiv_id":"1909.12051","date":"2019-09-26","proceeding":"ICLR 2020 1","authors":["Daniel Gissin","Shai Shalev-Shwartz","Amit Daniely"],"abstract":"A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditions on depth and initialization for which this phenomenon arises in deep linear models. Our main theoretical contribution is a dynamical depth separation result, proving that while shallow models can exhibit incremental learning dynamics, they require the initialization to be exponentially small for these dynamics to present themselves. However, once the model becomes deeper, the dependence becomes polynomial and incremental learning can arise in more natural settings. We complement our theoretical findings by experimenting with deep matrix sensing, quadratic neural networks and with binary classification using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning.","url_abs":"https://arxiv.org/abs/1909.12051v2","url_pdf":"https://arxiv.org/pdf/1909.12051v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-implicit-bias-of-depth-how-incremental","repo_url":"https://github.com/dsgissin/Incremental-Learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"incremental-learning","task_name":"Incremental Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1909.12051","atlas_url":"https://app.syntology.ai/?focus=1909.12051","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.12051"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dsgissin/Incremental-Learning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a4d5376b7b891798","entry":"OMP","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"OMP_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/OMP_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4d5376b7b891798"}},{"code_sha256_prefix":"aa7f2a54f01253aa","entry":"check_sets","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"OMP_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/OMP_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa7f2a54f01253aa"}},{"code_sha256_prefix":"27a76f532debd141","entry":"exp_loss","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"classification_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/classification_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27a76f532debd141"}},{"code_sha256_prefix":"46d7d64c9751bd24","entry":"exponential_acc","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"convolution_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/convolution_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46d7d64c9751bd24"}},{"code_sha256_prefix":"8990c6012616022d","entry":"get_canonical_model","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"classification_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/classification_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8990c6012616022d"}},{"code_sha256_prefix":"0c2b992c73e12273","entry":"get_canonical_model","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"convolution_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/convolution_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c2b992c73e12273"}},{"code_sha256_prefix":"1ff3248281e7f39e","entry":"get_canonical_model","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"matrix_sensing_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/matrix_sensing_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ff3248281e7f39e"}},{"code_sha256_prefix":"7ff75ea941baae2a","entry":"get_sorted_eig","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"quadratic_network_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/quadratic_network_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ff75ea941baae2a"}},{"code_sha256_prefix":"928ad3d19e0e28c9","entry":"run_toy_model","repo":"dsgissin/Incremental-Learning","repo_kind":"official","path":"OMP_experiment.py","file_url":"https://github.com/dsgissin/Incremental-Learning/blob/HEAD/OMP_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"928ad3d19e0e28c9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}