{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dichotomy-of-early-and-late-phase-implicit","title":"Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking","arxiv_id":"2311.18817","date":"2023-11-30","proceeding":null,"authors":["Kaifeng Lyu","Jikai Jin","Zhiyuan Li","Simon S. Du","Jason D. Lee","Wei Hu"],"abstract":"Recent work by Power et al. (2022) highlighted a surprising \"grokking\" phenomenon in learning arithmetic tasks: a neural net first \"memorizes\" the training set, resulting in perfect training accuracy but near-random test accuracy, and after training for sufficiently longer, it suddenly transitions to perfect test accuracy. This paper studies the grokking phenomenon in theoretical setups and shows that it can be induced by a dichotomy of early and late phase implicit biases. Specifically, when training homogeneous neural nets with large initialization and small weight decay on both classification and regression tasks, we prove that the training process gets trapped at a solution corresponding to a kernel predictor for a long time, and then a very sharp transition to min-norm/max-margin predictors occurs, leading to a dramatic change in test accuracy.","url_abs":"https://arxiv.org/abs/2311.18817v2","url_pdf":"https://arxiv.org/pdf/2311.18817v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dichotomy-of-early-and-late-phase-implicit","repo_url":"https://github.com/vfleaking/grokking-dichotomy","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.18817","atlas_url":"https://app.syntology.ai/?focus=2311.18817","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.18817"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vfleaking/grokking-dichotomy","reach":null}],"summary":{"ran_honours":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"84b0e4fd98457360","entry":"logistic_loss","repo":"vfleaking/grokking-dichotomy","repo_kind":"official","path":"train_diag_cls.py","file_url":"https://github.com/vfleaking/grokking-dichotomy/blob/HEAD/train_diag_cls.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84b0e4fd98457360"}},{"code_sha256_prefix":"8be0e28fe36e5414","entry":"separate_logits","repo":"vfleaking/grokking-dichotomy","repo_kind":"official","path":"train_mod_add.py","file_url":"https://github.com/vfleaking/grokking-dichotomy/blob/HEAD/train_mod_add.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8be0e28fe36e5414"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}