{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/characterizing-learning-curves-during","title":"Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability","arxiv_id":"2308.15419","date":"2023-08-29","proceeding":null,"authors":["Tyler A. Chang","Zhuowen Tu","Benjamin K. Bergen"],"abstract":"How do language models learn to make predictions during pre-training? To study this, we extract learning curves from five autoregressive English language model pre-training runs, for 1M unseen tokens in context. We observe that the language models generate short repetitive phrases before learning to generate longer and more coherent text. We also find that individual tokens often exhibit sudden increases or decreases in loss that are surprisingly consistent across pre-training runs. To better understand these fluctuations, we quantify the final surprisal, within-run variability, age of acquisition, forgettability, and cross-run variability of learning curves for individual tokens in context. More frequent tokens reach lower final surprisals, exhibit less variability within and across pre-training runs, are learned earlier, and are less likely to be \"forgotten\" during pre-training. Higher n-gram probabilities further accentuate these effects. Independent of the target token, shorter and more frequent contexts correlate with marginally more stable and quickly acquired predictions. Based on our results, we argue for the existence of sequential learning dependencies between different model capabilities, and we characterize language model learning as early n-gram learning before gradual refinement of tail n-gram predictions.","url_abs":"https://arxiv.org/abs/2308.15419v2","url_pdf":"https://arxiv.org/pdf/2308.15419v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"characterizing-learning-curves-during","repo_url":"https://github.com/tylerachang/lm-learning-curves","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.15419","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.15419"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tylerachang/lm-learning-curves","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"9f721eccfb89aaf4","entry":"get_average_features_dataframe","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"utils/annotator.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/utils/annotator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f721eccfb89aaf4"}},{"code_sha256_prefix":"2b817396b34bfb65","entry":"get_checkpoints","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"utils/data_utils.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b817396b34bfb65"}},{"code_sha256_prefix":"7e801bca91de63e1","entry":"get_correlations","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"analysis/generate_figures.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/analysis/generate_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e801bca91de63e1"}},{"code_sha256_prefix":"71ab3c3ebeb19aed","entry":"get_examples","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"utils/data_utils.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71ab3c3ebeb19aed"}},{"code_sha256_prefix":"232c49070110a903","entry":"get_hidden_states","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"232c49070110a903"}},{"code_sha256_prefix":"7df5ccdf9ea9cb93","entry":"prepare_tokenized_examples","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7df5ccdf9ea9cb93"}},{"code_sha256_prefix":"93f43924b399d93e","entry":"get_crossrun_variability","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"utils/annotator.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/utils/annotator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93f43924b399d93e"}},{"code_sha256_prefix":"bbfe378f04bfe631","entry":"get_features_dataframe","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"utils/annotator.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/utils/annotator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bbfe378f04bfe631"}},{"code_sha256_prefix":"b7ce496e8bd1b24d","entry":"get_rank","repo":"tylerachang/lm-learning-curves","repo_kind":"official","path":"analysis/run_crossrun_similarity.py","file_url":"https://github.com/tylerachang/lm-learning-curves/blob/HEAD/analysis/run_crossrun_similarity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7ce496e8bd1b24d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}