{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/latent-state-models-of-training-dynamics","title":"Latent State Models of Training Dynamics","arxiv_id":"2308.09543","date":"2023-08-18","proceeding":null,"authors":["Michael Y. Hu","Angelica Chen","Naomi Saphra","Kyunghyun Cho"],"abstract":"The impact of randomness on model training is poorly understood. How do differences in data order and initialization actually manifest in the model, such that some training runs outperform others or converge faster? Furthermore, how can we interpret the resulting training dynamics and the phase transitions that characterize different trajectories? To understand the effect of randomness on the dynamics and outcomes of neural network training, we train models multiple times with different random seeds and compute a variety of metrics throughout training, such as the $L_2$ norm, mean, and variance of the neural network's weights. We then fit a hidden Markov model (HMM) over the resulting sequences of metrics. The HMM represents training as a stochastic process of transitions between latent states, providing an intuitive overview of significant changes during training. Using our method, we produce a low-dimensional, discrete representation of training dynamics on grokking tasks, image classification, and masked language modeling. We use the HMM representation to study phase transitions and identify latent \"detour\" states that slow down convergence.","url_abs":"https://arxiv.org/abs/2308.09543v3","url_pdf":"https://arxiv.org/pdf/2308.09543v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"latent-state-models-of-training-dynamics","repo_url":"https://github.com/michahu/modeling-training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.09543","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.09543"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/michahu/modeling-training","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a5330d2e9abcc8f6","entry":"distance","repo":"michahu/modeling-training","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5330d2e9abcc8f6"}},{"code_sha256_prefix":"62de75d5b60ef7ed","entry":"get_markov_chain","repo":"michahu/modeling-training","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62de75d5b60ef7ed"}},{"code_sha256_prefix":"f643b738733925a8","entry":"make_hmm_data","repo":"michahu/modeling-training","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f643b738733925a8"}},{"code_sha256_prefix":"82b70fe3a513f175","entry":"unpack_biases_transformer","repo":"michahu/modeling-training","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"82b70fe3a513f175"}},{"code_sha256_prefix":"109a27e98c8e8ed7","entry":"unpack_weights_transformer","repo":"michahu/modeling-training","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"109a27e98c8e8ed7"}},{"code_sha256_prefix":"b835b29572e2af0e","entry":"get_stats_for_run","repo":"michahu/modeling-training","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b835b29572e2af0e"}},{"code_sha256_prefix":"d2caf6b967cb24fb","entry":"parse_args","repo":"michahu/modeling-training","repo_kind":"official","path":"src/compute_cka.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/compute_cka.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2caf6b967cb24fb"}},{"code_sha256_prefix":"79a7f3d9dbd168e1","entry":"patch_attention","repo":"michahu/modeling-training","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/michahu/modeling-training/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79a7f3d9dbd168e1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}