{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-21716","title":"A Defense of the Quadratic Model","arxiv_id":"2607.21716","date":"2026-07-23","proceeding":null,"authors":["Alexandru Meterez","Pranav Ajit Nair","Depen Morwani","Cengiz Pehlevan","Sham Kakade","Alex Damian"],"abstract":"Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics. In this work, we stress test the simplest possible model of optimization -- the quadratic model -- and show that it can be surprisingly predictive in an LLM setting with 150M parameters and 3B training tokens. Specifically, we show that Taylor expanding the model and the loss function at intermediate checkpoints through training can accurately predict the optimization dynamics over windows that can last up to 10\\% of training. Having established this agreement, we then turn to analyzing the structure of these local quadratic optimization problems through two lenses: the Hessian spectrum and local stability. Using Lanczos quadrature with extremely deep probes, we are able to estimate the Hessian spectrum deep into the tail, and we find a surprising amount of structure in both the eigenvalues and eigenvectors, which depends on the batch size, preconditioner, and training time. We also empirically test local linear stability at intermediate checkpoints and compare it to theoretical predictions to demonstrate that optimization in LLMs typically occurs at a stochastic edge of stability, whose nature is also determined by batch size. Our results indicate the quadratic model may be a theoretically tractable proxy for pretraining optimization dynamics.","url_abs":"https://arxiv.org/abs/2607.21716","url_pdf":"https://arxiv.org/pdf/2607.21716","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.21716","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.21716"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/alexandrumeterez/QuadraticModel","reach":null}],"summary":{"ran":14},"by_repo_kind":{"found_in_text":{"samples":14,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2639833fe54827fc","entry":"bool_column","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_training_figures.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_training_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2639833fe54827fc"}},{"code_sha256_prefix":"2fc663a12b313863","entry":"combined_positive","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_spectrum_summary.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_spectrum_summary.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fc663a12b313863"}},{"code_sha256_prefix":"e22c4ee58ab33073","entry":"curve","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_paper_panels.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_paper_panels.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e22c4ee58ab33073"}},{"code_sha256_prefix":"037dff4dbddd7a0e","entry":"displayed_bsz_mults","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_training_figures.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_training_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"037dff4dbddd7a0e"}},{"code_sha256_prefix":"dcf154f8d497bc60","entry":"get_max_flops_sec","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"util.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcf154f8d497bc60"}},{"code_sha256_prefix":"983834bb82d5b6b2","entry":"legend","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_source_conditions.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_source_conditions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"983834bb82d5b6b2"}},{"code_sha256_prefix":"95733eb983e3b0e3","entry":"lerp","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"opt.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/opt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95733eb983e3b0e3"}},{"code_sha256_prefix":"00c8304d153c4195","entry":"linearization_series","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_training_figures.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_training_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00c8304d153c4195"}},{"code_sha256_prefix":"be89a8d6d1fe6ca0","entry":"load","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_sample_size_comparison.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_sample_size_comparison.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be89a8d6d1fe6ca0"}},{"code_sha256_prefix":"1dd8832729f8b64d","entry":"load_curves","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_source_conditions.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_source_conditions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dd8832729f8b64d"}},{"code_sha256_prefix":"f0b5b1f90652a9a7","entry":"load_legacy","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_source_conditions.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_source_conditions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0b5b1f90652a9a7"}},{"code_sha256_prefix":"d6520e0f242ae372","entry":"positive_scale","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_spectrum_summary.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_spectrum_summary.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6520e0f242ae372"}},{"code_sha256_prefix":"cff72abe2634a6bc","entry":"tree_to_dict","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"util.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cff72abe2634a6bc"}},{"code_sha256_prefix":"a1ad40101802c3be","entry":"yfmt","repo":"alexandrumeterez/QuadraticModel","repo_kind":"found_in_text","path":"analysis/plot_paper_panels.py","file_url":"https://github.com/alexandrumeterez/QuadraticModel/blob/HEAD/analysis/plot_paper_panels.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1ad40101802c3be"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}