{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-causes-the-test-error-going-beyond-bias","title":"What causes the test error? Going beyond bias-variance via ANOVA","arxiv_id":"2010.05170","date":"2020-10-11","proceeding":null,"authors":["Licong Lin","Edgar Dobriban"],"abstract":"Modern machine learning methods are often overparametrized, allowing adaptation to the data at a fine level. This can seem puzzling; in the worst case, such models do not need to generalize. This puzzle inspired a great amount of work, arguing when overparametrization reduces test error, in a phenomenon called \"double descent\". Recent work aimed to understand in greater depth why overparametrization is helpful for generalization. This leads to discovering the unimodality of variance as a function of the level of parametrization, and to decomposing the variance into that arising from label noise, initialization, and randomness in the training data to understand the sources of the error. In this work we develop a deeper understanding of this area. Specifically, we propose using the analysis of variance (ANOVA) to decompose the variance in the test error in a symmetric way, for studying the generalization performance of certain two-layer linear and non-linear networks. The advantage of the analysis of variance is that it reveals the effects of initialization, label noise, and training data more clearly than prior approaches. Moreover, we also study the monotonicity and unimodality of the variance components. While prior work studied the unimodality of the overall variance, we study the properties of each term in variance decomposition. One key insight is that in typical settings, the interaction between training samples and initialization can dominate the variance; surprisingly being larger than their marginal effect. Also, we characterize \"phase transitions\" where the variance changes from unimodal to monotone. On a technical level, we leverage advanced deterministic equivalent techniques for Haar random matrices, that -- to our knowledge -- have not yet been used in the area. We also verify our results in numerical simulations and on empirical data examples.","url_abs":"https://arxiv.org/abs/2010.05170v3","url_pdf":"https://arxiv.org/pdf/2010.05170v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-causes-the-test-error-going-beyond-bias","repo_url":"https://github.com/licong-lin/VarianceDecomposition","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.05170","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.05170"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/licong-lin/VarianceDecomposition","reach":null}],"summary":{"ran_fixture":1,"ran_violates":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"7fc44bb0f2b93d9a","entry":"bias_var_mse","repo":"licong-lin/VarianceDecomposition","repo_kind":"official","path":"Numerical_simulations/bvm_anova/bvm_anova.py","file_url":"https://github.com/licong-lin/VarianceDecomposition/blob/HEAD/Numerical_simulations/bvm_anova/bvm_anova.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fc44bb0f2b93d9a"}},{"code_sha256_prefix":"df927e9c226e0d57","entry":"theta_1","repo":"licong-lin/VarianceDecomposition","repo_kind":"official","path":"Numerical_simulations/bvm_anova/bvm_anova.py","file_url":"https://github.com/licong-lin/VarianceDecomposition/blob/HEAD/Numerical_simulations/bvm_anova/bvm_anova.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df927e9c226e0d57"}},{"code_sha256_prefix":"a123de798686cc44","entry":"theta_2","repo":"licong-lin/VarianceDecomposition","repo_kind":"official","path":"Numerical_simulations/bvm_anova/bvm_anova.py","file_url":"https://github.com/licong-lin/VarianceDecomposition/blob/HEAD/Numerical_simulations/bvm_anova/bvm_anova.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a123de798686cc44"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}