{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/don-t-waste-your-time-early-stopping-cross","title":"Don't Waste Your Time: Early Stopping Cross-Validation","arxiv_id":"2405.03389","date":"2024-05-06","proceeding":null,"authors":["Edward Bergman","Lennart Purucker","Frank Hutter"],"abstract":"State-of-the-art automated machine learning systems for tabular data often employ cross-validation; ensuring that measured performances generalize to unseen data, or that subsequent ensembling does not overfit. However, using k-fold cross-validation instead of holdout validation drastically increases the computational cost of validating a single configuration. While ensuring better generalization and, by extension, better performance, the additional cost is often prohibitive for effective model selection within a time budget. We aim to make model selection with cross-validation more effective. Therefore, we study early stopping the process of cross-validation during model selection. We investigate the impact of early stopping on random search for two algorithms, MLP and random forest, across 36 classification datasets. We further analyze the impact of the number of folds by considering 3-, 5-, and 10-folds. In addition, we investigate the impact of early stopping with Bayesian optimization instead of random search and also repeated cross-validation. Our exploratory study shows that even a simple-to-understand and easy-to-implement method consistently allows model selection to converge faster; in ~94% of all datasets, on average by ~214%. Moreover, stopping cross-validation enables model selection to explore the search space more exhaustively by considering +167% configurations on average within one hour, while also obtaining better overall performance.","url_abs":"https://arxiv.org/abs/2405.03389v2","url_pdf":"https://arxiv.org/pdf/2405.03389v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"don-t-waste-your-time-early-stopping-cross","repo_url":"https://github.com/automl/dontwasteyourtime-early-stopping","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"},{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[{"method_slug":"early-stopping","method_name":"Early Stopping"},{"method_slug":"random-search","method_name":"Random Search"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.03389","atlas_url":"https://app.syntology.ai/?focus=2405.03389","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.03389"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/automl/dontwasteyourtime-early-stopping","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"521590451f559465","entry":"esc","repo":"automl/dontwasteyourtime-early-stopping","repo_kind":"official","path":"generate_searchspaces.py","file_url":"https://github.com/automl/dontwasteyourtime-early-stopping/blob/HEAD/generate_searchspaces.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"521590451f559465"}},{"code_sha256_prefix":"0d7883b0a60445e8","entry":"indent","repo":"automl/dontwasteyourtime-early-stopping","repo_kind":"official","path":"generate_searchspaces.py","file_url":"https://github.com/automl/dontwasteyourtime-early-stopping/blob/HEAD/generate_searchspaces.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0d7883b0a60445e8"}},{"code_sha256_prefix":"6aee015577fb1c8f","entry":"mlp_config_transform","repo":"automl/dontwasteyourtime-early-stopping","repo_kind":"official","path":"src/exps/pipelines.py","file_url":"https://github.com/automl/dontwasteyourtime-early-stopping/blob/HEAD/src/exps/pipelines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6aee015577fb1c8f"}},{"code_sha256_prefix":"2bb20ead542bd9ea","entry":"ready_for_l1_dist","repo":"automl/dontwasteyourtime-early-stopping","repo_kind":"official","path":"src/exps/footprint.py","file_url":"https://github.com/automl/dontwasteyourtime-early-stopping/blob/HEAD/src/exps/footprint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2bb20ead542bd9ea"}},{"code_sha256_prefix":"d87b93265c4151c4","entry":"rf_config_transform","repo":"automl/dontwasteyourtime-early-stopping","repo_kind":"official","path":"src/exps/pipelines.py","file_url":"https://github.com/automl/dontwasteyourtime-early-stopping/blob/HEAD/src/exps/pipelines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d87b93265c4151c4"}},{"code_sha256_prefix":"c9e8014f8c628603","entry":"wrap","repo":"automl/dontwasteyourtime-early-stopping","repo_kind":"official","path":"generate_searchspaces.py","file_url":"https://github.com/automl/dontwasteyourtime-early-stopping/blob/HEAD/generate_searchspaces.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c9e8014f8c628603"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}