{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nested-cross-validation-when-selecting","title":"Nested cross-validation when selecting classifiers is overzealous for most practical applications","arxiv_id":"1809.09446","date":"2018-09-25","proceeding":null,"authors":["Jacques Wainer","Gavin Cawley"],"abstract":"When selecting a classification algorithm to be applied to a particular\nproblem, one has to simultaneously select the best algorithm for that dataset\n\\emph{and} the best set of hyperparameters for the chosen model. The usual\napproach is to apply a nested cross-validation procedure; hyperparameter\nselection is performed in the inner cross-validation, while the outer\ncross-validation computes an unbiased estimate of the expected accuracy of the\nalgorithm \\emph{with cross-validation based hyperparameter tuning}. The\nalternative approach, which we shall call `flat cross-validation', uses a\nsingle cross-validation step both to select the optimal hyperparameter values\nand to provide an estimate of the expected accuracy of the algorithm, that\nwhile biased may nevertheless still be used to select the best learning\nalgorithm. We tested both procedures using 12 different algorithms on 115 real\nlife binary datasets and conclude that using the less computationally expensive\nflat cross-validation procedure will generally result in the selection of an\nalgorithm that is, for all practical purposes, of similar quality to that\nselected via nested cross-validation, provided the learning algorithms have\nrelatively few hyperparameters to be optimised.","url_abs":"http://arxiv.org/abs/1809.09446v1","url_pdf":"http://arxiv.org/pdf/1809.09446v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nested-cross-validation-when-selecting","repo_url":"https://github.com/a-wozniakowski/scikit-physlearn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1809.09446","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}