{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-validation-with-confidence","title":"Cross-Validation with Confidence","arxiv_id":"1703.07904","date":"2017-03-23","proceeding":null,"authors":["Jing Lei"],"abstract":"Cross-validation is one of the most popular model selection methods in\nstatistics and machine learning. Despite its wide applicability, traditional\ncross validation methods tend to select overfitting models, due to the\nignorance of the uncertainty in the testing sample. We develop a new,\nstatistically principled inference tool based on cross-validation that takes\ninto account the uncertainty in the testing sample. This new method outputs a\nset of highly competitive candidate models containing the best one with\nguaranteed probability. As a consequence, our method can achieve consistent\nvariable selection in a classical linear regression setting, for which existing\ncross-validation methods require unconventional split ratios. When used for\nregularizing tuning parameter selection, the method can provide a further\ntrade-off between prediction accuracy and model interpretability. We\ndemonstrate the performance of the proposed method in several simulated and\nreal data examples.","url_abs":"http://arxiv.org/abs/1703.07904v2","url_pdf":"http://arxiv.org/pdf/1703.07904v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-validation-with-confidence","repo_url":"https://github.com/tim-coleman/CVC_Caret","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"variable-selection","task_name":"Variable Selection"}],"methods":[{"method_slug":"linear-regression","method_name":"Linear Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1703.07904","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}