{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-validation-failure-small-sample-sizes","title":"Cross-validation failure: small sample sizes lead to large error bars","arxiv_id":"1706.07581","date":"2017-06-23","proceeding":null,"authors":["Gaël Varoquaux"],"abstract":"Predictive models ground many state-of-the-art developments in statistical\nbrain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers.\nThe principled approach to establish their validity and usefulness is\ncross-validation, testing prediction on unseen data. Here, I would like to\nraise awareness on error bars of cross-validation, which are often\nunderestimated. Simple experiments show that sample sizes of many neuroimaging\nstudies inherently lead to large error bars, eg $\\pm$10% for 100 samples. The\nstandard error across folds strongly underestimates them. These large error\nbars compromise the reliability of conclusions drawn with predictive models,\nsuch as biomarkers or methods developments where, unlike with cognitive\nneuroimaging MVPA approaches, more samples cannot be acquired by repeating the\nexperiment across many subjects. Solutions to increase sample size must be\ninvestigated, tackling possible increases in heterogeneity of the data.","url_abs":"http://arxiv.org/abs/1706.07581v1","url_pdf":"http://arxiv.org/pdf/1706.07581v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-validation-failure-small-sample-sizes","repo_url":"https://github.com/GaelVaroquaux/cross_validation_failure","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1706.07581","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}