{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/statistical-comparison-of-classifiers-through","title":"Statistical comparison of classifiers through Bayesian hierarchical modelling","arxiv_id":"1609.08905","date":"2016-09-28","proceeding":null,"authors":["Giorgio Corani","Alessio Benavoli","Janez Demšar","Francesca Mangili","Marco Zaffalon"],"abstract":"Usually one compares the accuracy of two competing classifiers via null\nhypothesis significance tests (nhst). Yet the nhst tests suffer from important\nshortcomings, which can be overcome by switching to Bayesian hypothesis\ntesting. We propose a Bayesian hierarchical model which jointly analyzes the\ncross-validation results obtained by two classifiers on multiple data sets. It\nreturns the posterior probability of the accuracies of the two classifiers\nbeing practically equivalent or significantly different. A further strength of\nthe hierarchical model is that, by jointly analyzing the results obtained on\nall data sets, it reduces the estimation error compared to the usual approach\nof averaging the cross-validation results obtained on a given data set.","url_abs":"http://arxiv.org/abs/1609.08905v3","url_pdf":"http://arxiv.org/pdf/1609.08905v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"statistical-comparison-of-classifiers-through","repo_url":"https://github.com/BayesianTestsML/tutorial","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"hypothesis-testing","task_name":"Two-sample testing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1609.08905","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}