{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-pac-bayesian-bounds-for-random-forests","title":"On PAC-Bayesian Bounds for Random Forests","arxiv_id":"1810.09746","date":"2018-10-23","proceeding":null,"authors":["Stephan Sloth Lorenzen","Christian Igel","Yevgeny Seldin"],"abstract":"Existing guarantees in terms of rigorous upper bounds on the generalization\nerror for the original random forest algorithm, one of the most frequently used\nmachine learning methods, are unsatisfying. We discuss and evaluate various\nPAC-Bayesian approaches to derive such bounds. The bounds do not require\nadditional hold-out data, because the out-of-bag samples from the bagging in\nthe training process can be exploited. A random forest predicts by taking a\nmajority vote of an ensemble of decision trees. The first approach is to bound\nthe error of the vote by twice the error of the corresponding Gibbs classifier\n(classifying with a single member of the ensemble selected at random). However,\nthis approach does not take into account the effect of averaging out of errors\nof individual classifiers when taking the majority vote. This effect provides a\nsignificant boost in performance when the errors are independent or negatively\ncorrelated, but when the correlations are strong the advantage from taking the\nmajority vote is small. The second approach based on PAC-Bayesian C-bounds\ntakes dependencies between ensemble members into account, but it requires\nestimating correlations between the errors of the individual classifiers. When\nthe correlations are high or the estimation is poor, the bounds degrade. In our\nexperiments, we compute generalization bounds for random forests on various\nbenchmark data sets. Because the individual decision trees already perform\nwell, their predictions are highly correlated and the C-bounds do not lead to\nsatisfactory results. For the same reason, the bounds based on the analysis of\nGibbs classifiers are typically superior and often reasonably tight. Bounds\nbased on a validation set coming at the cost of a smaller training set gave\nbetter performance guarantees, but worse performance in most experiments.","url_abs":"http://arxiv.org/abs/1810.09746v2","url_pdf":"http://arxiv.org/pdf/1810.09746v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-pac-bayesian-bounds-for-random-forests","repo_url":"https://github.com/stephanlorenzen/majorityvotebounds","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1810.09746","atlas_url":"https://app.syntology.ai/?focus=1810.09746","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}