{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-05786","title":"Selecting Hyperparameters for Tree-Boosting","arxiv_id":"2602.05786","date":"2026-02-05","proceeding":null,"authors":["Floris Jan Koster","Fabio Sigrist"],"abstract":"Tree-boosting is a widely used machine learning technique for tabular data. However, its out-of-sample accuracy is critically dependent on multiple hyperparameters. In this article, we empirically compare several popular methods for hyperparameter optimization for tree-boosting including random grid search, the tree-structured Parzen estimator (TPE), Gaussian-process-based Bayesian optimization (GP-BO), Hyperband, the sequential model-based algorithm configuration (SMAC) method, and deterministic full grid search using $59$ regression and binary classification data sets. We find that the SMAC method clearly outperforms all the other considered methods on average, and it gives stable performance across a diverse collection of tabular data sets under a fixed tuning budget, which is relevant for users who cannot afford extensive manual trial-and-error tuning. We further observe that (i) a relatively large number of trials larger than $100$ is typically required for accurate tuning, (ii) using default values for hyperparameters or a full search over a small grid often yields very inaccurate models, (iii) all considered hyperparameters can have a material effect on the accuracy of tree-boosting, i.e., there is no small set of hyperparameters that is more important than others, and (iv) choosing the number of boosting iterations using early stopping yields more accurate results compared to including it in the search space for regression tasks.","url_abs":"https://arxiv.org/abs/2602.05786","url_pdf":"https://arxiv.org/pdf/2602.05786","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.05786","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.05786"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/fl0risk/HPOTreeBoosting","reach":null}],"summary":{"ran":1,"unverified":10},"by_repo_kind":{"found_in_text":{"samples":11,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"2076b13e5ea7eb87","entry":"is_numeric","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2076b13e5ea7eb87"}},{"code_sha256_prefix":"2c627ea7f67c2cd9","entry":"create_param_space","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"model_gp_boosting.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/model_gp_boosting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c627ea7f67c2cd9"}},{"code_sha256_prefix":"2096e629050ebe6b","entry":"create_scores_dict","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_plot_maker.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_plot_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2096e629050ebe6b"}},{"code_sha256_prefix":"7b16ee6ae995a6d3","entry":"create_scores_dict","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_plots_comparison.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_plots_comparison.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b16ee6ae995a6d3"}},{"code_sha256_prefix":"ff72789bce455a1c","entry":"create_scores_dict_joint_tuning","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_plots_comparison.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_plots_comparison.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff72789bce455a1c"}},{"code_sha256_prefix":"0584b77c04b73e70","entry":"get_statistics","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_plot_maker.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_plot_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0584b77c04b73e70"}},{"code_sha256_prefix":"bb28afdcbfebc88d","entry":"is_1d_list","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb28afdcbfebc88d"}},{"code_sha256_prefix":"743b09f8b88855bf","entry":"normalize_scores","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_plot_maker.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_plot_maker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"743b09f8b88855bf"}},{"code_sha256_prefix":"84816301a3b34104","entry":"normalize_scores","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_plots_comparison.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_plots_comparison.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84816301a3b34104"}},{"code_sha256_prefix":"6d0cbe1f00d69999","entry":"read_and_preprocess_dataset","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_modelling.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_modelling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d0cbe1f00d69999"}},{"code_sha256_prefix":"ceac6415cc77626f","entry":"split_dataset","repo":"fl0risk/HPOTreeBoosting","repo_kind":"found_in_text","path":"utils_modelling.py","file_url":"https://github.com/fl0risk/HPOTreeBoosting/blob/HEAD/utils_modelling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ceac6415cc77626f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}