{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/performance-evaluation-and-hyperparameter","title":"Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data","arxiv_id":"1803.11266","date":"2018-03-29","proceeding":null,"authors":["Patrick Schratz","Jannes Muenchow","Eugenia Iturritxa","Jakob Richter","Alexander Brenning"],"abstract":"Machine-learning algorithms have gained popularity in recent years in the\nfield of ecological modeling due to their promising results in predictive\nperformance of classification problems. While the application of such\nalgorithms has been highly simplified in the last years due to their\nwell-documented integration in commonly used statistical programming languages\nsuch as R, there are several practical challenges in the field of ecological\nmodeling related to unbiased performance estimation, optimization of algorithms\nusing hyperparameter tuning and spatial autocorrelation. We address these\nissues in the comparison of several widely used machine-learning algorithms\nsuch as Boosted Regression Trees (BRT), k-Nearest Neighbor (WKNN), Random\nForest (RF) and Support Vector Machine (SVM) to traditional parametric\nalgorithms such as logistic regression (GLM) and semi-parametric ones like\ngeneralized additive models (GAM). Different nested cross-validation methods\nincluding hyperparameter tuning methods are used to evaluate model performances\nwith the aim to receive bias-reduced performance estimates. As a case study the\nspatial distribution of forest disease Diplodia sapinea in the Basque Country\nin Spain is investigated using common environmental variables such as\ntemperature, precipitation, soil or lithology as predictors. Results show that\nGAM and RF (mean AUROC estimates 0.708 and 0.699) outperform all other methods\nin predictive accuracy. The effect of hyperparameter tuning saturates at around\n50 iterations for this data set. The AUROC differences between the bias-reduced\n(spatial cross-validation) and overoptimistic (non-spatial cross-validation)\nperformance estimates of the GAM and RF are 0.167 (24%) and 0.213 (30%),\nrespectively. It is recommended to also use spatial partitioning for\ncross-validation hyperparameter tuning of spatial data.","url_abs":"http://arxiv.org/abs/1803.11266v1","url_pdf":"http://arxiv.org/pdf/1803.11266v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"performance-evaluation-and-hyperparameter","repo_url":"https://github.com/pat-s/pathogen-modeling","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"additive-models","task_name":"Additive models"},{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"gam","method_name":"GAM"},{"method_slug":"logistic-regression","method_name":"Logistic Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}