{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assembled-openml-creating-efficient","title":"Assembled-OpenML: Creating Efficient Benchmarks for Ensembles in AutoML with OpenML","arxiv_id":"2307.00285","date":"2023-07-01","proceeding":null,"authors":["Lennart Purucker","Joeran Beel"],"abstract":"Automated Machine Learning (AutoML) frameworks regularly use ensembles. Developers need to compare different ensemble techniques to select appropriate techniques for an AutoML framework from the many potential techniques. So far, the comparison of ensemble techniques is often computationally expensive, because many base models must be trained and evaluated one or multiple times. Therefore, we present Assembled-OpenML. Assembled-OpenML is a Python tool, which builds meta-datasets for ensembles using OpenML. A meta-dataset, called Metatask, consists of the data of an OpenML task, the task's dataset, and prediction data from model evaluations for the task. We can make the comparison of ensemble techniques computationally cheaper by using the predictions stored in a metatask instead of training and evaluating base models. To introduce Assembled-OpenML, we describe the first version of our tool. Moreover, we present an example of using Assembled-OpenML to compare a set of ensemble techniques. For this example comparison, we built a benchmark using Assembled-OpenML and implemented ensemble techniques expecting predictions instead of base models as input. In our example comparison, we gathered the prediction data of $1523$ base models for $31$ datasets. Obtaining the prediction data for all base models using Assembled-OpenML took ${\\sim} 1$ hour in total. In comparison, obtaining the prediction data by training and evaluating just one base model on the most computationally expensive dataset took ${\\sim} 37$ minutes.","url_abs":"https://arxiv.org/abs/2307.00285v1","url_pdf":"https://arxiv.org/pdf/2307.00285v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assembled-openml-creating-efficient","repo_url":"https://github.com/isg-siegen/assembled","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"automl","task_name":"AutoML"},{"task_slug":"prediction","task_name":"Prediction"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.00285","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.00285"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/isg-siegen/assembled","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e913d0385d70c60e","entry":"check_fold_data_for_ensemble","repo":"isg-siegen/assembled","repo_kind":"official","path":"assembled/utils/preprocessing.py","file_url":"https://github.com/isg-siegen/assembled/blob/HEAD/assembled/utils/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e913d0385d70c60e"}},{"code_sha256_prefix":"154a460210aaad8f","entry":"get_logger","repo":"isg-siegen/assembled","repo_kind":"official","path":"assembled/utils/logger.py","file_url":"https://github.com/isg-siegen/assembled/blob/HEAD/assembled/utils/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"154a460210aaad8f"}},{"code_sha256_prefix":"ac0438370fa2f27e","entry":"isolate_function","repo":"isg-siegen/assembled","repo_kind":"official","path":"assembled/utils/isolation.py","file_url":"https://github.com/isg-siegen/assembled/blob/HEAD/assembled/utils/isolation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac0438370fa2f27e"}},{"code_sha256_prefix":"db5bcd35c8bb8ff0","entry":"load_benchmark_details","repo":"isg-siegen/assembled","repo_kind":"official","path":"assembled/benchmaker.py","file_url":"https://github.com/isg-siegen/assembled/blob/HEAD/assembled/benchmaker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db5bcd35c8bb8ff0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}