{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pmlb-a-large-benchmark-suite-for-machine","title":"PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison","arxiv_id":"1703.00512","date":"2017-03-01","proceeding":null,"authors":["Randal S. Olson","William La Cava","Patryk Orzechowski","Ryan J. Urbanowicz","Jason H. Moore"],"abstract":"The selection, development, or comparison of machine learning methods in data\nmining can be a difficult task based on the target problem and goals of a\nparticular study. Numerous publicly available real-world and simulated\nbenchmark datasets have emerged from different sources, but their organization\nand adoption as standards have been inconsistent. As such, selecting and\ncurating specific benchmarks remains an unnecessary burden on machine learning\npractitioners and data scientists. The present study introduces an accessible,\ncurated, and developing public benchmark resource to facilitate identification\nof the strengths and weaknesses of different machine learning methodologies. We\ncompare meta-features among the current set of benchmark datasets in this\nresource to characterize the diversity of available data. Finally, we apply a\nnumber of established machine learning methods to the entire benchmark suite\nand analyze how datasets and algorithms cluster in terms of performance. This\nwork is an important first step towards understanding the limitations of\npopular benchmarking suites and developing a resource that connects existing\nbenchmarking standards to more diverse and efficient standards in the future.","url_abs":"http://arxiv.org/abs/1703.00512v1","url_pdf":"http://arxiv.org/pdf/1703.00512v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pmlb-a-large-benchmark-suite-for-machine","repo_url":"https://github.com/EpistasisLab/penn-ml-benchmarks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"diversity","task_name":"Diversity"}],"methods":[],"datasets_introduced":[{"slug":"pmlb","name":"PMLB","full_name":"Penn Machine Learning Benchmarks"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1703.00512","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1703.00512"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EpistasisLab/penn-ml-benchmarks","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ce43d6c0cb3bda94","entry":"check_if_all_integers","repo":"EpistasisLab/penn-ml-benchmarks","repo_kind":"official","path":"pmlb/support_funcs.py","file_url":"https://github.com/EpistasisLab/penn-ml-benchmarks/blob/HEAD/pmlb/support_funcs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce43d6c0cb3bda94"}},{"code_sha256_prefix":"2e35baa597c65d68","entry":"fetch_data","repo":"EpistasisLab/penn-ml-benchmarks","repo_kind":"official","path":"pmlb/pmlb.py","file_url":"https://github.com/EpistasisLab/penn-ml-benchmarks/blob/HEAD/pmlb/pmlb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e35baa597c65d68"}},{"code_sha256_prefix":"eee416a8d29dd1dd","entry":"get_dataset_url","repo":"EpistasisLab/penn-ml-benchmarks","repo_kind":"official","path":"pmlb/pmlb.py","file_url":"https://github.com/EpistasisLab/penn-ml-benchmarks/blob/HEAD/pmlb/pmlb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eee416a8d29dd1dd"}},{"code_sha256_prefix":"bab50b183b7a6eda","entry":"get_feature_type","repo":"EpistasisLab/penn-ml-benchmarks","repo_kind":"official","path":"pmlb/support_funcs.py","file_url":"https://github.com/EpistasisLab/penn-ml-benchmarks/blob/HEAD/pmlb/support_funcs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bab50b183b7a6eda"}},{"code_sha256_prefix":"9380f9916347c52f","entry":"get_target_type","repo":"EpistasisLab/penn-ml-benchmarks","repo_kind":"official","path":"pmlb/support_funcs.py","file_url":"https://github.com/EpistasisLab/penn-ml-benchmarks/blob/HEAD/pmlb/support_funcs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9380f9916347c52f"}},{"code_sha256_prefix":"6af5b956bb5eb5e8","entry":"get_updated_datasets","repo":"EpistasisLab/penn-ml-benchmarks","repo_kind":"official","path":"pmlb/pmlb.py","file_url":"https://github.com/EpistasisLab/penn-ml-benchmarks/blob/HEAD/pmlb/pmlb.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6af5b956bb5eb5e8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}