{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/better-than-classical-the-subtle-art-of","title":"Better than classical? The subtle art of benchmarking quantum machine learning models","arxiv_id":"2403.07059","date":"2024-03-11","proceeding":null,"authors":["Joseph Bowles","Shahnawaz Ahmed","Maria Schuld"],"abstract":"Benchmarking models via classical simulations is one of the main ways to judge ideas in quantum machine learning before noise-free hardware is available. However, the huge impact of the experimental design on the results, the small scales within reach today, as well as narratives influenced by the commercialisation of quantum technologies make it difficult to gain robust insights. To facilitate better decision-making we develop an open-source package based on the PennyLane software framework and use it to conduct a large-scale study that systematically tests 12 popular quantum machine learning models on 6 binary classification tasks used to create 160 individual datasets. We find that overall, out-of-the-box classical machine learning models outperform the quantum classifiers. Moreover, removing entanglement from a quantum model often results in as good or better performance, suggesting that \"quantumness\" may not be the crucial ingredient for the small learning tasks considered here. Our benchmarks also unlock investigations beyond simplistic leaderboard comparisons, and we identify five important questions for quantum model design that follow from our results.","url_abs":"https://arxiv.org/abs/2403.07059v2","url_pdf":"https://arxiv.org/pdf/2403.07059v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"better-than-classical-the-subtle-art-of","repo_url":"https://github.com/xanaduai/qml-benchmarks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"better-than-classical-the-subtle-art-of","repo_url":"https://github.com/PennyLaneAI/pennylane","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"better-than-classical-the-subtle-art-of","repo_url":"https://github.com/XanaduAI/pennylane","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"experimental-design","task_name":"Experimental Design"},{"task_slug":"quantum-machine-learning","task_name":"Quantum Machine Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.07059","atlas_url":"https://app.syntology.ai/?focus=2403.07059","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.07059"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xanaduai/qml-benchmarks","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PennyLaneAI/pennylane","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/XanaduAI/pennylane","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16bddb74f528382f","entry":"classproperty","repo":"PennyLaneAI/pennylane","repo_kind":"listed","path":"pennylane/_class_property.py","file_url":"https://github.com/PennyLaneAI/pennylane/blob/HEAD/pennylane/_class_property.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16bddb74f528382f"}},{"code_sha256_prefix":"5547eca66c2fd36b","entry":"custom_scorer","repo":"xanaduai/qml-benchmarks","repo_kind":"official","path":"scripts/run_hyperparameter_search.py","file_url":"https://github.com/xanaduai/qml-benchmarks/blob/HEAD/scripts/run_hyperparameter_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5547eca66c2fd36b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}