{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-benchmark-determining-best","title":"Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data","arxiv_id":"1909.07192","date":"2019-09-16","proceeding":null,"authors":["Morteza Noshad","Li Xu","Alfred Hero"],"abstract":"We address the problem of learning to benchmark the best achievable classifier performance. In this problem the objective is to establish statistically consistent estimates of the Bayes misclassification error rate without having to learn a Bayes-optimal classifier. Our learning to benchmark framework improves on previous work on learning bounds on Bayes misclassification rate since it learns the {\\it exact} Bayes error rate instead of a bound on error rate. We propose a benchmark learner based on an ensemble of $\\epsilon$-ball estimators and Chebyshev approximation. Under a smoothness assumption on the class densities we show that our estimator achieves an optimal (parametric) mean squared error (MSE) rate of $O(N^{-1})$, where $N$ is the number of samples. Experiments on both simulated and real datasets establish that our proposed benchmark learning algorithm produces estimates of the Bayes error that are more accurate than previous approaches for learning bounds on Bayes error probability.","url_abs":"https://arxiv.org/abs/1909.07192v1","url_pdf":"https://arxiv.org/pdf/1909.07192v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-benchmark-determining-best","repo_url":"https://github.com/Helmuthn/LearnBenchmark.jl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-benchmark-determining-best","repo_url":"https://github.com/mrtnoshad/Bayes_Error_Estimator","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1909.07192","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.07192"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mrtnoshad/Bayes_Error_Estimator","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Helmuthn/LearnBenchmark.jl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":1},"by_repo_kind":{"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"8ed381f48c700bc4","entry":"eps_neighbor_count","repo":"mrtnoshad/Bayes_Error_Estimator","repo_kind":"listed","path":"BER_estimator_KDtree.py","file_url":"https://github.com/mrtnoshad/Bayes_Error_Estimator/blob/HEAD/BER_estimator_KDtree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ed381f48c700bc4"}},{"code_sha256_prefix":"911df99b0dcef07f","entry":"bg_estimator","repo":"mrtnoshad/Bayes_Error_Estimator","repo_kind":"listed","path":"BER_estimator_KDtree.py","file_url":"https://github.com/mrtnoshad/Bayes_Error_Estimator/blob/HEAD/BER_estimator_KDtree.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"911df99b0dcef07f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}