{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-state-of-the-art-classification","title":"Evaluating State-of-the-Art Classification Models Against Bayes Optimality","arxiv_id":"2106.03357","date":"2021-06-07","proceeding":"NeurIPS 2021 12","authors":["Ryan Theisen","Huan Wang","Lav R. Varshney","Caiming Xiong","Richard Socher"],"abstract":"Evaluating the inherent difficulty of a given data-driven classification problem is important for establishing absolute benchmarks and evaluating progress in the field. To this end, a natural quantity to consider is the \\emph{Bayes error}, which measures the optimal classification error theoretically achievable for a given data distribution. While generally an intractable quantity, we show that we can compute the exact Bayes error of generative models learned using normalizing flows. Our technique relies on a fundamental result, which states that the Bayes error is invariant under invertible transformation. Therefore, we can compute the exact Bayes error of the learned flow models by computing it for Gaussian base distributions, which can be done efficiently using Holmes-Diaconis-Ross integration. Moreover, we show that by varying the temperature of the learned flow models, we can generate synthetic datasets that closely resemble standard benchmark datasets, but with almost any desired Bayes error. We use our approach to conduct a thorough investigation of state-of-the-art classification models, and find that in some -- but not all -- cases, these models are capable of obtaining accuracy very near optimal. Finally, we use our method to evaluate the intrinsic \"hardness\" of standard benchmark datasets, and classes within those datasets.","url_abs":"https://arxiv.org/abs/2106.03357v1","url_pdf":"https://arxiv.org/pdf/2106.03357v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-state-of-the-art-classification","repo_url":"https://github.com/salesforce/DataHardness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.03357","atlas_url":"https://app.syntology.ai/?focus=2106.03357","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.03357"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salesforce/DataHardness","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":1,"ran_fixture":2,"unverified":1},"by_repo_kind":{"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"54dc276d7472ea04","entry":"bits_per_dim","repo":"salesforce/DataHardness","repo_kind":"listed","path":"util/optim_util.py","file_url":"https://github.com/salesforce/DataHardness/blob/HEAD/util/optim_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"54dc276d7472ea04"}},{"code_sha256_prefix":"705e5c95625bd438","entry":"mean_dim","repo":"salesforce/DataHardness","repo_kind":"listed","path":"util/array_util.py","file_url":"https://github.com/salesforce/DataHardness/blob/HEAD/util/array_util.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"705e5c95625bd438"}},{"code_sha256_prefix":"f39b13439e3fc9cc","entry":"squeeze","repo":"salesforce/DataHardness","repo_kind":"listed","path":"models/glow/glow.py","file_url":"https://github.com/salesforce/DataHardness/blob/HEAD/models/glow/glow.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f39b13439e3fc9cc"}},{"code_sha256_prefix":"446a822fa47fa09c","entry":"sample","repo":"salesforce/DataHardness","repo_kind":"listed","path":"generate_dataset.py","file_url":"https://github.com/salesforce/DataHardness/blob/HEAD/generate_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"446a822fa47fa09c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}