{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/good-data-from-bad-models-foundations-of","title":"Promises and Pitfalls of Threshold-based Auto-labeling","arxiv_id":"2211.12620","date":"2022-11-22","proceeding":"NeurIPS 2023 11","authors":["Harit Vishwakarma","Heguang Lin","Frederic Sala","Ramya Korlakai Vinayak"],"abstract":"Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Threshold-based auto-labeling (TBAL), where validation data obtained from humans is used to find a confidence threshold above which the data is machine-labeled, reduces reliance on manual annotation. TBAL is emerging as a widely-used solution in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. This is the first work to analyze TBAL systems and derive sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of machine-labeled data. Our results provide two crucial insights. First, reasonable chunks of unlabeled data can be automatically and accurately labeled by seemingly bad models. Second, a hidden downside of TBAL systems is potentially prohibitive validation data usage. Together, these insights describe the promise and pitfalls of using such systems. We validate our theoretical guarantees with extensive experiments on synthetic and real datasets.","url_abs":"https://arxiv.org/abs/2211.12620v2","url_pdf":"https://arxiv.org/pdf/2211.12620v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"good-data-from-bad-models-foundations-of","repo_url":"https://github.com/harit7/tbal-neurips-23","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.12620","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.12620"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/harit7/tbal-neurips-23","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"999cea17d0121cdc","entry":"determine_threshold","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/threshold_estimation.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/threshold_estimation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"999cea17d0121cdc"}},{"code_sha256_prefix":"714479c11c15e671","entry":"get_counts","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/eval_utils.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"714479c11c15e671"}},{"code_sha256_prefix":"8f3979a6b9c23717","entry":"get_test_error","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/eval_utils.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f3979a6b9c23717"}},{"code_sha256_prefix":"3e9ddafcf8019ef9","entry":"get_test_error","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/model_utils.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9ddafcf8019ef9"}},{"code_sha256_prefix":"06b3caad923b59ac","entry":"run_active_learnling_auto_labeling","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/run_lib.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/run_lib.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06b3caad923b59ac"}},{"code_sha256_prefix":"11ad02da29544252","entry":"run_inference","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/model_utils.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11ad02da29544252"}},{"code_sha256_prefix":"e64b9a09effe7244","entry":"run_passive_labeling_auto_labeling","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/run_lib.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/run_lib.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e64b9a09effe7244"}},{"code_sha256_prefix":"c2d5eb14dae211a1","entry":"run_tbal_conf","repo":"harit7/tbal-neurips-23","repo_kind":"official","path":"core/run_lib.py","file_url":"https://github.com/harit7/tbal-neurips-23/blob/HEAD/core/run_lib.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2d5eb14dae211a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}