{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimating-frechet-bounds-for-validating","title":"Weak Supervision Performance Evaluation via Partial Identification","arxiv_id":"2312.04601","date":"2023-12-07","proceeding":null,"authors":["Felipe Maia Polo","Subha Maity","Mikhail Yurochkin","Moulinath Banerjee","Yuekai Sun"],"abstract":"Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels, utilizing weak labels from heuristics, crowdsourcing, or pre-trained models. However, the absence of ground truth complicates model evaluation, as traditional metrics such as accuracy, precision, and recall cannot be directly calculated. In this work, we present a novel method to address this challenge by framing model evaluation as a partial identification problem and estimating performance bounds using Fr\\'echet bounds. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques. Through scalable convex optimization, we obtain accurate and computationally efficient bounds for metrics including accuracy, precision, recall, and F1-score, even in high-dimensional settings. This framework offers a robust approach to assessing model quality without ground truth labels, enhancing the practicality of weakly supervised learning for real-world applications.","url_abs":"https://arxiv.org/abs/2312.04601v2","url_pdf":"https://arxiv.org/pdf/2312.04601v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"estimating-frechet-bounds-for-validating","repo_url":"https://github.com/felipemaiapolo/wsbounds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"weakly-supervised-learning","task_name":"Weakly-supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.04601","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.04601"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/felipemaiapolo/wsbounds","reach":null}],"summary":{"ran_honours":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"104935a2b237ae7a","entry":"truncate","repo":"felipemaiapolo/wsbounds","repo_kind":"official","path":"wsbounds/eval_pws.py","file_url":"https://github.com/felipemaiapolo/wsbounds/blob/HEAD/wsbounds/eval_pws.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"104935a2b237ae7a"}},{"code_sha256_prefix":"e880c10fbaf6ecb0","entry":"ztn","repo":"felipemaiapolo/wsbounds","repo_kind":"official","path":"wsbounds/eval_pws.py","file_url":"https://github.com/felipemaiapolo/wsbounds/blob/HEAD/wsbounds/eval_pws.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e880c10fbaf6ecb0"}},{"code_sha256_prefix":"31923c9e4e9ea55c","entry":"BoundExpectation","repo":"felipemaiapolo/wsbounds","repo_kind":"official","path":"wsbounds/bound_expectation.py","file_url":"https://github.com/felipemaiapolo/wsbounds/blob/HEAD/wsbounds/bound_expectation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31923c9e4e9ea55c"}},{"code_sha256_prefix":"6587ce3305d85d81","entry":"GetAccuracyTensor","repo":"felipemaiapolo/wsbounds","repo_kind":"official","path":"wsbounds/eval_pws.py","file_url":"https://github.com/felipemaiapolo/wsbounds/blob/HEAD/wsbounds/eval_pws.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6587ce3305d85d81"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}