{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stronger-than-you-think-benchmarking-weak","title":"Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks","arxiv_id":"2501.07727","date":"2025-01-13","proceeding":null,"authors":["Tianyi Zhang","Linrong Cai","Jeffrey Li","Nicholas Roberts","Neel Guha","Jinoh Lee","Frederic Sala"],"abstract":"Weak supervision (WS) is a popular approach for label-efficient learning, leveraging diverse sources of noisy but inexpensive weak labels to automatically annotate training data. Despite its wide usage, WS and its practical value are challenging to benchmark due to the many knobs in its setup, including: data sources, labeling functions (LFs), aggregation techniques (called label models), and end model pipelines. Existing evaluation suites tend to be limited, focusing on particular components or specialized use cases. Moreover, they often involve simplistic benchmark tasks or de-facto LF sets that are suboptimally written, producing insights that may not generalize to real-world settings. We address these limitations by introducing a new benchmark, BOXWRENCH, designed to more accurately reflect real-world usages of WS. This benchmark features tasks with (1) higher class cardinality and imbalance, (2) notable domain expertise requirements, and (3) opportunities to re-use LFs across parallel multilingual corpora. For all tasks, LFs are written using a careful procedure aimed at mimicking real-world settings. In contrast to existing WS benchmarks, we show that supervised learning requires substantial amounts (1000+) of labeled examples to match WS in many settings.","url_abs":"https://arxiv.org/abs/2501.07727v2","url_pdf":"https://arxiv.org/pdf/2501.07727v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stronger-than-you-think-benchmarking-weak","repo_url":"https://github.com/jeffreywpli/stronger-than-you-think","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.07727","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.07727"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jeffreywpli/stronger-than-you-think","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"851d43d773a34407","entry":"custom_stratified_sample","repo":"jeffreywpli/stronger-than-you-think","repo_kind":"official","path":"end_model_training/pipelines.py","file_url":"https://github.com/jeffreywpli/stronger-than-you-think/blob/HEAD/end_model_training/pipelines.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"851d43d773a34407"}},{"code_sha256_prefix":"fbd78623468b25e3","entry":"find_entity_indices","repo":"jeffreywpli/stronger-than-you-think","repo_kind":"official","path":"end_model_training/lable_function/chemprot/label_improve.py","file_url":"https://github.com/jeffreywpli/stronger-than-you-think/blob/HEAD/end_model_training/lable_function/chemprot/label_improve.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbd78623468b25e3"}},{"code_sha256_prefix":"61a13160fcc3a123","entry":"wrench_to_df","repo":"jeffreywpli/stronger-than-you-think","repo_kind":"official","path":"end_model_training/lable_function/chemprot/label_improve.py","file_url":"https://github.com/jeffreywpli/stronger-than-you-think/blob/HEAD/end_model_training/lable_function/chemprot/label_improve.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61a13160fcc3a123"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}