{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-programming-creating-large-training-sets","title":"Data Programming: Creating Large Training Sets, Quickly","arxiv_id":"1605.07723","date":"2016-05-25","proceeding":"NeurIPS 2016 12","authors":["Alexander Ratner","Christopher De Sa","Sen Wu","Daniel Selsam","Christopher Ré"],"abstract":"Large labeled training sets are the critical building blocks of supervised\nlearning methods and are key enablers of deep learning techniques. For some\napplications, creating labeled training sets is the most time-consuming and\nexpensive part of applying machine learning. We therefore propose a paradigm\nfor the programmatic creation of training sets called data programming in which\nusers express weak supervision strategies or domain heuristics as labeling\nfunctions, which are programs that label subsets of the data, but that are\nnoisy and may conflict. We show that by explicitly representing this training\nset labeling process as a generative model, we can \"denoise\" the generated\ntraining set, and establish theoretically that we can recover the parameters of\nthese generative models in a handful of settings. We then show how to modify a\ndiscriminative loss function to make it noise-aware, and demonstrate our method\nover a range of discriminative models including logistic regression and LSTMs.\nExperimentally, on the 2014 TAC-KBP Slot Filling challenge, we show that data\nprogramming would have led to a new winning score, and also show that applying\ndata programming to an LSTM model leads to a TAC-KBP score almost 6 F1 points\nover a state-of-the-art LSTM baseline (and into second place in the\ncompetition). Additionally, in initial user studies we observed that data\nprogramming may be an easier way for non-experts to create machine learning\nmodels when training data is limited or unavailable.","url_abs":"http://arxiv.org/abs/1605.07723v3","url_pdf":"http://arxiv.org/pdf/1605.07723v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-programming-creating-large-training-sets","repo_url":"https://github.com/HazyResearch/metal","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"data-programming-creating-large-training-sets","repo_url":"https://github.com/autonlab/KeyClass","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"data-programming-creating-large-training-sets","repo_url":"https://github.com/megagonlabs/ruler","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"data-programming-creating-large-training-sets","repo_url":"https://github.com/HazyResearch/snorkel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"slot-filling","task_name":"Slot Filling"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"logistic-regression","method_name":"Logistic Regression"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1605.07723","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1605.07723"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HazyResearch/snorkel","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megagonlabs/ruler","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/autonlab/KeyClass","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HazyResearch/metal","reach":null}],"summary":{"unverified":8},"by_repo_kind":{"listed":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"66efc3fece34a64c","entry":"collate_dicts","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/classification/data.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/classification/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66efc3fece34a64c"}},{"code_sha256_prefix":"512345b35b221e44","entry":"cross_entropy_with_probs","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/classification/loss.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/classification/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"512345b35b221e44"}},{"code_sha256_prefix":"07a59a1416da68da","entry":"get_hashable","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/map/core.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/map/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07a59a1416da68da"}},{"code_sha256_prefix":"9daba8a219f31d0d","entry":"get_parameters","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/map/core.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/map/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9daba8a219f31d0d"}},{"code_sha256_prefix":"0dec48119b40e4e3","entry":"is_hashable","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/map/core.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/map/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0dec48119b40e4e3"}},{"code_sha256_prefix":"43b0055f130323d8","entry":"preds_to_probs","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/utils/core.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/utils/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43b0055f130323d8"}},{"code_sha256_prefix":"0a0cd678df3dfdc0","entry":"probs_to_preds","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/utils/core.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/utils/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a0cd678df3dfdc0"}},{"code_sha256_prefix":"6d6a7b0071d97035","entry":"to_int_label_array","repo":"HazyResearch/snorkel","repo_kind":"listed","path":"snorkel/utils/core.py","file_url":"https://github.com/HazyResearch/snorkel/blob/HEAD/snorkel/utils/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d6a7b0071d97035"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}