{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-from-noisy-singly-labeled-data","title":"Learning From Noisy Singly-labeled Data","arxiv_id":"1712.04577","date":"2017-12-13","proceeding":"ICLR 2018 1","authors":["Ashish Khetan","Zachary C. Lipton","Anima Anandkumar"],"abstract":"Supervised learning depends on annotated examples, which are taken to be the\n\\emph{ground truth}. But these labels often come from noisy crowdsourcing\nplatforms, like Amazon Mechanical Turk. Practitioners typically collect\nmultiple labels per example and aggregate the results to mitigate noise (the\nclassic crowdsourcing problem). Given a fixed annotation budget and unlimited\nunlabeled data, redundant annotation comes at the expense of fewer labeled\nexamples. This raises two fundamental questions: (1) How can we best learn from\nnoisy workers? (2) How should we allocate our labeling budget to maximize the\nperformance of a classifier? We propose a new algorithm for jointly modeling\nlabels and worker quality from noisy crowd-sourced data. The alternating\nminimization proceeds in rounds, estimating worker quality from disagreement\nwith the current model and then updating the model by optimizing a loss\nfunction that accounts for the current estimate of worker quality. Unlike\nprevious approaches, even with only one annotation per example, our algorithm\ncan estimate worker quality. We establish a generalization error bound for\nmodels learned with our algorithm and establish theoretically that it's better\nto label many examples once (vs less multiply) when worker quality is above a\nthreshold. Experiments conducted on both ImageNet (with simulated noisy\nworkers) and MS-COCO (using the real crowdsourced labels) confirm our\nalgorithm's benefits.","url_abs":"http://arxiv.org/abs/1712.04577v2","url_pdf":"http://arxiv.org/pdf/1712.04577v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-from-noisy-singly-labeled-data","repo_url":"https://github.com/khetan2/MBEM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"mxnet","reach":{"status":"unanswered"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1712.04577","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}