{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/utilizing-supervised-models-to-infer","title":"CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators","arxiv_id":"2210.06812","date":"2022-10-13","proceeding":null,"authors":["Hui Wen Goh","Ulyana Tkachenko","Jonas Mueller"],"abstract":"Real-world data for classification is often labeled by multiple annotators. For analyzing such data, we introduce CROWDLAB, a straightforward approach to utilize any trained classifier to estimate: (1) A consensus label for each example that aggregates the available annotations; (2) A confidence score for how likely each consensus label is correct; (3) A rating for each annotator quantifying the overall correctness of their labels. Existing algorithms to estimate related quantities in crowdsourcing often rely on sophisticated generative models with iterative inference. CROWDLAB instead uses a straightforward weighted ensemble. Existing algorithms often rely solely on annotator statistics, ignoring the features of the examples from which the annotations derive. CROWDLAB utilizes any classifier model trained on these features, and can thus better generalize between examples with similar features. On real-world multi-annotator image data, our proposed method provides superior estimates for (1)-(3) than existing algorithms like Dawid-Skene/GLAD.","url_abs":"https://arxiv.org/abs/2210.06812v2","url_pdf":"https://arxiv.org/pdf/2210.06812v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"utilizing-supervised-models-to-infer","repo_url":"https://github.com/cleanlab/cleanlab","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"utilizing-supervised-models-to-infer","repo_url":"https://github.com/cleanlab/multiannotator-benchmarks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.06812","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.06812"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cleanlab/cleanlab","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cleanlab/multiannotator-benchmarks","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"99b009a70defaf35","entry":"add_new_annotator","repo":"cleanlab/multiannotator-benchmarks","repo_kind":"official","path":"active_learning_benchmarks/extra-benchmarks/single-vs-multiannotator/utils/active_learning.py","file_url":"https://github.com/cleanlab/multiannotator-benchmarks/blob/HEAD/active_learning_benchmarks/extra-benchmarks/single-vs-multiannotator/utils/active_learning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99b009a70defaf35"}},{"code_sha256_prefix":"7241e5406cf81c73","entry":"convert_long_to_wide_dataset","repo":"cleanlab/multiannotator-benchmarks","repo_kind":"official","path":"cleanlab/cleanlab/multiannotator.py","file_url":"https://github.com/cleanlab/multiannotator-benchmarks/blob/HEAD/cleanlab/cleanlab/multiannotator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7241e5406cf81c73"}},{"code_sha256_prefix":"39feeaa68d76c9f6","entry":"get_consensus_label","repo":"cleanlab/multiannotator-benchmarks","repo_kind":"official","path":"cleanlab/cleanlab/multiannotator.py","file_url":"https://github.com/cleanlab/multiannotator-benchmarks/blob/HEAD/cleanlab/cleanlab/multiannotator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39feeaa68d76c9f6"}},{"code_sha256_prefix":"dff282e739a467b7","entry":"setup_next_iter_data","repo":"cleanlab/multiannotator-benchmarks","repo_kind":"official","path":"active_learning_benchmarks/extra-benchmarks/single-vs-multiannotator/utils/active_learning.py","file_url":"https://github.com/cleanlab/multiannotator-benchmarks/blob/HEAD/active_learning_benchmarks/extra-benchmarks/single-vs-multiannotator/utils/active_learning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dff282e739a467b7"}},{"code_sha256_prefix":"3409914a7dc265e1","entry":"setup_next_iter_data_single","repo":"cleanlab/multiannotator-benchmarks","repo_kind":"official","path":"active_learning_benchmarks/extra-benchmarks/single-vs-multiannotator/utils/active_learning.py","file_url":"https://github.com/cleanlab/multiannotator-benchmarks/blob/HEAD/active_learning_benchmarks/extra-benchmarks/single-vs-multiannotator/utils/active_learning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3409914a7dc265e1"}},{"code_sha256_prefix":"40573427c196c6d0","entry":"sum_xval_folds","repo":"cleanlab/multiannotator-benchmarks","repo_kind":"official","path":"active_learning_benchmarks/cifar-10h/utils/model_training.py","file_url":"https://github.com/cleanlab/multiannotator-benchmarks/blob/HEAD/active_learning_benchmarks/cifar-10h/utils/model_training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40573427c196c6d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}