{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/truth-discovery-in-sequence-labels-from","title":"Truth Discovery in Sequence Labels from Crowds","arxiv_id":"2109.04470","date":"2021-09-09","proceeding":null,"authors":["Nasim Sabetpour","Adithya Kulkarni","Sihong Xie","Qi Li"],"abstract":"Annotation quality and quantity positively affect the learning performance of sequence labeling, a vital task in Natural Language Processing. Hiring domain experts to annotate a corpus is very costly in terms of money and time. Crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), have been deployed to assist in this purpose. However, the annotations collected this way are prone to human errors due to the lack of expertise of the crowd workers. Existing literature in annotation aggregation assumes that annotations are independent and thus faces challenges when handling the sequential label aggregation tasks with complex dependencies. To conquer the challenges, we propose an optimization-based method that infers the ground truth labels using annotations provided by workers for sequential labeling tasks. The proposed Aggregation method for Sequential Labels from Crowds ($AggSLC$) jointly considers the characteristics of sequential labeling tasks, workers' reliabilities, and advanced machine learning techniques. Theoretical analysis on the algorithm's convergence further demonstrates that the proposed $AggSLC$ halts after a finite number of iterations. We evaluate $AggSLC$ on different crowdsourced datasets for Named Entity Recognition (NER) tasks and Information Extraction tasks in biomedical (PICO), as well as a simulated dataset. Our results show that the proposed method outperforms the state-of-the-art aggregation methods. To achieve insights into the framework, we study the effectiveness of $AggSLC$'s components through ablation studies.","url_abs":"https://arxiv.org/abs/2109.04470v2","url_pdf":"https://arxiv.org/pdf/2109.04470v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"truth-discovery-in-sequence-labels-from","repo_url":"https://github.com/nasimisu/truth-discovery-in-sequence-labels-from-crowds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"cg","task_name":"NER"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"pico","task_name":"PICO"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.04470","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.04470"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nasimisu/truth-discovery-in-sequence-labels-from-crowds","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"3f97c773ca83c0ef","entry":"array_to_sentence","repo":"nasimisu/truth-discovery-in-sequence-labels-from-crowds","repo_kind":"official","path":"execution.py","file_url":"https://github.com/nasimisu/truth-discovery-in-sequence-labels-from-crowds/blob/HEAD/execution.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f97c773ca83c0ef"}},{"code_sha256_prefix":"de7d6b24d6d234ff","entry":"read_conll","repo":"nasimisu/truth-discovery-in-sequence-labels-from-crowds","repo_kind":"official","path":"data_preprocessing.py","file_url":"https://github.com/nasimisu/truth-discovery-in-sequence-labels-from-crowds/blob/HEAD/data_preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de7d6b24d6d234ff"}},{"code_sha256_prefix":"35705fd6f9774602","entry":"score","repo":"nasimisu/truth-discovery-in-sequence-labels-from-crowds","repo_kind":"official","path":"calculations.py","file_url":"https://github.com/nasimisu/truth-discovery-in-sequence-labels-from-crowds/blob/HEAD/calculations.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35705fd6f9774602"}},{"code_sha256_prefix":"4c08bc92fd2e7ab0","entry":"confidence_measurement_word","repo":"nasimisu/truth-discovery-in-sequence-labels-from-crowds","repo_kind":"official","path":"execution.py","file_url":"https://github.com/nasimisu/truth-discovery-in-sequence-labels-from-crowds/blob/HEAD/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c08bc92fd2e7ab0"}},{"code_sha256_prefix":"e3fc3dee4d629211","entry":"confidence_measurement_word_cnn","repo":"nasimisu/truth-discovery-in-sequence-labels-from-crowds","repo_kind":"official","path":"execution.py","file_url":"https://github.com/nasimisu/truth-discovery-in-sequence-labels-from-crowds/blob/HEAD/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3fc3dee4d629211"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}