{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-from-uneven-training-data-unlabeled","title":"Learning with Different Amounts of Annotation: From Zero to Many Labels","arxiv_id":"2109.04408","date":"2021-09-09","proceeding":"EMNLP 2021 11","authors":["Shujian Zhang","Chengyue Gong","Eunsol Choi"],"abstract":"Training NLP systems typically assumes access to annotated data that has a single human label per example. Given imperfect labeling from annotators and inherent ambiguity of language, we hypothesize that single label is not sufficient to learn the spectrum of language interpretation. We explore new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples. Introducing such multi label examples at the cost of annotating fewer examples brings clear gains on natural language inference task and entity typing task, even when we simply first train with a single label data and then fine tune with multi label examples. Extending a MixUp data augmentation framework, we propose a learning algorithm that can learn from training examples with different amount of annotation (with zero, one, or multiple labels). This algorithm efficiently combines signals from uneven training data and brings additional gains in low annotation budget and cross domain settings. Together, our method achieves consistent gains in two tasks, suggesting distributing labels unevenly among training examples can be beneficial for many NLP tasks.","url_abs":"https://arxiv.org/abs/2109.04408v2","url_pdf":"https://arxiv.org/pdf/2109.04408v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-from-uneven-training-data-unlabeled","repo_url":"https://github.com/szhang42/uneven_training_data","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"entity-typing","task_name":"Entity Typing"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"}],"methods":[{"method_slug":"mixup","method_name":"Mixup"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.04408","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.04408"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/szhang42/uneven_training_data","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"323760bd46af190a","entry":"Convert","repo":"szhang42/uneven_training_data","repo_kind":"official","path":"word_level_augment.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/word_level_augment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"323760bd46af190a"}},{"code_sha256_prefix":"06296d00f2634d11","entry":"filter_unicode","repo":"szhang42/uneven_training_data","repo_kind":"official","path":"word_level_augment.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/word_level_augment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06296d00f2634d11"}},{"code_sha256_prefix":"ca9b245b51991089","entry":"EfficientRandomGen","repo":"szhang42/uneven_training_data","repo_kind":"official","path":"word_level_augment.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/word_level_augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca9b245b51991089"}},{"code_sha256_prefix":"73205b5c7ba69704","entry":"TfIdfWordRep","repo":"szhang42/uneven_training_data","repo_kind":"official","path":"word_level_augment.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/word_level_augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73205b5c7ba69704"}},{"code_sha256_prefix":"a236e58761652e0e","entry":"UnifRep","repo":"szhang42/uneven_training_data","repo_kind":"official","path":"word_level_augment.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/word_level_augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a236e58761652e0e"}},{"code_sha256_prefix":"4e7cda3fdccf7fcd","entry":"word_level_augment","repo":"szhang42/uneven_training_data","repo_kind":"official","path":"word_level_augment.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/word_level_augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e7cda3fdccf7fcd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}