{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/varierr-nli-separating-annotation-error-from","title":"VariErr NLI: Separating Annotation Error from Human Label Variation","arxiv_id":"2403.01931","date":"2024-03-04","proceeding":null,"authors":["Leon Weber-Genzel","Siyao Peng","Marie-Catherine de Marneffe","Barbara Plank"],"abstract":"Human label variation arises when annotators assign different labels to the same item for valid reasons, while annotation errors occur when labels are assigned for invalid reasons. These two issues are prevalent in NLP benchmarks, yet existing research has studied them in isolation. To the best of our knowledge, there exists no prior work that focuses on teasing apart error from signal, especially in cases where signal is beyond black-and-white. To fill this gap, we introduce a systematic methodology and a new dataset, VariErr (variation versus error), focusing on the NLI task in English. We propose a 2-round annotation procedure with annotators explaining each label and subsequently judging the validity of label-explanation pairs. VariErr contains 7,732 validity judgments on 1,933 explanations for 500 re-annotated MNLI items. We assess the effectiveness of various automatic error detection (AED) methods and GPTs in uncovering errors versus human label variation. We find that state-of-the-art AED methods significantly underperform GPTs and humans. While GPT-4 is the best system, it still falls short of human performance. Our methodology is applicable beyond NLI, offering fertile ground for future research on error versus plausible variation, which in turn can yield better and more trustworthy NLP systems.","url_abs":"https://arxiv.org/abs/2403.01931v2","url_pdf":"https://arxiv.org/pdf/2403.01931v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":null,"task_name":"valid"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.01931","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.01931"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mainlp/VariErr-NLI","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"81217cf08fe12d42","entry":"compute_metrics","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"build_score_table.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/build_score_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81217cf08fe12d42"}},{"code_sha256_prefix":"dca3f599335ed6d6","entry":"get_reason_to_score","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"predict_llms.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/predict_llms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dca3f599335ed6d6"}},{"code_sha256_prefix":"1726b2eaa67e19ad","entry":"instance_to_starting_prompt","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"eval_llms.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/eval_llms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1726b2eaa67e19ad"}},{"code_sha256_prefix":"04aa42119c9d143f","entry":"precision_at_k","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"build_score_table.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/build_score_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04aa42119c9d143f"}},{"code_sha256_prefix":"d72b6a140ff417e5","entry":"recall_at_k","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"build_score_table.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/build_score_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d72b6a140ff417e5"}},{"code_sha256_prefix":"184bba6c793855a8","entry":"aggregate_scores","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"predict_llms.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/predict_llms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"184bba6c793855a8"}},{"code_sha256_prefix":"f1ddd1d063b9b7a4","entry":"get_logits_per_epoch","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"predict_scorers.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/predict_scorers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1ddd1d063b9b7a4"}},{"code_sha256_prefix":"1e7a15f18668ad1d","entry":"get_probs_per_epoch","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"predict_scorers.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/predict_scorers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e7a15f18668ad1d"}},{"code_sha256_prefix":"f5c93c97adfd0c96","entry":"predict_datamap_means","repo":"mainlp/VariErr-NLI","repo_kind":"found_in_text","path":"predict_scorers.py","file_url":"https://github.com/mainlp/VariErr-NLI/blob/HEAD/predict_scorers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5c93c97adfd0c96"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}