{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evidence-based-verification-for-real-world","title":"AmbiFC: Fact-Checking Ambiguous Claims with Evidence","arxiv_id":"2104.00640","date":"2021-04-01","proceeding":null,"authors":["Max Glockner","Ieva Staliūnaitė","James Thorne","Gisela Vallejo","Andreas Vlachos","Iryna Gurevych"],"abstract":"Automated fact-checking systems verify claims against evidence to predict their veracity. In real-world scenarios, the retrieved evidence may not unambiguously support or refute the claim and yield conflicting but valid interpretations. Existing fact-checking datasets assume that the models developed with them predict a single veracity label for each claim, thus discouraging the handling of such ambiguity. To address this issue we present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. It contains fine-grained evidence annotations of 50k passages from 5k Wikipedia pages. We analyze the disagreements arising from ambiguity when comparing claims against evidence in AmbiFC, observing a strong correlation of annotator disagreement with linguistic phenomena such as underspecification and probabilistic reasoning. We develop models for predicting veracity handling this ambiguity via soft labels and find that a pipeline that learns the label distribution for sentence-level evidence selection and veracity prediction yields the best performance. We compare models trained on different subsets of AmbiFC and show that models trained on the ambiguous instances perform better when faced with the identified linguistic phenomena.","url_abs":"https://arxiv.org/abs/2104.00640v4","url_pdf":"https://arxiv.org/pdf/2104.00640v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evidence-based-verification-for-real-world","repo_url":"https://github.com/CambridgeNLIP/verification-real-world-info-needs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"evidence-based-verification-for-real-world","repo_url":"https://github.com/knot-fit-but/claimdissector","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"claim-verification","task_name":"Claim Verification"},{"task_slug":"evidence-selection","task_name":"Evidence Selection"},{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"stance-classification","task_name":"Stance Classification"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.00640","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.00640"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/knot-fit-but/claimdissector","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CambridgeNLIP/verification-real-world-info-needs","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"listed":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"efa3643614e9c5f5","entry":"exact_match_score","repo":"knot-fit-but/claimdissector","repo_kind":"listed","path":"src/common/eval_utils.py","file_url":"https://github.com/knot-fit-but/claimdissector/blob/HEAD/src/common/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efa3643614e9c5f5"}},{"code_sha256_prefix":"90cb252a65c0643e","entry":"arg_topk","repo":"knot-fit-but/claimdissector","repo_kind":"listed","path":"src/common/utility.py","file_url":"https://github.com/knot-fit-but/claimdissector/blob/HEAD/src/common/utility.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90cb252a65c0643e"}},{"code_sha256_prefix":"6b3ccf76356a23b7","entry":"f1_score","repo":"knot-fit-but/claimdissector","repo_kind":"listed","path":"src/common/eval_utils.py","file_url":"https://github.com/knot-fit-but/claimdissector/blob/HEAD/src/common/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b3ccf76356a23b7"}},{"code_sha256_prefix":"c26a0a5333ae5ba2","entry":"find_sub_list","repo":"knot-fit-but/claimdissector","repo_kind":"listed","path":"src/common/utility.py","file_url":"https://github.com/knot-fit-but/claimdissector/blob/HEAD/src/common/utility.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c26a0a5333ae5ba2"}},{"code_sha256_prefix":"8ddaf8aae0f76d1e","entry":"normalize_answer","repo":"knot-fit-but/claimdissector","repo_kind":"listed","path":"src/common/eval_utils.py","file_url":"https://github.com/knot-fit-but/claimdissector/blob/HEAD/src/common/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ddaf8aae0f76d1e"}},{"code_sha256_prefix":"dad0b1b31bc66fa9","entry":"share_list","repo":"knot-fit-but/claimdissector","repo_kind":"listed","path":"src/common/distributed_utils.py","file_url":"https://github.com/knot-fit-but/claimdissector/blob/HEAD/src/common/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dad0b1b31bc66fa9"}},{"code_sha256_prefix":"ccff200d6bb6d6fd","entry":"unicode_normalize","repo":"knot-fit-but/claimdissector","repo_kind":"listed","path":"src/common/utility.py","file_url":"https://github.com/knot-fit-but/claimdissector/blob/HEAD/src/common/utility.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ccff200d6bb6d6fd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}