{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-richly-annotated-corpus-for-different-tasks-1","title":"A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking","arxiv_id":"1911.01214","date":"2019-10-29","proceeding":"CONLL 2019 11","authors":["Andreas Hanselowski","Christian Stab","Claudia Schulz","Zile Li","Iryna Gurevych"],"abstract":"Automated fact-checking based on machine learning is a promising approach to identify false information distributed on the web. In order to achieve satisfactory performance, machine learning methods require a large corpus with reliable annotations for the different tasks in the fact-checking process. Having analyzed existing fact-checking corpora, we found that none of them meets these criteria in full. They are either too small in size, do not provide detailed annotations, or are limited to a single domain. Motivated by this gap, we present a new substantially sized mixed-domain corpus with annotations of good quality for the core fact-checking tasks: document retrieval, evidence extraction, stance detection, and claim validation. To aid future corpus construction, we describe our methodology for corpus creation and annotation, and demonstrate that it results in substantial inter-annotator agreement. As baselines for future research, we perform experiments on our corpus with a number of model architectures that reach high performance in similar problem settings. Finally, to support the development of future models, we provide a detailed error analysis for each of the tasks. Our results show that the realistic, multi-domain setting defined by our data poses new challenges for the existing models, providing opportunities for considerable improvement by future systems.","url_abs":"https://arxiv.org/abs/1911.01214v1","url_pdf":"https://arxiv.org/pdf/1911.01214v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-richly-annotated-corpus-for-different-tasks-1","repo_url":"https://github.com/UKPLab/conll2019-snopes-crawling","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"a-richly-annotated-corpus-for-different-tasks-1","repo_url":"https://github.com/UKPLab/conll2019-snopes-experiments","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"stance-detection","task_name":"Stance Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1911.01214","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.01214"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/UKPLab/conll2019-snopes-crawling","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/UKPLab/conll2019-snopes-experiments","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7c139ca14cd2a791","entry":"get_whole_evidence","repo":"UKPLab/conll2019-snopes-experiments","repo_kind":"official","path":"src/scripts/evidence_extraction.py","file_url":"https://github.com/UKPLab/conll2019-snopes-experiments/blob/HEAD/src/scripts/evidence_extraction.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7c139ca14cd2a791"}},{"code_sha256_prefix":"d6644ab1b10b325d","entry":"post_processing","repo":"UKPLab/conll2019-snopes-experiments","repo_kind":"official","path":"src/scripts/evidence_extraction.py","file_url":"https://github.com/UKPLab/conll2019-snopes-experiments/blob/HEAD/src/scripts/evidence_extraction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d6644ab1b10b325d"}},{"code_sha256_prefix":"ce24c5f65dd1f635","entry":"post_processing_baseline","repo":"UKPLab/conll2019-snopes-experiments","repo_kind":"official","path":"src/scripts/evidence_extraction.py","file_url":"https://github.com/UKPLab/conll2019-snopes-experiments/blob/HEAD/src/scripts/evidence_extraction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce24c5f65dd1f635"}},{"code_sha256_prefix":"ebb1b9c0b24a6210","entry":"read_new_corpus","repo":"UKPLab/conll2019-snopes-crawling","repo_kind":"official","path":"corpus4_construction/predict.py","file_url":"https://github.com/UKPLab/conll2019-snopes-crawling/blob/HEAD/corpus4_construction/predict.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebb1b9c0b24a6210"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}