{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tabfact-a-large-scale-dataset-for-table-based","title":"TabFact: A Large-scale Dataset for Table-based Fact Verification","arxiv_id":"1909.02164","date":"2019-09-05","proceeding":"ICLR 2020 1","authors":["Wenhu Chen","Hongmin Wang","Jianshu Chen","Yunkai Zhang","Hong Wang","Shiyang Li","Xiyou Zhou","William Yang Wang"],"abstract":"The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities. The data and code of the dataset are provided in \\url{https://github.com/wenhuchen/Table-Fact-Checking}.","url_abs":"https://arxiv.org/abs/1909.02164v5","url_pdf":"https://arxiv.org/pdf/1909.02164v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tabfact-a-large-scale-dataset-for-table-based","repo_url":"https://github.com/wenhuchen/Table-Fact-Checking","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"16k","task_name":"16k"},{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"fact-verification","task_name":"Fact Verification"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"table-based-fact-verification","task_name":"Table-based Fact Verification"}],"methods":[],"datasets_introduced":[{"slug":"tabfact","name":"TabFact","full_name":"TabFact"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/table-based-fact-verification-on-tabfact","task":"Table-based Fact Verification","dataset":"TabFact","model":"Table-BERT-Horizontal-T+F-Template","rank_in_archive_order":14,"of":15,"metrics":{"Test":"65.12","Val":"66.1"},"uses_additional_data":false},{"leaderboard":"/sota/table-based-fact-verification-on-tabfact","task":"Table-based Fact Verification","dataset":"TabFact","model":"BERT classifier w/o Table","rank_in_archive_order":15,"of":15,"metrics":{"Test":"50.5","Val":"50.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1909.02164","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.02164"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wenhuchen/Table-Fact-Checking","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"cac113ca87b9d9f3","entry":"acc_and_f1","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"cac113ca87b9d9f3"}},{"code_sha256_prefix":"405ee22d52c27fac","entry":"convert_examples_to_features","repo":"wenhuchen/Table-Fact-Checking","repo_kind":"official","path":"code/run_BERT.py","file_url":"https://github.com/wenhuchen/Table-Fact-Checking/blob/HEAD/code/run_BERT.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"405ee22d52c27fac"}},{"code_sha256_prefix":"3c241ecfe3749a6d","entry":"simple_accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3c241ecfe3749a6d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}