{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/numtemp-a-real-world-benchmark-to-verify","title":"QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims","arxiv_id":"2403.17169","date":"2024-03-25","proceeding":null,"authors":["Venktesh V","Abhijit Anand","Avishek Anand","Vinay Setty"],"abstract":"Automated fact checking has gained immense interest to tackle the growing misinformation in the digital era. Existing systems primarily focus on synthetic claims on Wikipedia, and noteworthy progress has also been made on real-world claims. In this work, we release QuanTemp, a diverse, multi-domain dataset focused exclusively on numerical claims, encompassing temporal, statistical and diverse aspects with fine-grained metadata and an evidence collection without leakage. This addresses the challenge of verifying real-world numerical claims, which are complex and often lack precise information, not addressed by existing works that mainly focus on synthetic claims. We evaluate and quantify the limitations of existing solutions for the task of verifying numerical claims. We also evaluate claim decomposition based methods, numerical understanding based models and our best baselines achieves a macro-F1 of 58.32. This demonstrates that QuanTemp serves as a challenging evaluation set for numerical claim verification.","url_abs":"https://arxiv.org/abs/2403.17169v3","url_pdf":"https://arxiv.org/pdf/2403.17169v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"numtemp-a-real-world-benchmark-to-verify","repo_url":"https://github.com/factiverse/QuanTemp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"claim-verification","task_name":"Claim Verification"},{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"misinformation","task_name":"Misinformation"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.17169","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.17169"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/factiverse/QuanTemp","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"1a922b941953a1d5","entry":"get_subtree","repo":"factiverse/QuanTemp","repo_kind":"official","path":"code/data_processing/quantity_segmenter.py","file_url":"https://github.com/factiverse/QuanTemp/blob/HEAD/code/data_processing/quantity_segmenter.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a922b941953a1d5"}},{"code_sha256_prefix":"e38732e9dba8f18c","entry":"get_tree","repo":"factiverse/QuanTemp","repo_kind":"official","path":"code/data_processing/quantity_segmenter.py","file_url":"https://github.com/factiverse/QuanTemp/blob/HEAD/code/data_processing/quantity_segmenter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e38732e9dba8f18c"}},{"code_sha256_prefix":"00660898c2b216db","entry":"read_claims_data","repo":"factiverse/QuanTemp","repo_kind":"official","path":"code/evaluation/eval_veracity_prediction.py","file_url":"https://github.com/factiverse/QuanTemp/blob/HEAD/code/evaluation/eval_veracity_prediction.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00660898c2b216db"}},{"code_sha256_prefix":"b785dde8762c3539","entry":"construct_parse","repo":"factiverse/QuanTemp","repo_kind":"official","path":"code/data_processing/quantity_segmenter.py","file_url":"https://github.com/factiverse/QuanTemp/blob/HEAD/code/data_processing/quantity_segmenter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b785dde8762c3539"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}