{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xfinder-robust-and-pinpoint-answer-extraction","title":"xFinder: Robust and Pinpoint Answer Extraction for Large Language Models","arxiv_id":"2405.11874","date":"2024-05-20","proceeding":null,"authors":["Qingchen Yu","Zifan Zheng","Shichao Song","Zhiyu Li","Feiyu Xiong","Bo Tang","Ding Chen"],"abstract":"The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of subjective or non-subjective cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. Since evaluation frameworks often utilize Regular Expression (RegEx) for answer extraction, some models may adjust their responses to comply with specific formats that are easily extractable by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. This paper conducts a comprehensive analysis of the entire LLM evaluation chain, demonstrating that optimizing the key answer extraction module can improve extraction accuracy, reduce LLMs' reliance on specific answer formats, and enhance the reliability of LLM evaluation. To address these issues, we propose xFinder, a model specifically designed for key answer extraction. As part of this process, we create a specialized dataset, the Key Answer Finder (KAF) dataset, to ensure effective model training and evaluation. Through generalization testing and evaluation in real-world scenarios, the results demonstrate that the smallest xFinder model with only 500 million parameters achieves an average answer extraction accuracy of 93.42%. In contrast, RegEx accuracy in the best evaluation framework is 74.38%. xFinder exhibits stronger robustness and higher accuracy compared to existing evaluation frameworks.","url_abs":"https://arxiv.org/abs/2405.11874v2","url_pdf":"https://arxiv.org/pdf/2405.11874v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xfinder-robust-and-pinpoint-answer-extraction","repo_url":"https://github.com/iaar-shanghai/xfinder","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.11874","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.11874"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/IAAR-Shanghai/xFinder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iaar-shanghai/xfinder","reach":null}],"summary":{"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"352121e0de5ba153","entry":"read_jsonl","repo":"IAAR-Shanghai/xFinder","repo_kind":"official","path":"scripts/dataset_construction/create_benchmark_dataset.py","file_url":"https://github.com/IAAR-Shanghai/xFinder/blob/HEAD/scripts/dataset_construction/create_benchmark_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"352121e0de5ba153"}},{"code_sha256_prefix":"8875e4bc83a27f31","entry":"find_json_files_with_content","repo":"IAAR-Shanghai/xFinder","repo_kind":"official","path":"scripts/dataset_construction/create_benchmark_dataset.py","file_url":"https://github.com/IAAR-Shanghai/xFinder/blob/HEAD/scripts/dataset_construction/create_benchmark_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8875e4bc83a27f31"}},{"code_sha256_prefix":"ca90f4c9f084240f","entry":"process_row","repo":"IAAR-Shanghai/xFinder","repo_kind":"official","path":"scripts/dataset_construction/create_benchmark_dataset.py","file_url":"https://github.com/IAAR-Shanghai/xFinder/blob/HEAD/scripts/dataset_construction/create_benchmark_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ca90f4c9f084240f"}},{"code_sha256_prefix":"bc9453be7dc5d8fc","entry":"Extractor","repo":"iaar-shanghai/xfinder","repo_kind":"official","path":"src/xfinder/modules/answer_extractor.py","file_url":"https://github.com/iaar-shanghai/xfinder/blob/HEAD/src/xfinder/modules/answer_extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bc9453be7dc5d8fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}