{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-faithful-and-robust-llm-specialists","title":"Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering","arxiv_id":"2402.08277","date":"2024-02-13","proceeding":null,"authors":["Tobias Schimanski","Jingwei Ni","Mathias Kraus","Elliott Ash","Markus Leippold"],"abstract":"Advances towards more faithful and traceable answers of Large Language Models (LLMs) are crucial for various research and practical endeavors. One avenue in reaching this goal is basing the answers on reliable sources. However, this Evidence-Based QA has proven to work insufficiently with LLMs in terms of citing the correct sources (source quality) and truthfully representing the information within sources (answer attributability). In this work, we systematically investigate how to robustly fine-tune LLMs for better source quality and answer attributability. Specifically, we introduce a data generation pipeline with automated data quality filters, which can synthesize diversified high-quality training and testing data at scale. We further introduce four test sets to benchmark the robustness of fine-tuned specialist models. Extensive evaluation shows that fine-tuning on synthetic data improves performance on both in- and out-of-distribution. Furthermore, we show that data quality, which can be drastically improved by proposed quality filters, matters more than quantity in improving Evidence-Based QA.","url_abs":"https://arxiv.org/abs/2402.08277v5","url_pdf":"https://arxiv.org/pdf/2402.08277v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-faithful-and-robust-llm-specialists","repo_url":"https://github.com/EdisonNi-hku/Robust_Evidence_Based_QA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.08277","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.08277"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EdisonNi-hku/Robust_Evidence_Based_QA","reach":null}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4fe5cd007605b269","entry":"createQuestions","repo":"EdisonNi-hku/Robust_Evidence_Based_QA","repo_kind":"official","path":"data_generation/CreateSyntheticData.py","file_url":"https://github.com/EdisonNi-hku/Robust_Evidence_Based_QA/blob/HEAD/data_generation/CreateSyntheticData.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fe5cd007605b269"}},{"code_sha256_prefix":"c90d225e793f6530","entry":"createSources","repo":"EdisonNi-hku/Robust_Evidence_Based_QA","repo_kind":"official","path":"data_generation/CreateSyntheticData.py","file_url":"https://github.com/EdisonNi-hku/Robust_Evidence_Based_QA/blob/HEAD/data_generation/CreateSyntheticData.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c90d225e793f6530"}},{"code_sha256_prefix":"52d8e56fc22190ac","entry":"createTopicData","repo":"EdisonNi-hku/Robust_Evidence_Based_QA","repo_kind":"official","path":"data_generation/CreateSyntheticData.py","file_url":"https://github.com/EdisonNi-hku/Robust_Evidence_Based_QA/blob/HEAD/data_generation/CreateSyntheticData.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52d8e56fc22190ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}