{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-the-factuality-of-large-language","title":"Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs","arxiv_id":"2404.00942","date":"2024-04-01","proceeding":null,"authors":["Xiaoze Liu","Feijie Wu","Tianyang Xu","Zhuo Chen","Yichi Zhang","Xiaoqian Wang","Jing Gao"],"abstract":"The advent of Large Language Models (LLMs) has significantly transformed the AI landscape, enhancing machine learning and AI capabilities. Factuality issue is a critical concern for LLMs, as they may generate factually incorrect responses. In this paper, we propose GraphEval to evaluate an LLM's performance using a substantially large test dataset. Specifically, the test dataset is retrieved from a large knowledge graph with more than 10 million facts without expensive human efforts. Unlike conventional methods that evaluate LLMs based on generated responses, GraphEval streamlines the evaluation process by creating a judge model to estimate the correctness of the answers given by the LLM. Our experiments demonstrate that the judge model's factuality assessment aligns closely with the correctness of the LLM's generated outputs, while also substantially reducing evaluation costs. Besides, our findings offer valuable insights into LLM performance across different metrics and highlight the potential for future improvements in ensuring the factual integrity of LLM outputs. The code is publicly available at https://github.com/xz-liu/GraphEval.","url_abs":"https://arxiv.org/abs/2404.00942v1","url_pdf":"https://arxiv.org/pdf/2404.00942v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-the-factuality-of-large-language","repo_url":"https://github.com/xz-liu/grapheval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.00942","atlas_url":"https://app.syntology.ai/?focus=2404.00942","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.00942"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xz-liu/grapheval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6226e3dad016f9d2","entry":"add_system_prompt","repo":"xz-liu/grapheval","repo_kind":"official","path":"data_gen/llm_clients.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/data_gen/llm_clients.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6226e3dad016f9d2"}},{"code_sha256_prefix":"a6c195c9c89ceadf","entry":"default_construct_instruct","repo":"xz-liu/grapheval","repo_kind":"official","path":"data_gen/data_generator.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/data_gen/data_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6c195c9c89ceadf"}},{"code_sha256_prefix":"119ea02d1a7381e8","entry":"judge_construct_instruct","repo":"xz-liu/grapheval","repo_kind":"official","path":"data_gen/data_generator.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/data_gen/data_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"119ea02d1a7381e8"}},{"code_sha256_prefix":"a4b513b497aa558a","entry":"judge_label_transform","repo":"xz-liu/grapheval","repo_kind":"official","path":"data_gen/data_generator.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/data_gen/data_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4b513b497aa558a"}},{"code_sha256_prefix":"a767c526ef6de6f0","entry":"trained_clf_path","repo":"xz-liu/grapheval","repo_kind":"official","path":"args/config.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/args/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a767c526ef6de6f0"}},{"code_sha256_prefix":"c0de8d188874159b","entry":"trained_prompt_encoder_path","repo":"xz-liu/grapheval","repo_kind":"official","path":"args/config.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/args/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0de8d188874159b"}},{"code_sha256_prefix":"5392f7f93e7db6c3","entry":"training_triples_path","repo":"xz-liu/grapheval","repo_kind":"official","path":"args/config.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/args/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5392f7f93e7db6c3"}},{"code_sha256_prefix":"ef1daf1476254943","entry":"prepare_pageview_based_triples","repo":"xz-liu/grapheval","repo_kind":"official","path":"data_gen/triple_sampler.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/data_gen/triple_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef1daf1476254943"}},{"code_sha256_prefix":"4cfdd6346fb4740a","entry":"prepare_random_triples","repo":"xz-liu/grapheval","repo_kind":"official","path":"data_gen/triple_sampler.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/data_gen/triple_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cfdd6346fb4740a"}},{"code_sha256_prefix":"0c87eb893d43e1cb","entry":"prepare_triples_based_on_entities","repo":"xz-liu/grapheval","repo_kind":"official","path":"data_gen/triple_sampler.py","file_url":"https://github.com/xz-liu/grapheval/blob/HEAD/data_gen/triple_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c87eb893d43e1cb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}