{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kieval-a-knowledge-grounded-interactive","title":"KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models","arxiv_id":"2402.15043","date":"2024-02-23","proceeding":null,"authors":["Zhuohao Yu","Chang Gao","Wenjin Yao","Yidong Wang","Wei Ye","Jindong Wang","Xing Xie","Yue Zhang","Shikun Zhang"],"abstract":"Automatic evaluation methods for large language models (LLMs) are hindered by data contamination, leading to inflated assessments of their effectiveness. Existing strategies, which aim to detect contaminated texts, focus on quantifying contamination status instead of accurately gauging model performance. In this paper, we introduce KIEval, a Knowledge-grounded Interactive Evaluation framework, which incorporates an LLM-powered \"interactor\" role for the first time to accomplish a dynamic contamination-resilient evaluation. Starting with a question in a conventional LLM benchmark involving domain-specific knowledge, KIEval utilizes dynamically generated, multi-round, and knowledge-focused dialogues to determine whether a model's response is merely a recall of benchmark answers or demonstrates a deep comprehension to apply knowledge in more complex conversations. Extensive experiments on seven leading LLMs across five datasets validate KIEval's effectiveness and generalization. We also reveal that data contamination brings no contribution or even negative effect to models' real-world applicability and understanding, and existing contamination detection methods for LLMs can only identify contamination in pre-training but not during supervised fine-tuning.","url_abs":"https://arxiv.org/abs/2402.15043v2","url_pdf":"https://arxiv.org/pdf/2402.15043v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kieval-a-knowledge-grounded-interactive","repo_url":"https://github.com/hiyouga/llama-factory","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"kieval-a-knowledge-grounded-interactive","repo_url":"https://github.com/zhuohaoyu/kieval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[{"slug":"kieval","name":"KIEval","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.15043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.15043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhuohaoyu/kieval","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hiyouga/llama-factory","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"844fde9037fc17dc","entry":"apply_multiple_choice_prompt","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"kieval/prompts.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/kieval/prompts.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"844fde9037fc17dc"}},{"code_sha256_prefix":"677ad450c06f8357","entry":"apply_system_prompt","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"kieval/prompts.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/kieval/prompts.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"677ad450c06f8357"}},{"code_sha256_prefix":"804d60471cce5a15","entry":"calculate_inference_endpoint_hash","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"kieval/utils.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/kieval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"804d60471cce5a15"}},{"code_sha256_prefix":"2c7dd36578b477c7","entry":"get_attribute_score","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"get_scores.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/get_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c7dd36578b477c7"}},{"code_sha256_prefix":"e45c7e74926c2fe9","entry":"get_score_list","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"get_scores.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/get_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e45c7e74926c2fe9"}},{"code_sha256_prefix":"d95a76eb2a21e027","entry":"get_stop_reasons","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"get_scores.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/get_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d95a76eb2a21e027"}},{"code_sha256_prefix":"c4b858826a69a60c","entry":"run_pool","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"kieval/models/openai_model.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/kieval/models/openai_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4b858826a69a60c"}},{"code_sha256_prefix":"130bb3f6e9a905b4","entry":"get_gpu_memory","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"kieval/models/local_hf_model.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/kieval/models/local_hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"130bb3f6e9a905b4"}},{"code_sha256_prefix":"36e7c84d050efc4f","entry":"load_model","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"kieval/models/local_hf_model.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/kieval/models/local_hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36e7c84d050efc4f"}},{"code_sha256_prefix":"04ef89ff4ad6b811","entry":"load_pretrained_model","repo":"zhuohaoyu/kieval","repo_kind":"official","path":"kieval/models/local_hf_model.py","file_url":"https://github.com/zhuohaoyu/kieval/blob/HEAD/kieval/models/local_hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04ef89ff4ad6b811"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}