{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/freeeval-a-modular-framework-for-trustworthy","title":"FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models","arxiv_id":"2404.06003","date":"2024-04-09","proceeding":null,"authors":["Zhuohao Yu","Chang Gao","Wenjin Yao","Yidong Wang","Zhengran Zeng","Wei Ye","Jindong Wang","Yue Zhang","Shikun Zhang"],"abstract":"The rapid development of large language model (LLM) evaluation methodologies and datasets has led to a profound challenge: integrating state-of-the-art evaluation techniques cost-effectively while ensuring reliability, reproducibility, and efficiency. Currently, there is a notable absence of a unified and adaptable framework that seamlessly integrates various evaluation approaches. Moreover, the reliability of evaluation findings is often questionable due to potential data contamination, with the evaluation efficiency commonly overlooked when facing the substantial costs associated with LLM inference. In response to these challenges, we introduce FreeEval, a modular and scalable framework crafted to enable trustworthy and efficient automatic evaluations of LLMs. Firstly, FreeEval's unified abstractions simplify the integration and improve the transparency of diverse evaluation methodologies, encompassing dynamic evaluation that demand sophisticated LLM interactions. Secondly, the framework integrates meta-evaluation techniques like human evaluation and data contamination detection, which, along with dynamic evaluation modules in the platform, enhance the fairness of the evaluation outcomes. Lastly, FreeEval is designed with a high-performance infrastructure, including distributed computation and caching strategies, enabling extensive evaluations across multi-node, multi-GPU clusters for open-source and proprietary LLMs.","url_abs":"https://arxiv.org/abs/2404.06003v1","url_pdf":"https://arxiv.org/pdf/2404.06003v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"freeeval-a-modular-framework-for-trustworthy","repo_url":"https://github.com/wisdomshell/freeeval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.06003","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.06003"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wisdomshell/freeeval","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"844fde9037fc17dc","entry":"apply_multiple_choice_prompt","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/prompts.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"844fde9037fc17dc"}},{"code_sha256_prefix":"677ad450c06f8357","entry":"apply_system_prompt","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/prompts.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"677ad450c06f8357"}},{"code_sha256_prefix":"804d60471cce5a15","entry":"calculate_inference_endpoint_hash","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/utils.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"804d60471cce5a15"}},{"code_sha256_prefix":"a5c8d7b6364573b9","entry":"get_model_nicename","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/utils.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5c8d7b6364573b9"}},{"code_sha256_prefix":"c4b858826a69a60c","entry":"run_pool","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/models/openai_model.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/models/openai_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4b858826a69a60c"}},{"code_sha256_prefix":"130bb3f6e9a905b4","entry":"get_gpu_memory","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/models/local_hf_model.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/models/local_hf_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"130bb3f6e9a905b4"}},{"code_sha256_prefix":"36e7c84d050efc4f","entry":"load_model","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/models/local_hf_model.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/models/local_hf_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36e7c84d050efc4f"}},{"code_sha256_prefix":"04ef89ff4ad6b811","entry":"load_pretrained_model","repo":"wisdomshell/freeeval","repo_kind":"official","path":"freeeval/models/local_hf_model.py","file_url":"https://github.com/wisdomshell/freeeval/blob/HEAD/freeeval/models/local_hf_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04ef89ff4ad6b811"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}