{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/idgen-item-discrimination-induced-prompt","title":"IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation","arxiv_id":"2409.18892","date":"2024-09-27","proceeding":null,"authors":["Fan Lin","Shuyi Xie","Yong Dai","Wenlin Yao","Tianjiao Lang","Zishan Xu","Zhichao Hu","Xiao Xiao","Yuhong Liu","Yu Zhang"],"abstract":"As Large Language Models (LLMs) grow increasingly adept at managing complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items to differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs to ensure the evaluation set can continually update and refine according to model abilities. Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains. To produce high-quality data, we incorporate a self-correct mechanism into our generalization framework, and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves an average score of 51.92, accompanied by a variance of 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain an average score exceeding 67, with a variance below 3.2. The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.","url_abs":"https://arxiv.org/abs/2409.18892v2","url_pdf":"https://arxiv.org/pdf/2409.18892v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"idgen-item-discrimination-induced-prompt","repo_url":"https://github.com/DUTlf/IDGen","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"specificity","task_name":"Specificity"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.18892","atlas_url":"https://app.syntology.ai/?focus=2409.18892","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.18892"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DUTlf/IDGen","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"8f1ffe32d80d0e4f","entry":"DifficultyPredictor","repo":"DUTlf/IDGen","repo_kind":"official","path":"src/mainline.py","file_url":"https://github.com/DUTlf/IDGen/blob/HEAD/src/mainline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f1ffe32d80d0e4f"}},{"code_sha256_prefix":"7cee32eeb4998ed7","entry":"DiscriminationPredictor","repo":"DUTlf/IDGen","repo_kind":"official","path":"src/mainline.py","file_url":"https://github.com/DUTlf/IDGen/blob/HEAD/src/mainline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7cee32eeb4998ed7"}},{"code_sha256_prefix":"1f9fa5747947630b","entry":"QuestionProcessor","repo":"DUTlf/IDGen","repo_kind":"official","path":"src/mainline.py","file_url":"https://github.com/DUTlf/IDGen/blob/HEAD/src/mainline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f9fa5747947630b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}