{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-large-language-models-on","title":"Benchmarking Large Language Models on Controllable Generation under Diversified Instructions","arxiv_id":"2401.00690","date":"2024-01-01","proceeding":null,"authors":["Yihan Chen","Benfeng Xu","Quan Wang","Yi Liu","Zhendong Mao"],"abstract":"While large language models (LLMs) have exhibited impressive instruction-following capabilities, it is still unclear whether and to what extent they can respond to explicit constraints that might be entailed in various instructions. As a significant aspect of LLM alignment, it is thus important to formulate such a specialized set of instructions as well as investigate the resulting behavior of LLMs. To address this vacancy, we propose a new benchmark CoDI-Eval to systematically and comprehensively evaluate LLMs' responses to instructions with various constraints. We construct a large collection of constraints-attributed instructions as a test suite focused on both generalization and coverage. Specifically, we advocate an instruction diversification process to synthesize diverse forms of constraint expression and also deliberate the candidate task taxonomy with even finer-grained sub-categories. Finally, we automate the entire evaluation process to facilitate further developments. Different from existing studies on controllable text generation, CoDI-Eval extends the scope to the prevalent instruction-following paradigm for the first time. We provide extensive evaluations of representative LLMs (e.g., ChatGPT, Vicuna) on CoDI-Eval, revealing their limitations in following instructions with specific constraints and there is still a significant gap between open-source and commercial closed-source LLMs. We believe this benchmark will facilitate research into improving the controllability of LLMs' responses to instructions. Our data and code are available at https://github.com/Xt-cyh/CoDI-Eval.","url_abs":"https://arxiv.org/abs/2401.00690v1","url_pdf":"https://arxiv.org/pdf/2401.00690v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-large-language-models-on","repo_url":"https://github.com/xt-cyh/codi-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.00690","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.00690"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xt-cyh/codi-eval","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Xt-cyh/CoDI-Eval","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":5},"by_repo_kind":{"official":{"samples":18,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"5049acee0b271f75","entry":"cal_max_avg_toxicity","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"statistic.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/statistic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5049acee0b271f75"}},{"code_sha256_prefix":"71e266cfd0e6e573","entry":"classify_sentiment","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71e266cfd0e6e573"}},{"code_sha256_prefix":"174067d7b3c2275d","entry":"classify_topic","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"174067d7b3c2275d"}},{"code_sha256_prefix":"e551f21cb1015ac1","entry":"extract_text","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/eval_utils.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e551f21cb1015ac1"}},{"code_sha256_prefix":"280739482fbffd74","entry":"get_data","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"statistic.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/statistic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"280739482fbffd74"}},{"code_sha256_prefix":"09c18b0f6797ff78","entry":"get_interval","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/length_eval.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/length_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09c18b0f6797ff78"}},{"code_sha256_prefix":"cc2b254a5f917403","entry":"get_test_data","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc2b254a5f917403"}},{"code_sha256_prefix":"ac6fcd5d8bf62759","entry":"get_test_data","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"generate.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac6fcd5d8bf62759"}},{"code_sha256_prefix":"39b482520756e734","entry":"number2word","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/length_eval.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/length_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39b482520756e734"}},{"code_sha256_prefix":"cd407b6ff1fc7aef","entry":"parse_topic_dict","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/query_huggingface.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/query_huggingface.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd407b6ff1fc7aef"}},{"code_sha256_prefix":"aa85076767ed9c7a","entry":"query","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/query_huggingface.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/query_huggingface.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa85076767ed9c7a"}},{"code_sha256_prefix":"92ff2d437c56a6f0","entry":"query_chatglm","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/query_llms.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/query_llms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92ff2d437c56a6f0"}},{"code_sha256_prefix":"89f24dd774d0e047","entry":"query_chatgpt","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/query_llms.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/query_llms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89f24dd774d0e047"}},{"code_sha256_prefix":"4514bd59c017bc70","entry":"calculate_meteor","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/eval_utils.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4514bd59c017bc70"}},{"code_sha256_prefix":"850fb04ea11f610e","entry":"get_open_model","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"generate.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"850fb04ea11f610e"}},{"code_sha256_prefix":"ac4e1313c5c2e9c4","entry":"preprocess","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/eval_utils.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac4e1313c5c2e9c4"}},{"code_sha256_prefix":"8aa7d1340bb82140","entry":"query_api_model","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"generate.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8aa7d1340bb82140"}},{"code_sha256_prefix":"a54a222cec75c54a","entry":"request_query","repo":"Xt-cyh/CoDI-Eval","repo_kind":"official","path":"utils/query_huggingface.py","file_url":"https://github.com/Xt-cyh/CoDI-Eval/blob/HEAD/utils/query_huggingface.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a54a222cec75c54a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}