{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/followbench-a-multi-level-fine-grained","title":"FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models","arxiv_id":"2310.20410","date":"2023-10-31","proceeding":null,"authors":["Yuxin Jiang","YuFei Wang","Xingshan Zeng","Wanjun Zhong","Liangyou Li","Fei Mi","Lifeng Shang","Xin Jiang","Qun Liu","Wei Wang"],"abstract":"The ability to follow instructions is crucial for Large Language Models (LLMs) to handle various real-world applications. Existing benchmarks primarily focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs. FollowBench comprehensively includes five different types (i.e., Content, Situation, Style, Format, and Example) of fine-grained constraints. To enable a precise constraint following estimation on diverse difficulties, we introduce a Multi-level mechanism that incrementally adds a single constraint to the initial instruction at each increased level. To assess whether LLMs' outputs have satisfied every individual constraint, we propose to prompt strong LLMs with constraint-evolution paths to handle challenging open-ended instructions. By evaluating 13 closed-source and open-source popular LLMs on FollowBench, we highlight the weaknesses of LLMs in instruction following and point towards potential avenues for future work. The data and code are publicly available at https://github.com/YJiangcm/FollowBench.","url_abs":"https://arxiv.org/abs/2310.20410v3","url_pdf":"https://arxiv.org/pdf/2310.20410v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"followbench-a-multi-level-fine-grained","repo_url":"https://github.com/yjiangcm/followbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.20410","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.20410"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yjiangcm/followbench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":18,"unverified":1},"by_repo_kind":{"official":{"samples":19,"ran":18,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d6ef831d2b39bd8","entry":"aggregate","repo":"yjiangcm/followbench","repo_kind":"official","path":"aggregator.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/aggregator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d6ef831d2b39bd8"}},{"code_sha256_prefix":"7d3a5daea3ca8f59","entry":"contain_word","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/rule_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/rule_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d3a5daea3ca8f59"}},{"code_sha256_prefix":"cf41c4c50a39aa5e","entry":"contain_word","repo":"yjiangcm/followbench","repo_kind":"official","path":"code_zh/rule_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code_zh/rule_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf41c4c50a39aa5e"}},{"code_sha256_prefix":"6e03f4aa34917617","entry":"content_evaluation_prompt","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/gpt4_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/gpt4_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e03f4aa34917617"}},{"code_sha256_prefix":"63623e6c11b6cfb6","entry":"content_evaluation_prompt","repo":"yjiangcm/followbench","repo_kind":"official","path":"code_zh/gpt4_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code_zh/gpt4_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63623e6c11b6cfb6"}},{"code_sha256_prefix":"aac716e321357184","entry":"count_sentences","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/rule_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/rule_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aac716e321357184"}},{"code_sha256_prefix":"4c2603fe3629bf97","entry":"count_sentences","repo":"yjiangcm/followbench","repo_kind":"official","path":"code_zh/rule_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code_zh/rule_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c2603fe3629bf97"}},{"code_sha256_prefix":"427b4358831b5967","entry":"create_example_constraint","repo":"yjiangcm/followbench","repo_kind":"official","path":"create_example_constraint.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/create_example_constraint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"427b4358831b5967"}},{"code_sha256_prefix":"b790283c581282d6","entry":"data_match_api_output","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/utils.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b790283c581282d6"}},{"code_sha256_prefix":"64ccb272d8fc116a","entry":"generate_prompts","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/model_inference_vllm.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/model_inference_vllm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64ccb272d8fc116a"}},{"code_sha256_prefix":"4cd97990a8cad73f","entry":"get_eval","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/llm_eval.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/llm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cd97990a8cad73f"}},{"code_sha256_prefix":"570dd47754890fe2","entry":"get_eval","repo":"yjiangcm/followbench","repo_kind":"official","path":"code_zh/llm_eval.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code_zh/llm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"570dd47754890fe2"}},{"code_sha256_prefix":"5d17b604c968b41d","entry":"get_json_list","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/llm_eval.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/llm_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d17b604c968b41d"}},{"code_sha256_prefix":"e41d3beb442d579b","entry":"situation_evaluation_prompt","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/gpt4_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/gpt4_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e41d3beb442d579b"}},{"code_sha256_prefix":"76ddbccc5e0a80d1","entry":"situation_evaluation_prompt","repo":"yjiangcm/followbench","repo_kind":"official","path":"code_zh/gpt4_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code_zh/gpt4_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76ddbccc5e0a80d1"}},{"code_sha256_prefix":"8773f84b53d306df","entry":"split_integer","repo":"yjiangcm/followbench","repo_kind":"official","path":"create_example_constraint.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/create_example_constraint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8773f84b53d306df"}},{"code_sha256_prefix":"575501debe301b98","entry":"style_evaluation_prompt","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/gpt4_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/gpt4_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"575501debe301b98"}},{"code_sha256_prefix":"e9818a2f75966147","entry":"style_evaluation_prompt","repo":"yjiangcm/followbench","repo_kind":"official","path":"code_zh/gpt4_based_evaluation.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code_zh/gpt4_based_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9818a2f75966147"}},{"code_sha256_prefix":"d607ca385609c6fa","entry":"generate_responses","repo":"yjiangcm/followbench","repo_kind":"official","path":"code/model_inference_vllm.py","file_url":"https://github.com/yjiangcm/followbench/blob/HEAD/code/model_inference_vllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d607ca385609c6fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}