{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/draw-all-your-imagine-a-holistic-benchmark","title":"Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation","arxiv_id":"2505.24787","date":"2025-05-30","proceeding":null,"authors":["Yucheng Zhou","Jiahao Yuan","Qianning Wang"],"abstract":"Recent advancements in text-to-image (T2I) generation have enabled models to produce high-quality images from textual descriptions. However, these models often struggle with complex instructions involving multiple objects, attributes, and spatial relationships. Existing benchmarks for evaluating T2I models primarily focus on general text-image alignment and fail to capture the nuanced requirements of complex, multi-faceted prompts. Given this gap, we introduce LongBench-T2I, a comprehensive benchmark specifically designed to evaluate T2I models under complex instructions. LongBench-T2I consists of 500 intricately designed prompts spanning nine diverse visual evaluation dimensions, enabling a thorough assessment of a model's ability to follow complex instructions. Beyond benchmarking, we propose an agent framework (Plan2Gen) that facilitates complex instruction-driven image generation without requiring additional model training. This framework integrates seamlessly with existing T2I models, using large language models to interpret and decompose complex prompts, thereby guiding the generation process more effectively. As existing evaluation metrics, such as CLIPScore, fail to adequately capture the nuances of complex instructions, we introduce an evaluation toolkit that automates the quality assessment of generated images using a set of multi-dimensional metrics. The data and code are released at https://github.com/yczhou001/LongBench-T2I.","url_abs":"https://arxiv.org/abs/2505.24787v1","url_pdf":"https://arxiv.org/pdf/2505.24787v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"draw-all-your-imagine-a-holistic-benchmark","repo_url":"https://github.com/yczhou001/longbench-t2i","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.24787","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.24787"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yczhou001/longbench-t2i","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c577dc3012980327","entry":"build_transform","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"utils/evaluator.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/utils/evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c577dc3012980327"}},{"code_sha256_prefix":"f81449fe50195856","entry":"dynamic_preprocess","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"utils/evaluator.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/utils/evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f81449fe50195856"}},{"code_sha256_prefix":"4dd5fef7794ff9f7","entry":"extract_summary","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dd5fef7794ff9f7"}},{"code_sha256_prefix":"19b3dd09dfd84329","entry":"find_deepest_step_folders","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19b3dd09dfd84329"}},{"code_sha256_prefix":"1797c9b5ac67965f","entry":"get_latest_edited_image_file","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1797c9b5ac67965f"}},{"code_sha256_prefix":"3f38b3ee482473e5","entry":"hash_scene","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f38b3ee482473e5"}},{"code_sha256_prefix":"c4e616558cb2713e","entry":"is_valid_json","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4e616558cb2713e"}},{"code_sha256_prefix":"8ff2516a7846c307","entry":"load_image","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"utils/evaluator.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/utils/evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ff2516a7846c307"}},{"code_sha256_prefix":"fdea7da6487b6a46","entry":"load_jsonl","repo":"yczhou001/longbench-t2i","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/yczhou001/longbench-t2i/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdea7da6487b6a46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}