{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multihuman-testbench-benchmarking-image","title":"MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans","arxiv_id":"2506.20879","date":"2025-06-25","proceeding":null,"authors":["Shubhankar Borse","Seokeon Choi","Sunghyun Park","Jeongho Kim","Shreya Kadambi","Risheek Garrepalli","Sungrack Yun","Munawar Hayat","Fatih Porikli"],"abstract":"Generation of images containing multiple humans, performing complex actions, while preserving their facial identities, is a significant challenge. A major factor contributing to this is the lack of a a dedicated benchmark. To address this, we introduce MultiHuman-Testbench, a novel benchmark for rigorously evaluating generative models for multi-human generation. The benchmark comprises 1800 samples, including carefully curated text prompts, describing a range of simple to complex human actions. These prompts are matched with a total of 5,550 unique human face images, sampled uniformly to ensure diversity across age, ethnic background, and gender. Alongside captions, we provide human-selected pose conditioning images which accurately match the prompt. We propose a multi-faceted evaluation suite employing four key metrics to quantify face count, ID similarity, prompt alignment, and action detection. We conduct a thorough evaluation of a diverse set of models, including zero-shot approaches and training-based methods, with and without regional priors. We also propose novel techniques to incorporate image and region isolation using human segmentation and Hungarian matching, significantly improving ID similarity. Our proposed benchmark and key findings provide valuable insights and a standardized tool for advancing research in multi-human image generation.","url_abs":"https://arxiv.org/abs/2506.20879v1","url_pdf":"https://arxiv.org/pdf/2506.20879v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.20879","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.20879"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/feifeiobama/RectifID","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Qualcomm-AI-research/MultiHuman-Testbench","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/eclipse-t2i/lambda-eclipse-inference","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":5,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"f44d9ec80f04b12d","entry":"load_lookup_table","repo":"Qualcomm-AI-research/MultiHuman-Testbench","repo_kind":"found_in_text","path":"data_generation/download_faces.py","file_url":"https://github.com/Qualcomm-AI-research/MultiHuman-Testbench/blob/HEAD/data_generation/download_faces.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"f44d9ec80f04b12d"}},{"code_sha256_prefix":"b6a6189df614c4c0","entry":"load_metadata","repo":"Qualcomm-AI-research/MultiHuman-Testbench","repo_kind":"found_in_text","path":"inference/generate_omnigen_images.py","file_url":"https://github.com/Qualcomm-AI-research/MultiHuman-Testbench/blob/HEAD/inference/generate_omnigen_images.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"b6a6189df614c4c0"}},{"code_sha256_prefix":"fe4870673b073f05","entry":"get_image_grid","repo":"eclipse-t2i/lambda-eclipse-inference","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/eclipse-t2i/lambda-eclipse-inference/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe4870673b073f05"}},{"code_sha256_prefix":"86a14efcb395eae8","entry":"linear_interpolation","repo":"eclipse-t2i/lambda-eclipse-inference","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/eclipse-t2i/lambda-eclipse-inference/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86a14efcb395eae8"}},{"code_sha256_prefix":"f76e8b9fc2869a1f","entry":"slerp","repo":"eclipse-t2i/lambda-eclipse-inference","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/eclipse-t2i/lambda-eclipse-inference/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f76e8b9fc2869a1f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}