{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bigbench-a-unified-benchmark-for-social-bias","title":"BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models","arxiv_id":"2407.15240","date":"2024-07-21","proceeding":null,"authors":["Hanjun Luo","Haoyu Huang","Ziye Deng","Xinfeng Li","Hewei Wang","Yingbin Jin","Yang Liu","Wenyuan Xu","Zuozhu Liu"],"abstract":"Text-to-Image (T2I) generative models are becoming increasingly crucial due to their ability to generate high-quality images, but also raise concerns about social biases, particularly in human image generation. Sociological research has established systematic classifications of bias. Yet, existing studies on bias in T2I models largely conflate different types of bias, impeding methodological progress. In this paper, we introduce BIGbench, a unified benchmark for Biases of Image Generation, featuring a carefully designed dataset. Unlike existing benchmarks, BIGbench classifies and evaluates biases across four dimensions to enable a more granular evaluation and deeper analysis. Furthermore, BIGbench applies advanced multi-modal large language models to achieve fully automated and highly accurate evaluations. We apply BIGbench to evaluate eight representative T2I models and three debiasing methods. Our human evaluation results by trained evaluators from different races underscore BIGbench's effectiveness in aligning images and identifying various biases. Moreover, our study also reveals new research directions about biases with insightful analysis of our results. Our work is openly accessible at https://github.com/BIGbench2024/BIGbench2024/.","url_abs":"https://arxiv.org/abs/2407.15240v6","url_pdf":"https://arxiv.org/pdf/2407.15240v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bigbench-a-unified-benchmark-for-social-bias","repo_url":"https://github.com/bigbench2024/bigbench2024","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.15240","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.15240"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigbench2024/bigbench2024","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":10,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"57e95166ade5f25c","entry":"build_transform","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/internViT_pkg/internvl_detection.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/internViT_pkg/internvl_detection.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"57e95166ade5f25c"}},{"code_sha256_prefix":"1cfda480b860f7a7","entry":"calculate_cosine_similarity","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/evaluate/implicit.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/evaluate/implicit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1cfda480b860f7a7"}},{"code_sha256_prefix":"7cf9188777086665","entry":"calculate_levels","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/evaluate/explicit.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/evaluate/explicit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7cf9188777086665"}},{"code_sha256_prefix":"c780b9b7a97706aa","entry":"calculate_similarity_for_prompts","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/evaluate/implicit.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/evaluate/implicit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c780b9b7a97706aa"}},{"code_sha256_prefix":"4d05bcc50be7d25e","entry":"dynamic_preprocess","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/internViT_pkg/internvl_detection.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/internViT_pkg/internvl_detection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4d05bcc50be7d25e"}},{"code_sha256_prefix":"d0487594e4d5c510","entry":"find_closest_aspect_ratio","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/internViT_pkg/internvl_detection.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/internViT_pkg/internvl_detection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d0487594e4d5c510"}},{"code_sha256_prefix":"cf58e9dc07391ea5","entry":"load_json","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/evaluate/eta.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/evaluate/eta.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cf58e9dc07391ea5"}},{"code_sha256_prefix":"35cf0bcb51d887c8","entry":"load_json","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/evaluate/implicit.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/evaluate/implicit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"35cf0bcb51d887c8"}},{"code_sha256_prefix":"a2205613ccd243e1","entry":"load_prompts","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/generate/generate.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/generate/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a2205613ccd243e1"}},{"code_sha256_prefix":"048495666545e7a4","entry":"process_data","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/evaluate/explicit.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/evaluate/explicit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"048495666545e7a4"}},{"code_sha256_prefix":"859bc0e0407bbfa7","entry":"truncate_prompt","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/generate/generate.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/generate/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"859bc0e0407bbfa7"}},{"code_sha256_prefix":"bcac058322b4e067","entry":"calculate_eta","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/evaluate/eta.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/evaluate/eta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bcac058322b4e067"}},{"code_sha256_prefix":"0b5d2e235d9be7ce","entry":"queue_prompt","repo":"bigbench2024/bigbench2024","repo_kind":"official","path":"benchmark/generate/generate.py","file_url":"https://github.com/bigbench2024/bigbench2024/blob/HEAD/benchmark/generate/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0b5d2e235d9be7ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}