{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dall-eval-probing-the-reasoning-skills-and","title":"DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models","arxiv_id":"2202.04053","date":"2022-02-08","proceeding":"ICCV 2023 1","authors":["Jaemin Cho","Abhay Zala","Mohit Bansal"],"abstract":"Recently, DALL-E, a multimodal transformer language model, and its variants, including diffusion models, have shown high-quality text-to-image generation capabilities. However, despite the realistic image generation results, there has not been a detailed analysis of how to evaluate such models. In this work, we investigate the visual reasoning capabilities and social biases of different text-to-image models, covering both multimodal transformer language models and diffusion models. First, we measure three visual reasoning skills: object recognition, object counting, and spatial relation understanding. For this, we propose PaintSkills, a compositional diagnostic evaluation dataset that measures these skills. Despite the high-fidelity image generation capability, a large gap exists between the performance of recent models and the upper bound accuracy in object counting and spatial relation understanding skills. Second, we assess the gender and skin tone biases by measuring the gender/skin tone distribution of generated images across various professions and attributes. We demonstrate that recent text-to-image generation models learn specific biases about gender and skin tone from web image-text pairs. We hope our work will help guide future progress in improving text-to-image generation models on visual reasoning skills and learning socially unbiased representations. Code and data: https://github.com/j-min/DallEval","url_abs":"https://arxiv.org/abs/2202.04053v3","url_pdf":"https://arxiv.org/pdf/2202.04053v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dall-eval-probing-the-reasoning-skills-and","repo_url":"https://github.com/j-min/dalleval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"dall-eval-probing-the-reasoning-skills-and","repo_url":"https://github.com/kakaobrain/rq-vae-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-counting","task_name":"Object Counting"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2202.04053","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.04053"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kakaobrain/rq-vae-transformer","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/j-min/dalleval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"ran_fixture":1,"ran_violates":1,"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"177298ecedde13ed","entry":"calculate_activation_statistics","repo":"j-min/dalleval","repo_kind":"official","path":"quality/FID/fid_score.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/quality/FID/fid_score.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"177298ecedde13ed"}},{"code_sha256_prefix":"25ef415d07e43e6f","entry":"calculate_frechet_distance","repo":"j-min/dalleval","repo_kind":"official","path":"quality/FID/fid_score.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/quality/FID/fid_score.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25ef415d07e43e6f"}},{"code_sha256_prefix":"7ae941c6130274cb","entry":"cp_path_to_dir","repo":"j-min/dalleval","repo_kind":"official","path":"models/dalle_small/DALLE-pytorch/finetune_dalle.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/models/dalle_small/DALLE-pytorch/finetune_dalle.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ae941c6130274cb"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"j-min/dalleval","repo_kind":"official","path":"models/dalle_small/DALLE-pytorch/finetune_dalle.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/models/dalle_small/DALLE-pytorch/finetune_dalle.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"eb21740d88a30ecb","entry":"get_activations","repo":"j-min/dalleval","repo_kind":"official","path":"quality/FID/fid_score.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/quality/FID/fid_score.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb21740d88a30ecb"}},{"code_sha256_prefix":"863c2d358b6285ad","entry":"get_trainable_params","repo":"j-min/dalleval","repo_kind":"official","path":"models/dalle_small/DALLE-pytorch/finetune_dalle.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/models/dalle_small/DALLE-pytorch/finetune_dalle.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"863c2d358b6285ad"}},{"code_sha256_prefix":"0365a0718555a518","entry":"run","repo":"j-min/dalleval","repo_kind":"official","path":"models/dalle_small/DALLE-pytorch/collect_env.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/models/dalle_small/DALLE-pytorch/collect_env.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0365a0718555a518"}},{"code_sha256_prefix":"35f46cc480ac8cea","entry":"np_mad","repo":"j-min/dalleval","repo_kind":"official","path":"biases/compute_mad.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/biases/compute_mad.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35f46cc480ac8cea"}},{"code_sha256_prefix":"ab7aeddfa9e9fb2e","entry":"run_and_parse_first_match","repo":"j-min/dalleval","repo_kind":"official","path":"models/dalle_small/DALLE-pytorch/collect_env.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/models/dalle_small/DALLE-pytorch/collect_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab7aeddfa9e9fb2e"}},{"code_sha256_prefix":"84905132fbc263e3","entry":"run_and_read_all","repo":"j-min/dalleval","repo_kind":"official","path":"models/dalle_small/DALLE-pytorch/collect_env.py","file_url":"https://github.com/j-min/dalleval/blob/HEAD/models/dalle_small/DALLE-pytorch/collect_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84905132fbc263e3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}