{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generate-any-scene-evaluating-and-improving","title":"Generate Any Scene: Evaluating and Improving Text-to-Vision Generation with Scene Graph Programming","arxiv_id":"2412.08221","date":"2024-12-11","proceeding":null,"authors":["Ziqi Gao","Weikai Huang","Jieyu Zhang","Aniruddha Kembhavi","Ranjay Krishna"],"abstract":"DALL-E and Sora have gained attention by producing implausible images, such as \"astronauts riding a horse in space.\" Despite the proliferation of text-to-vision models that have inundated the internet with synthetic visuals, from images to 3D assets, current benchmarks predominantly evaluate these models on real-world scenes paired with captions. We introduce Generate Any Scene, a framework that systematically enumerates scene graphs representing a vast array of visual scenes, spanning realistic to imaginative compositions. Generate Any Scene leverages 'scene graph programming', a method for dynamically constructing scene graphs of varying complexity from a structured taxonomy of visual elements. This taxonomy includes numerous objects, attributes, and relations, enabling the synthesis of an almost infinite variety of scene graphs. Using these structured representations, Generate Any Scene translates each scene graph into a caption, enabling scalable evaluation of text-to-vision models through standard metrics. We conduct extensive evaluations across multiple text-to-image, text-to-video, and text-to-3D models, presenting key findings on model performance. We find that DiT-backbone text-to-image models align more closely with input captions than UNet-backbone models. Text-to-video models struggle with balancing dynamics and consistency, while both text-to-video and text-to-3D models show notable gaps in human preference alignment. We demonstrate the effectiveness of Generate Any Scene by conducting three practical applications leveraging captions generated by Generate Any Scene: 1) a self-improving framework where models iteratively enhance their performance using generated data, 2) a distillation process to transfer specific strengths from proprietary models to open-source counterparts, and 3) improvements in content moderation by identifying and generating challenging synthetic data.","url_abs":"https://arxiv.org/abs/2412.08221v2","url_pdf":"https://arxiv.org/pdf/2412.08221v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generate-any-scene-evaluating-and-improving","repo_url":"https://github.com/RAIVNLab/GenerateAnyScene","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"text-to-3d","task_name":"Text to 3D"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.08221","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.08221"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RAIVNLab/GenerateAnyScene","reach":null}],"summary":{"ran":1,"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a7e2d6002ce5485b","entry":"StableDiffusion2","repo":"RAIVNLab/GenerateAnyScene","repo_kind":"official","path":"gas/models/gen_model/text2image_model.py","file_url":"https://github.com/RAIVNLab/GenerateAnyScene/blob/HEAD/gas/models/gen_model/text2image_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7e2d6002ce5485b"}},{"code_sha256_prefix":"9970c1cdbd2b5873","entry":"add_seed_graph_to_template_graph","repo":"raivnlab/generateanyscene","repo_kind":"official","path":"gas/captions_generation/scene_graph.py","file_url":"https://github.com/raivnlab/generateanyscene/blob/HEAD/gas/captions_generation/scene_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9970c1cdbd2b5873"}},{"code_sha256_prefix":"1cd04068b5ed4f68","entry":"find_isomorphisms","repo":"raivnlab/generateanyscene","repo_kind":"official","path":"gas/captions_generation/scene_graph.py","file_url":"https://github.com/raivnlab/generateanyscene/blob/HEAD/gas/captions_generation/scene_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1cd04068b5ed4f68"}},{"code_sha256_prefix":"5022dcff291c3815","entry":"generate_prompt","repo":"RAIVNLab/GenerateAnyScene","repo_kind":"official","path":"generation.py","file_url":"https://github.com/RAIVNLab/GenerateAnyScene/blob/HEAD/generation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5022dcff291c3815"}},{"code_sha256_prefix":"3591898e154d97a1","entry":"label_repeated_objects_in_sg","repo":"raivnlab/generateanyscene","repo_kind":"official","path":"gas/captions_generation/scene_graph.py","file_url":"https://github.com/raivnlab/generateanyscene/blob/HEAD/gas/captions_generation/scene_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3591898e154d97a1"}},{"code_sha256_prefix":"f987c876bc076858","entry":"GenModelInstance","repo":"RAIVNLab/GenerateAnyScene","repo_kind":"official","path":"gas/models/gen_model/text2image_model.py","file_url":"https://github.com/RAIVNLab/GenerateAnyScene/blob/HEAD/gas/models/gen_model/text2image_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f987c876bc076858"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}