{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-generation-harnessing-text-to-image","title":"Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation","arxiv_id":"2309.05956","date":"2023-09-12","proceeding":null,"authors":["Yunhao Ge","Jiashu Xu","Brian Nlong Zhao","Neel Joshi","Laurent Itti","Vibhav Vineet"],"abstract":"We propose a new paradigm to automatically generate training data with accurate labels at scale using the text-to-image synthesis frameworks (e.g., DALL-E, Stable Diffusion, etc.). The proposed approach1 decouples training data generation into foreground object generation, and contextually coherent background generation. To generate foreground objects, we employ a straightforward textual template, incorporating the object class name as input prompts. This is fed into a text-to-image synthesis framework, producing various foreground images set against isolated backgrounds. A foreground-background segmentation algorithm is then used to generate foreground object masks. To generate context images, we begin by creating language descriptions of the context. This is achieved by applying an image captioning method to a small set of images representing the desired context. These textual descriptions are then transformed into a diverse array of context images via a text-to-image synthesis framework. Subsequently, we composite these with the foreground object masks produced in the initial step, utilizing a cut-and-paste method, to formulate the training data. We demonstrate the advantages of our approach on five object detection and segmentation datasets, including Pascal VOC and COCO. We found that detectors trained solely on synthetic data produced by our method achieve performance comparable to those trained on real data (Fig. 1). Moreover, a combination of real and synthetic data yields even much better results. Further analysis indicates that the synthetic data distribution complements the real data distribution effectively. Additionally, we emphasize the compositional nature of our data generation approach in out-of-distribution and zero-shot data generation scenarios. We open-source our code at https://github.com/gyhandy/Text2Image-for-Detection","url_abs":"https://arxiv.org/abs/2309.05956v1","url_pdf":"https://arxiv.org/pdf/2309.05956v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-generation-harnessing-text-to-image","repo_url":"https://github.com/gyhandy/text2image-for-detection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.05956","atlas_url":"https://app.syntology.ai/?focus=2309.05956","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.05956"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gyhandy/text2image-for-detection","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"dec93047c9706811","entry":"binarize_mask","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/background.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/background.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dec93047c9706811"}},{"code_sha256_prefix":"47caad7b315514da","entry":"create_mask","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/pb.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/pb.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47caad7b315514da"}},{"code_sha256_prefix":"338cddb61947c898","entry":"get_area","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/foreground.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/foreground.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"338cddb61947c898"}},{"code_sha256_prefix":"f4d263d621b7c43a","entry":"get_box","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/foreground.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/foreground.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f4d263d621b7c43a"}},{"code_sha256_prefix":"45759338adac308c","entry":"get_gradient_sum","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/pb.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/pb.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45759338adac308c"}},{"code_sha256_prefix":"ec845b137200d97a","entry":"get_mixed_gradient_sum","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/pb.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/pb.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec845b137200d97a"}},{"code_sha256_prefix":"43bfb3c107e16439","entry":"sort_images","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/clip_postprocess.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/clip_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43bfb3c107e16439"}},{"code_sha256_prefix":"605bfe5472a780c1","entry":"get_CLIP_score","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/clip_postprocess.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/clip_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"605bfe5472a780c1"}},{"code_sha256_prefix":"b16d4b8d3120eee3","entry":"scores_for_one_caption","repo":"gyhandy/text2image-for-detection","repo_kind":"official","path":"cutpaste/clip_postprocess.py","file_url":"https://github.com/gyhandy/text2image-for-detection/blob/HEAD/cutpaste/clip_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b16d4b8d3120eee3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}