{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rtgen-generating-region-text-pairs-for-open","title":"RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detection","arxiv_id":"2405.19854","date":"2024-05-30","proceeding":null,"authors":["Fangyi Chen","Han Zhang","Zhantao Yang","Hao Chen","Kai Hu","Marios Savvides"],"abstract":"Open-vocabulary object detection (OVD) requires solid modeling of the region-semantic relationship, which could be learned from massive region-text pairs. However, such data is limited in practice due to significant annotation costs. In this work, we propose RTGen to generate scalable open-vocabulary region-text pairs and demonstrate its capability to boost the performance of open-vocabulary object detection. RTGen includes both text-to-region and region-to-text generation processes on scalable image-caption data. The text-to-region generation is powered by image inpainting, directed by our proposed scene-aware inpainting guider for overall layout harmony. For region-to-text generation, we perform multiple region-level image captioning with various prompts and select the best matching text according to CLIP similarity. To facilitate detection training on region-text pairs, we also introduce a localization-aware region-text contrastive loss that learns object proposals tailored with different localization qualities. Extensive experiments demonstrate that our RTGen can serve as a scalable, semantically rich, and effective source for open-vocabulary object detection and continue to improve the model performance when more data is utilized, delivering superior performance compared to the existing state-of-the-art methods.","url_abs":"https://arxiv.org/abs/2405.19854v1","url_pdf":"https://arxiv.org/pdf/2405.19854v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rtgen-generating-region-text-pairs-for-open","repo_url":"https://github.com/seermer/RTGen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-inpainting","task_name":"Image Inpainting"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open-vocabulary object detection"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"pixel-prediction","method_name":"Inpainting"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-object-detection-on-lvis-v1-0","task":"Open Vocabulary Object Detection","dataset":"LVIS v1.0","model":"RTGen","rank_in_archive_order":12,"of":28,"metrics":{"AP novel-LVIS base training":"30.2"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.19854","atlas_url":"https://app.syntology.ai/?focus=2405.19854","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}