{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cp-detr-concept-prompt-guide-detr-toward","title":"CP-DETR: Concept Prompt Guide DETR Toward Stronger Universal Object Detection","arxiv_id":"2412.09799","date":"2024-12-13","proceeding":null,"authors":["Qibo Chen","Weizhong Jin","Jianyue Ge","Mengdi Liu","Yuchao Yan","Jian Jiang","Li Yu","Xuanjiang Guo","Shuchang Li","Jianzhong Chen"],"abstract":"Recent research on universal object detection aims to introduce language in a SoTA closed-set detector and then generalize the open-set concepts by constructing large-scale (text-region) datasets for training. However, these methods face two main challenges: (i) how to efficiently use the prior information in the prompts to genericise objects and (ii) how to reduce alignment bias in the downstream tasks, both leading to sub-optimal performance in some scenarios beyond pre-training. To address these challenges, we propose a strong universal detection foundation model called CP-DETR, which is competitive in almost all scenarios, with only one pre-training weight. Specifically, we design an efficient prompt visual hybrid encoder that enhances the information interaction between prompt and visual through scale-by-scale and multi-scale fusion modules. Then, the hybrid encoder is facilitated to fully utilize the prompted information by prompt multi-label loss and auxiliary detection head. In addition to text prompts, we have designed two practical concept prompt generation methods, visual prompt and optimized prompt, to extract abstract concepts through concrete visual examples and stably reduce alignment bias in downstream tasks. With these effective designs, CP-DETR demonstrates superior universal detection performance in a broad spectrum of scenarios. For example, our Swin-T backbone model achieves 47.6 zero-shot AP on LVIS, and the Swin-L backbone model achieves 32.2 zero-shot AP on ODinW35. Furthermore, our visual prompt generation method achieves 68.4 AP on COCO val by interactive detection, and the optimized prompt achieves 73.1 fully-shot AP on ODinW13.","url_abs":"https://arxiv.org/abs/2412.09799v1","url_pdf":"https://arxiv.org/pdf/2412.09799v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"zero-shot-object-detection","task_name":"Zero-Shot Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"CP-DETR-L Swin-L(Fine tuning separately in COCO)","rank_in_archive_order":11,"of":220,"metrics":{"box AP":"64.1"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-on-lvis-v1-0-minival","task":"Object Detection","dataset":"LVIS v1.0 minival","model":"CP-DETR-L Swin-L(with chunk)","rank_in_archive_order":2,"of":6,"metrics":{"box AP":"69.2"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-on-odinw-full-shot-13-tasks","task":"Object Detection","dataset":"ODinW Full-Shot 13 Tasks","model":"CP-DETR-L(only optimize prompt)","rank_in_archive_order":1,"of":8,"metrics":{"AP":"73.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-object-detection-on-lvis-v1-0","task":"Zero-Shot Object Detection","dataset":"LVIS v1.0 minival","model":"CP-DETR-Pro(without LVIS data)","rank_in_archive_order":1,"of":11,"metrics":{"AP":"58.2"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-object-detection-on-lvis-v1-0-val","task":"Zero-Shot Object Detection","dataset":"LVIS v1.0 val","model":"CP-DETR-Pro(without LVIS data)","rank_in_archive_order":1,"of":9,"metrics":{"AP":"51.6"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-object-detection-on-mscoco","task":"Zero-Shot Object Detection","dataset":"MSCOCO","model":"CP-DETR-Pro((without COCO data))","rank_in_archive_order":2,"of":7,"metrics":{"AP":"55.4"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-object-detection-on-odinw","task":"Zero-Shot Object Detection","dataset":"ODinW","model":"CP-DETR-L Swin-L","rank_in_archive_order":1,"of":5,"metrics":{"Average Score":"32.2"},"uses_additional_data":true}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}