{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/detgpt-detect-what-you-need-via-reasoning","title":"DetGPT: Detect What You Need via Reasoning","arxiv_id":"2305.14167","date":"2023-05-23","proceeding":null,"authors":["Renjie Pi","Jiahui Gao","Shizhe Diao","Rui Pan","Hanze Dong","Jipeng Zhang","Lewei Yao","Jianhua Han","Hang Xu","Lingpeng Kong","Tong Zhang"],"abstract":"In recent years, the field of computer vision has seen significant advancements thanks to the development of large language models (LLMs). These models have enabled more effective and sophisticated interactions between humans and machines, paving the way for novel techniques that blur the lines between human and machine intelligence. In this paper, we introduce a new paradigm for object detection that we call reasoning-based object detection. Unlike conventional object detection methods that rely on specific object names, our approach enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity. Our proposed method, called DetGPT, leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user's instructions and the visual scene. This enables DetGPT to automatically locate the object of interest based on the user's expressed desires, even if the object is not explicitly mentioned. For instance, if a user expresses a desire for a cold beverage, DetGPT can analyze the image, identify a fridge, and use its knowledge of typical fridge contents to locate the beverage. This flexibility makes our system applicable across a wide range of fields, from robotics and automation to autonomous driving. Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines. We hope that our proposed paradigm and approach will provide inspiration to the community and open the door to more interative and versatile object detection systems. Our project page is launched at detgpt.github.io.","url_abs":"https://arxiv.org/abs/2305.14167v2","url_pdf":"https://arxiv.org/pdf/2305.14167v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"detgpt-detect-what-you-need-via-reasoning","repo_url":"https://github.com/optimalscale/detgpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2305.14167","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.14167"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/optimalscale/detgpt","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_fixture":1,"ran_violates":1,"ran":3,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"listed":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b4dff79d5e6102c","entry":"apply_rotary_pos_emb","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/models/modeling_llama.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9b4dff79d5e6102c"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/models/blip2.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/models/blip2.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"98589643273920ba","entry":"main_process","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/common/dist_utils.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/common/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"98589643273920ba"}},{"code_sha256_prefix":"791c72070c0b1cfb","entry":"node_to_dict","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/common/config.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/common/config.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"791c72070c0b1cfb"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/models/modeling_llama.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/models/base_model.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/models/base_model.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"13bb35cf371e2548","entry":"apply_rotary_pos_emb_hf","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/models/modeling_llama.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"13bb35cf371e2548"}},{"code_sha256_prefix":"643539116e049080","entry":"download_cached_file","repo":"optimalscale/detgpt","repo_kind":"listed","path":"detgpt/common/dist_utils.py","file_url":"https://github.com/optimalscale/detgpt/blob/HEAD/detgpt/common/dist_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"643539116e049080"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}