{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generative-region-language-pretraining-for","title":"Generative Region-Language Pretraining for Open-Ended Object Detection","arxiv_id":"2403.10191","date":"2024-03-15","proceeding":"CVPR 2024 1","authors":["Chuang Lin","Yi Jiang","Lizhen Qu","Zehuan Yuan","Jianfei Cai"],"abstract":"In recent research, significant attention has been devoted to the open-vocabulary object detection task, aiming to generalize beyond the limited number of classes labeled during training and detect objects described by arbitrary category names at inference. Compared with conventional object detection, open vocabulary object detection largely extends the object detection categories. However, it relies on calculating the similarity between image regions and a set of arbitrary category names with a pretrained vision-and-language model. This implies that, despite its open-set nature, the task still needs the predefined object categories during the inference stage. This raises the question: What if we do not have exact knowledge of object categories during inference? In this paper, we call such a new setting as generative open-ended object detection, which is a more general and practical problem. To address it, we formulate object detection as a generative problem and propose a simple framework named GenerateU, which can detect dense objects and generate their names in a free-form way. Particularly, we employ Deformable DETR as a region proposal generator with a language model translating visual regions to object names. To assess the free-form object detection task, we introduce an evaluation method designed to quantitatively measure the performance of generative outcomes. Extensive experiments demonstrate strong zero-shot detection performance of our GenerateU. For example, on the LVIS dataset, our GenerateU achieves comparable results to the open-vocabulary object detection method GLIP, even though the category names are not seen by GenerateU during inference. Code is available at: https:// github.com/FoundationVision/GenerateU .","url_abs":"https://arxiv.org/abs/2403.10191v1","url_pdf":"https://arxiv.org/pdf/2403.10191v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generative-region-language-pretraining-for","repo_url":"https://github.com/foundationvision/generateu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open-vocabulary object detection"},{"task_slug":"region-proposal","task_name":"Region Proposal"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"deformable-attention-module","method_name":"Deformable Attention Module"},{"method_slug":"deformable-detr","method_name":"Deformable DETR"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"detr","method_name":"Detr"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.10191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.10191"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/foundationvision/generateu","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/FoundationVision/GenerateU","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_fixture":2,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d7527a665dfb8f27","entry":"batched_nms","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/nms.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/nms.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7527a665dfb8f27"}},{"code_sha256_prefix":"cee8870d4fbccc39","entry":"ciou_loss","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/losses.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/losses.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cee8870d4fbccc39"}},{"code_sha256_prefix":"e8526a516b4f9206","entry":"convert_basic_c2_names","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/checkpoint/c2_model_loading.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/checkpoint/c2_model_loading.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8526a516b4f9206"}},{"code_sha256_prefix":"e36f87059208bb7e","entry":"convert_c2_detectron_names","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/checkpoint/c2_model_loading.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/checkpoint/c2_model_loading.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e36f87059208bb7e"}},{"code_sha256_prefix":"26f3066ead0d2eec","entry":"diou_loss","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/losses.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/losses.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26f3066ead0d2eec"}},{"code_sha256_prefix":"406f84cf799ed34d","entry":"pad_masks","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/mask_ops.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/mask_ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"406f84cf799ed34d"}},{"code_sha256_prefix":"d03592674030077f","entry":"paste_mask_in_image_old","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/mask_ops.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/mask_ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d03592674030077f"}},{"code_sha256_prefix":"c0eac7dcd1335fa1","entry":"paste_masks_in_image","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/mask_ops.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/mask_ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0eac7dcd1335fa1"}},{"code_sha256_prefix":"ac2638463572c152","entry":"align_and_update_state_dicts","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/checkpoint/c2_model_loading.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/checkpoint/c2_model_loading.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac2638463572c152"}},{"code_sha256_prefix":"627f9dec92f2462b","entry":"batched_nms_rotated","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/nms.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/nms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"627f9dec92f2462b"}},{"code_sha256_prefix":"d15c7cc7a28e4a67","entry":"nms_rotated","repo":"FoundationVision/GenerateU","repo_kind":"official","path":"detectron2/layers/nms.py","file_url":"https://github.com/FoundationVision/GenerateU/blob/HEAD/detectron2/layers/nms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d15c7cc7a28e4a67"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}