{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/desco-learning-object-recognition-with-rich","title":"DesCo: Learning Object Recognition with Rich Language Descriptions","arxiv_id":"2306.14060","date":"2023-06-24","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Recent development in vision-language approaches has instigated a paradigm shift in learning visual recognition models from language supervision. These approaches align objects with language queries (e.g. \"a photo of a cat\") and improve the models' adaptability to identify novel objects and domains. Recently, several studies have attempted to query these models with complex language expressions that include specifications of fine-grained semantic details, such as attributes, shapes, textures, and relations. However, simply incorporating language descriptions as queries does not guarantee accurate interpretation by the models. In fact, our experiments show that GLIP, the state-of-the-art vision-language model for object detection, often disregards contextual information in the language descriptions and instead relies heavily on detecting objects solely by their names. To tackle the challenges, we propose a new description-conditioned (DesCo) paradigm of learning object recognition models with rich language descriptions consisting of two major innovations: 1) we employ a large language model as a commonsense knowledge engine to generate rich language descriptions of objects based on object names and the raw image-text caption; 2) we design context-sensitive queries to improve the model's ability in deciphering intricate nuances embedded within descriptions and enforce the model to focus on context rather than object names alone. On two novel object detection benchmarks, LVIS and OminiLabel, under the zero-shot detection setting, our approach achieves 34.8 APr minival (+9.1) and 29.3 AP (+3.6), respectively, surpassing the prior state-of-the-art models, GLIP and FIBER, by a large margin.","url_abs":"https://arxiv.org/abs/2306.14060v1","url_pdf":"https://arxiv.org/pdf/2306.14060v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"desco-learning-object-recognition-with-rich","repo_url":"https://github.com/liunian-harold-li/desco","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"novel-object-detection","task_name":"Novel Object Detection"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.14060","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.14060"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liunian-harold-li/desco","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"3857170c2d5eb61d","entry":"add_conv","repo":"liunian-harold-li/desco","repo_kind":"listed","path":"maskrcnn_benchmark/layers/deform_pool.py","file_url":"https://github.com/liunian-harold-li/desco/blob/HEAD/maskrcnn_benchmark/layers/deform_pool.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"3857170c2d5eb61d"}},{"code_sha256_prefix":"33363ef0803b607a","entry":"add_sepconv","repo":"liunian-harold-li/desco","repo_kind":"listed","path":"maskrcnn_benchmark/layers/deform_pool.py","file_url":"https://github.com/liunian-harold-li/desco/blob/HEAD/maskrcnn_benchmark/layers/deform_pool.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"33363ef0803b607a"}},{"code_sha256_prefix":"3592f11df8bd6c1d","entry":"make_divisible","repo":"liunian-harold-li/desco","repo_kind":"listed","path":"maskrcnn_benchmark/layers/deform_pool.py","file_url":"https://github.com/liunian-harold-li/desco/blob/HEAD/maskrcnn_benchmark/layers/deform_pool.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"3592f11df8bd6c1d"}},{"code_sha256_prefix":"2902bf4410253ff7","entry":"interpolate","repo":"liunian-harold-li/desco","repo_kind":"listed","path":"maskrcnn_benchmark/layers/misc.py","file_url":"https://github.com/liunian-harold-li/desco/blob/HEAD/maskrcnn_benchmark/layers/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"2902bf4410253ff7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}