{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/grit-a-generative-region-to-text-transformer","title":"GRiT: A Generative Region-to-text Transformer for Object Understanding","arxiv_id":"2212.00280","date":"2022-12-01","proceeding":null,"authors":["Jialian Wu","JianFeng Wang","Zhengyuan Yang","Zhe Gan","Zicheng Liu","Junsong Yuan","Lijuan Wang"],"abstract":"This paper presents a Generative RegIon-to-Text transformer, GRiT, for object understanding. The spirit of GRiT is to formulate object understanding as <region, text> pairs, where region locates objects and text describes objects. For example, the text in object detection denotes class names while that in dense captioning refers to descriptive sentences. Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions. With the same model architecture, GRiT can understand objects via not only simple nouns, but also rich descriptive sentences including object attributes or actions. Experimentally, we apply GRiT to object detection and dense captioning tasks. GRiT achieves 60.4 AP on COCO 2017 test-dev for object detection and 15.5 mAP on Visual Genome for dense captioning. Code is available at https://github.com/JialianW/GRiT","url_abs":"https://arxiv.org/abs/2212.00280v1","url_pdf":"https://arxiv.org/pdf/2212.00280v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"grit-a-generative-region-to-text-transformer","repo_url":"https://github.com/JialianW/GRiT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"dense-captioning","task_name":"Dense Captioning"},{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/dense-captioning-on-visual-genome","task":"Dense Captioning","dataset":"Visual Genome","model":"GRiT (ViT-B)","rank_in_archive_order":2,"of":4,"metrics":{"mAP":"15.5"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"GRiT (ViT-H, single-scale testing)","rank_in_archive_order":29,"of":225,"metrics":{"box mAP":"60.4"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco-o","task":"Object Detection","dataset":"COCO-O","model":"GRiT\n(ViT-H)","rank_in_archive_order":4,"of":45,"metrics":{"Average mAP":"42.9","Effective Robustness":"15.72"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.00280","atlas_url":"https://app.syntology.ai/?focus=2212.00280","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.00280"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JialianW/GRiT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_honours":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"edb9869f131321d2","entry":"get_rel_pos","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/modeling/backbone/utils.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/modeling/backbone/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"edb9869f131321d2"}},{"code_sha256_prefix":"08bf7ecdb2fffe84","entry":"get_vit_lr_decay_rate","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/custom_solver.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/custom_solver.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"08bf7ecdb2fffe84"}},{"code_sha256_prefix":"070e8bf494b411b6","entry":"qk2attn","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/modeling/text/modeling_bert.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/modeling/text/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"070e8bf494b411b6"}},{"code_sha256_prefix":"5eff22fa0a651276","entry":"url_to_filename","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/modeling/text/file_utils.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/modeling/text/file_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5eff22fa0a651276"}},{"code_sha256_prefix":"750facb8c52e8864","entry":"window_partition","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/modeling/backbone/utils.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/modeling/backbone/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"750facb8c52e8864"}},{"code_sha256_prefix":"535942f820542fb9","entry":"window_unpartition","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/modeling/backbone/utils.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/modeling/backbone/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"535942f820542fb9"}},{"code_sha256_prefix":"4e91fb4390008086","entry":"cached_path","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/modeling/text/file_utils.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/modeling/text/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"4e91fb4390008086"}},{"code_sha256_prefix":"a6aab787fcf8b10d","entry":"filename_to_url","repo":"JialianW/GRiT","repo_kind":"official","path":"grit/modeling/text/file_utils.py","file_url":"https://github.com/JialianW/GRiT/blob/HEAD/grit/modeling/text/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a6aab787fcf8b10d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}