{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crossing-the-format-boundary-of-text-and","title":"UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling","arxiv_id":"2111.12085","date":"2021-11-23","proceeding":null,"authors":["Zhengyuan Yang","Zhe Gan","JianFeng Wang","Xiaowei Hu","Faisal Ahmed","Zicheng Liu","Yumao Lu","Lijuan Wang"],"abstract":"We propose UniTAB that Unifies Text And Box outputs for grounded vision-language (VL) modeling. Grounded VL tasks such as grounded captioning require the model to generate a text description and align predicted words with object regions. To achieve this, models must generate desired text and box outputs together, and meanwhile indicate the alignments between words and boxes. In contrast to existing solutions that use multiple separate modules for different outputs, UniTAB represents both text and box outputs with a shared token sequence, and introduces a special <obj> token to naturally indicate word-box alignments in the sequence. UniTAB thus could provide a more comprehensive and interpretable image description, by freely grounding generated words to object regions. On grounded captioning, UniTAB presents a simpler solution with a single output head, and significantly outperforms state of the art in both grounding and captioning evaluations. On general VL tasks that have different desired output formats (i.e., text, box, or their combination), UniTAB with a single network achieves better or comparable performance than task-specific state of the art. Experiments cover 7 VL benchmarks, including grounded captioning, visual grounding, image captioning, and visual question answering. Furthermore, UniTAB's unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks.","url_abs":"https://arxiv.org/abs/2111.12085v2","url_pdf":"https://arxiv.org/pdf/2111.12085v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crossing-the-format-boundary-of-text-and","repo_url":"https://github.com/microsoft/UniTAB","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":null,"task_name":"Image Description"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object-localization","task_name":"Object Localization"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.12085","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.12085"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/UniTAB","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_draft_wrong":5,"unverified":9},"by_repo_kind":{"official":{"samples":15,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f31a58bf6457ced5","entry":"convert_to_xywh","repo":"microsoft/UniTAB","repo_kind":"official","path":"datasets/coco_eval.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/datasets/coco_eval.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f31a58bf6457ced5"}},{"code_sha256_prefix":"4c1b107103b7c856","entry":"cook_refs","repo":"microsoft/UniTAB","repo_kind":"official","path":"cococapeval/cider/cider_scorer.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/cococapeval/cider/cider_scorer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c1b107103b7c856"}},{"code_sha256_prefix":"3573c4bc85f65dfd","entry":"cook_test","repo":"microsoft/UniTAB","repo_kind":"official","path":"cococapeval/cider/cider_scorer.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/cococapeval/cider/cider_scorer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3573c4bc85f65dfd"}},{"code_sha256_prefix":"fe0ddcc2d420c9a0","entry":"evaluate","repo":"microsoft/UniTAB","repo_kind":"official","path":"datasets/coco_eval.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/datasets/coco_eval.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe0ddcc2d420c9a0"}},{"code_sha256_prefix":"ec4448373d72d636","entry":"precook","repo":"microsoft/UniTAB","repo_kind":"official","path":"cococapeval/bleu/bleu_scorer.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/cococapeval/bleu/bleu_scorer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec4448373d72d636"}},{"code_sha256_prefix":"9560e3e3e41bc1b2","entry":"precook","repo":"microsoft/UniTAB","repo_kind":"official","path":"cococapeval/cider/cider_scorer.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/cococapeval/cider/cider_scorer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9560e3e3e41bc1b2"}},{"code_sha256_prefix":"cea3f92defccfb9c","entry":"build_position_encoding","repo":"microsoft/UniTAB","repo_kind":"official","path":"models/position_encoding.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/models/position_encoding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cea3f92defccfb9c"}},{"code_sha256_prefix":"0b02672adf92e20a","entry":"build_transformer","repo":"microsoft/UniTAB","repo_kind":"official","path":"models/transformer_unitab.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/models/transformer_unitab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b02672adf92e20a"}},{"code_sha256_prefix":"7586e1d32f7325f7","entry":"cook_refs","repo":"microsoft/UniTAB","repo_kind":"official","path":"cococapeval/bleu/bleu_scorer.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/cococapeval/bleu/bleu_scorer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7586e1d32f7325f7"}},{"code_sha256_prefix":"99e54843980299e2","entry":"cook_test","repo":"microsoft/UniTAB","repo_kind":"official","path":"cococapeval/bleu/bleu_scorer.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/cococapeval/bleu/bleu_scorer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99e54843980299e2"}},{"code_sha256_prefix":"5c1b500d83c0cf64","entry":"create_positive_map","repo":"microsoft/UniTAB","repo_kind":"official","path":"datasets/coco.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/datasets/coco.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c1b500d83c0cf64"}},{"code_sha256_prefix":"a3646da4ef0cdb56","entry":"seq2logits","repo":"microsoft/UniTAB","repo_kind":"official","path":"models/postprocessors.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/models/postprocessors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3646da4ef0cdb56"}},{"code_sha256_prefix":"e4584fdac028654e","entry":"target2gtind","repo":"microsoft/UniTAB","repo_kind":"official","path":"models/unitab.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/models/unitab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4584fdac028654e"}},{"code_sha256_prefix":"11cd2a3ee603af97","entry":"target2prevind","repo":"microsoft/UniTAB","repo_kind":"official","path":"models/unitab.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/models/unitab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11cd2a3ee603af97"}},{"code_sha256_prefix":"97aaacff6f0b7fee","entry":"withbbox_subseq","repo":"microsoft/UniTAB","repo_kind":"official","path":"models/unitab.py","file_url":"https://github.com/microsoft/UniTAB/blob/HEAD/models/unitab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97aaacff6f0b7fee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}