{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transvg-end-to-end-visual-grounding-with","title":"TransVG: End-to-End Visual Grounding with Transformers","arxiv_id":"2104.08541","date":"2021-04-17","proceeding":"ICCV 2021 10","authors":["Jiajun Deng","Zhengyuan Yang","Tianlang Chen","Wengang Zhou","Houqiang Li"],"abstract":"In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image. The state-of-the-art methods, including two-stage or one-stage ones, rely on a complex module with manually-designed mechanisms to perform the query reasoning and multi-modal fusion. However, the involvement of certain mechanisms in fusion module design, such as query decomposition and image scene graph, makes the models easily overfit to datasets with specific scenarios, and limits the plenitudinous interaction between the visual-linguistic context. To avoid this caveat, we propose to establish the multi-modal correspondence by leveraging transformers, and empirically show that the complex fusion modules e.g., modular attention network, dynamic graph, and multi-modal tree) can be replaced by a simple stack of transformer encoder layers with higher performance. Moreover, we re-formulate the visual grounding as a direct coordinates regression problem and avoid making predictions out of a set of candidates i.e., region proposals or anchor boxes). Extensive experiments are conducted on five widely used datasets, and a series of state-of-the-art records are set by our TransVG. We build the benchmark of transformer-based visual grounding framework and make the code available at \\url{https://github.com/djiajunustc/TransVG}.","url_abs":"https://arxiv.org/abs/2104.08541v4","url_pdf":"https://arxiv.org/pdf/2104.08541v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transvg-end-to-end-visual-grounding-with","repo_url":"https://github.com/djiajunustc/TransVG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"transvg-end-to-end-visual-grounding-with","repo_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.08541","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08541"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/djiajunustc/TransVG","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nku-shengzheliu/Pytorch-TransVG","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":3,"unverified":14},"by_repo_kind":{"listed":{"samples":20,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f1ea5386ce472284","entry":"GIoU_Loss","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"models/loss.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1ea5386ce472284"}},{"code_sha256_prefix":"33a4a4477537ad1c","entry":"Reg_Loss","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"models/loss.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33a4a4477537ad1c"}},{"code_sha256_prefix":"aa6da43d05cfaf65","entry":"all_gather","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/misc.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa6da43d05cfaf65"}},{"code_sha256_prefix":"3f82133185c8bcb4","entry":"collate_fn","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/misc.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f82133185c8bcb4"}},{"code_sha256_prefix":"05992505df9ef0cd","entry":"reduce_dict","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/misc.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05992505df9ef0cd"}},{"code_sha256_prefix":"3d629255e4cf4ea2","entry":"xyxy2xywh","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/utils.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d629255e4cf4ea2"}},{"code_sha256_prefix":"ec48fe4bebce9e11","entry":"bbox_iou_numpy","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/utils.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec48fe4bebce9e11"}},{"code_sha256_prefix":"75b3520299159feb","entry":"build_VLFusion","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"models/detr.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/models/detr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75b3520299159feb"}},{"code_sha256_prefix":"87992961bac26fc7","entry":"build_transformer","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"models/transformer.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/models/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87992961bac26fc7"}},{"code_sha256_prefix":"5ed44f7213d4ad07","entry":"build_vis_transformer","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"models/transformer.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/models/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ed44f7213d4ad07"}},{"code_sha256_prefix":"694e413f977737ba","entry":"convert_examples_to_features","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"dataset/data_loader.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/dataset/data_loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"694e413f977737ba"}},{"code_sha256_prefix":"0336cb59d505cda3","entry":"generate_coord","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"models/model.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/models/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0336cb59d505cda3"}},{"code_sha256_prefix":"b202ce3cdac3a1ce","entry":"letterbox","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/transforms.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b202ce3cdac3a1ce"}},{"code_sha256_prefix":"8dc1056f0f6f5443","entry":"load_pretrain","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/checkpoint.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/checkpoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dc1056f0f6f5443"}},{"code_sha256_prefix":"44ac945d7427e3cc","entry":"load_resume","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/checkpoint.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/checkpoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44ac945d7427e3cc"}},{"code_sha256_prefix":"395306b13bd68afc","entry":"load_weights","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"models/model.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/models/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"395306b13bd68afc"}},{"code_sha256_prefix":"408bfee1417b817e","entry":"random_affine","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/transforms.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"408bfee1417b817e"}},{"code_sha256_prefix":"e4f50d4eeb6144e9","entry":"read_examples","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"dataset/data_loader.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/dataset/data_loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4f50d4eeb6144e9"}},{"code_sha256_prefix":"cb5d2317201b6913","entry":"wrap_points","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/transforms.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb5d2317201b6913"}},{"code_sha256_prefix":"7950108b58fb67bc","entry":"xywh2xyxy","repo":"nku-shengzheliu/Pytorch-TransVG","repo_kind":"listed","path":"utils/utils.py","file_url":"https://github.com/nku-shengzheliu/Pytorch-TransVG/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7950108b58fb67bc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}