{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/joint-visual-grounding-and-tracking-with","title":"Joint Visual Grounding and Tracking with Natural Language Specification","arxiv_id":"2303.12027","date":"2023-03-21","proceeding":"CVPR 2023 1","authors":["Li Zhou","Zikun Zhou","Kaige Mao","Zhenyu He"],"abstract":"Tracking by natural language specification aims to locate the referred target in a sequence based on the natural language description. Existing algorithms solve this issue in two steps, visual grounding and tracking, and accordingly deploy the separated grounding model and tracking model to implement these two steps, respectively. Such a separated framework overlooks the link between visual grounding and tracking, which is that the natural language descriptions provide global semantic cues for localizing the target for both two steps. Besides, the separated framework can hardly be trained end-to-end. To handle these issues, we propose a joint visual grounding and tracking framework, which reformulates grounding and tracking as a unified task: localizing the referred target based on the given visual-language references. Specifically, we propose a multi-source relation modeling module to effectively build the relation between the visual-language references and the test image. In addition, we design a temporal modeling module to provide a temporal clue with the guidance of the global semantic information for our model, which effectively improves the adaptability to the appearance variations of the target. Extensive experimental results on TNL2K, LaSOT, OTB99, and RefCOCOg demonstrate that our method performs favorably against state-of-the-art algorithms for both tracking and grounding. Code is available at https://github.com/lizhou-cs/JointNLT.","url_abs":"https://arxiv.org/abs/2303.12027v1","url_pdf":"https://arxiv.org/pdf/2303.12027v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"joint-visual-grounding-and-tracking-with","repo_url":"https://github.com/lizhou-cs/jointnlt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"visual-tracking","task_name":"Visual Tracking"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-tracking-on-tnl2k","task":"Visual Tracking","dataset":"TNL2K","model":"JointNLT","rank_in_archive_order":3,"of":6,"metrics":{"AUC":"56.9","precision":"58.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.12027","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.12027"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lizhou-cs/jointnlt","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/lizhou-cs/JointNLT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d2de70e6e6e2beee","entry":"NestedTensor","repo":"lizhou-cs/JointNLT","repo_kind":"official","path":"lib/models/JointNLT.py","file_url":"https://github.com/lizhou-cs/JointNLT/blob/HEAD/lib/models/JointNLT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2de70e6e6e2beee"}},{"code_sha256_prefix":"b6f03f9f1a7951fa","entry":"JointNLT","repo":"lizhou-cs/JointNLT","repo_kind":"official","path":"lib/models/JointNLT.py","file_url":"https://github.com/lizhou-cs/JointNLT/blob/HEAD/lib/models/JointNLT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6f03f9f1a7951fa"}},{"code_sha256_prefix":"e8b8ffeb4d46dbf4","entry":"_init_weights_trunc_normal","repo":"lizhou-cs/JointNLT","repo_kind":"official","path":"lib/models/JointNLT.py","file_url":"https://github.com/lizhou-cs/JointNLT/blob/HEAD/lib/models/JointNLT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8b8ffeb4d46dbf4"}},{"code_sha256_prefix":"40deb43ff3f75b19","entry":"_init_weights_xavier","repo":"lizhou-cs/JointNLT","repo_kind":"official","path":"lib/models/JointNLT.py","file_url":"https://github.com/lizhou-cs/JointNLT/blob/HEAD/lib/models/JointNLT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40deb43ff3f75b19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}