{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tracking-meets-lora-faster-training-larger","title":"Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance","arxiv_id":"2403.05231","date":"2024-03-08","proceeding":null,"authors":["Liting Lin","Heng Fan","Zhipeng Zhang","YaoWei Wang","Yong Xu","Haibin Ling"],"abstract":"Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language models, we propose LoRAT, a method that unveils the power of large ViT model for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. However, unique challenges and potential domain gaps make this transfer not as easy as the first intuition. Firstly, a transformer-based tracker constructs unshared position embedding for template and search image. This poses a challenge for the transfer of LoRA, usually requiring consistency in the design when applied to the pre-trained backbone, to downstream tasks. Secondly, the inductive bias inherent in convolutional heads diminishes the effectiveness of parameter-efficient fine-tuning in tracking models. To overcome these limitations, we first decouple the position embeddings in transformer-based trackers into shared spatial ones and independent type ones. The shared embeddings, which describe the absolute coordinates of multi-resolution images (namely, the template and search images), are inherited from the pre-trained backbones. In contrast, the independent embeddings indicate the sources of each token and are learned from scratch. Furthermore, we design an anchor-free head solely based on MLP to adapt PETR, enabling better performance with less computational overhead. With our design, 1) it becomes practical to train trackers with the ViT-g backbone on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve the LaSOT SUC score from 0.703 to 0.742 with the L-224 variant; 4) we fast the inference speed of the L-224 variant from 52 to 119 FPS. Code and models are available at https://github.com/LitingLin/LoRAT.","url_abs":"https://arxiv.org/abs/2403.05231v2","url_pdf":"https://arxiv.org/pdf/2403.05231v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tracking-meets-lora-faster-training-larger","repo_url":"https://github.com/litinglin/lorat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"visual-object-tracking","task_name":"Visual Object Tracking"},{"task_slug":"visual-tracking","task_name":"Visual Tracking"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-object-tracking-on-got-10k","task":"Visual Object Tracking","dataset":"GOT-10k","model":"LoRAT-g-378","rank_in_archive_order":8,"of":42,"metrics":{"Average Overlap":"78.9","Success Rate 0.5":"87.8","Success Rate 0.75":"80.7"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-got-10k","task":"Visual Object Tracking","dataset":"GOT-10k","model":"LoRAT-L-378","rank_in_archive_order":13,"of":42,"metrics":{"Average Overlap":"77.5","Success Rate 0.5":"86.2","Success Rate 0.75":"78.1"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-lasot","task":"Visual Object Tracking","dataset":"LaSOT","model":"LoRAT-g-378","rank_in_archive_order":4,"of":46,"metrics":{"AUC":"76.2","Normalized Precision":"85.3","Precision":"83.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-lasot","task":"Visual Object Tracking","dataset":"LaSOT","model":"LoRAT-L-378","rank_in_archive_order":6,"of":46,"metrics":{"AUC":"75.1","Normalized Precision":"84.1","Precision":"82.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-lasot-ext","task":"Visual Object Tracking","dataset":"LaSOT-ext","model":"LoRAT-L-378","rank_in_archive_order":3,"of":18,"metrics":{"AUC":"56.6","Normalized Precision":"69.0","Precision":"65.1"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-lasot-ext","task":"Visual Object Tracking","dataset":"LaSOT-ext","model":"LoRAT-g-378","rank_in_archive_order":4,"of":18,"metrics":{"AUC":"56.5","Normalized Precision":"69.0","Precision":"64.9"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-needforspeed","task":"Visual Object Tracking","dataset":"NeedForSpeed","model":"LoRAT-g-378","rank_in_archive_order":5,"of":10,"metrics":{"AUC":"0.681"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-needforspeed","task":"Visual Object Tracking","dataset":"NeedForSpeed","model":"LoRAT-L-378","rank_in_archive_order":8,"of":10,"metrics":{"AUC":"0.667"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-tnl2k","task":"Visual Object Tracking","dataset":"TNL2K","model":"LoRAT-g-378","rank_in_archive_order":5,"of":16,"metrics":{"AUC":"62.7","precision":"67.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-tnl2k","task":"Visual Object Tracking","dataset":"TNL2K","model":"LoRAT-L-378","rank_in_archive_order":6,"of":16,"metrics":{"AUC":"62.3","precision":"67.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-trackingnet","task":"Visual Object Tracking","dataset":"TrackingNet","model":"LoRAT-g-378","rank_in_archive_order":9,"of":40,"metrics":{"Accuracy":"86.0","Normalized Precision":"90.2","Precision":"86.1"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-trackingnet","task":"Visual Object Tracking","dataset":"TrackingNet","model":"LoRAT-L-378","rank_in_archive_order":10,"of":40,"metrics":{"Accuracy":"85.6","Normalized Precision":"89.7","Precision":"85.4"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-uav123","task":"Visual Object Tracking","dataset":"UAV123","model":"LoRAT-g-378","rank_in_archive_order":1,"of":16,"metrics":{"AUC":"0.739"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-uav123","task":"Visual Object Tracking","dataset":"UAV123","model":"LoRAT-L-378","rank_in_archive_order":3,"of":16,"metrics":{"AUC":"0.725"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.05231","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05231"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/litinglin/lorat","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/LitingLin/LoRAT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5a695ef0811cab18","entry":"find_all_frozen_nn_linear_names","repo":"LitingLin/LoRAT","repo_kind":"official","path":"trackit/models/methods/LoRAT/funcs/vit_lora_utils.py","file_url":"https://github.com/LitingLin/LoRAT/blob/HEAD/trackit/models/methods/LoRAT/funcs/vit_lora_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a695ef0811cab18"}},{"code_sha256_prefix":"4b916f2f46e88d85","entry":"freeze_vit_backbone_","repo":"LitingLin/LoRAT","repo_kind":"official","path":"trackit/models/methods/LoRAT/funcs/vit_backbone_freeze.py","file_url":"https://github.com/LitingLin/LoRAT/blob/HEAD/trackit/models/methods/LoRAT/funcs/vit_backbone_freeze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b916f2f46e88d85"}},{"code_sha256_prefix":"c311b3c84e280fa0","entry":"generate_LoRAT_sample_data","repo":"LitingLin/LoRAT","repo_kind":"official","path":"trackit/models/methods/LoRAT/funcs/sample_data.py","file_url":"https://github.com/LitingLin/LoRAT/blob/HEAD/trackit/models/methods/LoRAT/funcs/sample_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c311b3c84e280fa0"}},{"code_sha256_prefix":"7dcc0654aaf67867","entry":"enable_lora_","repo":"LitingLin/LoRAT","repo_kind":"official","path":"trackit/models/methods/LoRAT/funcs/vit_lora_utils.py","file_url":"https://github.com/LitingLin/LoRAT/blob/HEAD/trackit/models/methods/LoRAT/funcs/vit_lora_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7dcc0654aaf67867"}},{"code_sha256_prefix":"f6f3949d8af0fe7f","entry":"state_dict_with_lora_meta_attributes","repo":"LitingLin/LoRAT","repo_kind":"official","path":"trackit/models/methods/LoRAT/funcs/vit_lora_utils.py","file_url":"https://github.com/LitingLin/LoRAT/blob/HEAD/trackit/models/methods/LoRAT/funcs/vit_lora_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6f3949d8af0fe7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}