{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-modal-orthogonal-high-rank-augmentation","title":"Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers","arxiv_id":"2307.04129","date":"2023-07-09","proceeding":"ICCV 2023 1","authors":["Zhiyu Zhu","Junhui Hou","Dapeng Oliver Wu"],"abstract":"This paper addresses the problem of cross-modal object tracking from RGB videos and event data. Rather than constructing a complex cross-modal fusion network, we explore the great potential of a pre-trained vision Transformer (ViT). Particularly, we delicately investigate plug-and-play training augmentations that encourage the ViT to bridge the vast distribution gap between the two modalities, enabling comprehensive cross-modal information interaction and thus enhancing its ability. Specifically, we propose a mask modeling strategy that randomly masks a specific modality of some tokens to enforce the interaction between tokens from different modalities interacting proactively. To mitigate network oscillations resulting from the masking strategy and further amplify its positive effect, we then theoretically propose an orthogonal high-rank loss to regularize the attention matrix. Extensive experiments demonstrate that our plug-and-play training augmentation techniques can significantly boost state-of-the-art one-stream and twostream trackers to a large extent in terms of both tracking precision and success rate. Our new perspective and findings will potentially bring insights to the field of leveraging powerful pre-trained ViTs to model cross-modal data. The code will be publicly available.","url_abs":"https://arxiv.org/abs/2307.04129v2","url_pdf":"https://arxiv.org/pdf/2307.04129v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-modal-orthogonal-high-rank-augmentation","repo_url":"https://github.com/ZHU-Zhiyu/High-Rank_RGB-Event_Tracker","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"cross-modal-orthogonal-high-rank-augmentation","repo_url":"https://github.com/zhu-zhiyu/nvs_solver","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"object-tracking","task_name":"Object Tracking"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-tracking-on-coesot","task":"Object Tracking","dataset":"COESOT","model":"HR-CEUTrack-Large","rank_in_archive_order":1,"of":12,"metrics":{"Precision Rate":"73.8","Success Rate":"65.0"},"uses_additional_data":false},{"leaderboard":"/sota/object-tracking-on-coesot","task":"Object Tracking","dataset":"COESOT","model":"HR-CEUTrack-Base","rank_in_archive_order":2,"of":12,"metrics":{"Precision Rate":"71.9","Success Rate":"63.2"},"uses_additional_data":false},{"leaderboard":"/sota/object-tracking-on-fe108","task":"Object Tracking","dataset":"FE108","model":"HR-MonTrack-Base","rank_in_archive_order":1,"of":8,"metrics":{"Averaged Precision":"96.2","Success Rate":"68.5"},"uses_additional_data":false},{"leaderboard":"/sota/object-tracking-on-fe108","task":"Object Tracking","dataset":"FE108","model":"HR-MonTrack-Tiny","rank_in_archive_order":2,"of":8,"metrics":{"Averaged Precision":"95.3","Success Rate":"66.3"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.04129","atlas_url":"https://app.syntology.ai/?focus=2307.04129","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.04129"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhu-zhiyu/nvs_solver","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZHU-Zhiyu/High-Rank_RGB-Event_Tracker","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"768c216e0d96a716","entry":"candidate_elimination","repo":"ZHU-Zhiyu/High-Rank_RGB-Event_Tracker","repo_kind":"official","path":"HRCEUTrack/Base/lib/models/layers/attn_blocks.py","file_url":"https://github.com/ZHU-Zhiyu/High-Rank_RGB-Event_Tracker/blob/HEAD/HRCEUTrack/Base/lib/models/layers/attn_blocks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"768c216e0d96a716"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}