Papers › Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens

Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens

3 Jan 2024arXiv:2401.01674archive 2025-07-28

Dengdi Sun, Yajie Pan, Andong Lu, Chenglong Li, Bin Luo

Many RGBT tracking researches primarily focus on modal fusion design, while overlooking the effective handling of target appearance changes. While some approaches have introduced historical frames or fuse and replace initial templates to incorporate temporal information, they have the risk of disrupting the original target appearance and accumulating errors over time. To alleviate these limitations, we propose a novel Transformer RGBT tracking approach, which mixes spatio-temporal multimodal tokens from the static multimodal templates and multimodal search regions in Transformer to handle target appearance changes, for robust RGBT tracking. We introduce independent dynamic template tokens to interact with the search region, embedding temporal information to address appearance changes, while also retaining the involvement of the initial static template tokens in the joint feature extraction process to ensure the preservation of the original reliable target appearance information that prevent deviations from the target appearance caused by traditional temporal updates. We also use attention mechanisms to enhance the target features of multimodal template tokens by incorporating supplementary modal cues, and make the multimodal search region tokens interact with multimodal dynamic template tokens via attention mechanisms, which facilitates the conveyance of multimodal-enhanced target change information. Our module is inserted into the transformer backbone network and inherits joint feature extraction, search-template matching, and cross-modal interaction. Extensive experiments on three RGBT benchmark datasets show that the proposed approach maintains competitive performance compared to other state-of-the-art tracking algorithms while running at 39.1 FPS.

Paper PDF

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Rgb-T Tracking Template Matching

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Rgb-T Tracking	LasHeR	STMT	Precision	67.4	#27 of 39	Archive leaderboard	report
Rgb-T Tracking	LasHeR	STMT	Success	53.7	#27 of 39	Archive leaderboard	report
Rgb-T Tracking	RGBT210	STMT	Precision	83.0	#15 of 19	Archive leaderboard	report
Rgb-T Tracking	RGBT210	STMT	Success	59.5	#15 of 19	Archive leaderboard	report
Rgb-T Tracking	RGBT234	STMT	Precision	86.5	#27 of 42	Archive leaderboard	report
Rgb-T Tracking	RGBT234	STMT	Success	63.8	#27 of 42	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Focus Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections