Papers › Video Sparse Transformer With Attention-Guided Memory for Video Object Detection

Video Sparse Transformer With Attention-Guided Memory for Video Object Detection

17 Jun 2022IEEE Access 2022 6archive 2025-07-28

Masato Fujitake, Akihiro Sugimoto

Detecting objects in a video, known as Video Object Detection (VOD), is challenging since appearance changes of objects over time may bring detection errors. Recent research has focused on aggregating features from adjacent frames to compensate for the deteriorated appearances of a frame. Moreover, using distant frames is also proposed to deal with deteriorated appearances over several frames. Since an object’s position may change significantly at a distant frame, they only use features of object candidate regions, which do not depend on their position. However, such methods rely on object candidate regions’ detection performance and are not practical for deteriorated appearances. In this paper, we enhance features element-wisely before the object candidate region detection, proposing Video Sparse Transformer with Attention-guided Memory (VSTAM). Furthermore, we propose aggregating element-wise features sparsely to reduce processing time and memory cost. In addition, we introduce an external memory update strategy based on the utilization of the aggregation to hold long-term information effectively. Our method achieved 8.3% and 11.1% accuracy gain from the baseline on ImageNet VID and UA-DETRAC datasets. Our method demonstrates superior performance against state-of-the-art results on widely used VOD datasets.

Paper PDF Code

Code

Malik1998/VSTAM pytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Object Object Detection Video Instance Segmentation Video Object Detection object-detection

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Object Detection	UA-DETRAC	VSTAM	mAP	90.39	#1 of 9	Archive leaderboard	report
Video Instance Segmentation	YouTube-VIS validation	VSTAM	mask AP	39.0	#26 of 44	Archive leaderboard	report
Video Object Detection	ImageNet VID	VSTAM	MAP	91.1	#4 of 33	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention Attention Dropout BPE Cosine Annealing Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Linear Warmup With Cosine Annealing Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Sparse Transformer Transformer Weight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections