Papers › Efficient Temporal Action Segmentation via Boundary-aware Query Voting

Efficient Temporal Action Segmentation via Boundary-aware Query Voting

25 May 2024arXiv:2405.15995archive 2025-07-28

Peiyao Wang, Yuewei Lin, Erik Blasch, Jie Wei, Haibin Ling

Although the performance of Temporal Action Segmentation (TAS) has improved in recent years, achieving promising results often comes with a high computational cost due to dense inputs, complex model structures, and resource-intensive post-processing requirements. To improve the efficiency while keeping the performance, we present a novel perspective centered on per-segment classification. By harnessing the capabilities of Transformers, we tokenize each video segment as an instance token, endowed with intrinsic instance segmentation. To realize efficient action segmentation, we introduce BaFormer, a boundary-aware Transformer network. It employs instance queries for instance segmentation and a global query for class-agnostic boundary prediction, yielding continuous segment proposals. During inference, BaFormer employs a simple yet effective voting strategy to classify boundary-wise segments based on instance segmentation. Remarkably, as a single-stage approach, BaFormer significantly reduces the computational costs, utilizing only 6% of the running time compared to state-of-the-art method DiffAct, while producing better or comparable accuracy over several popular benchmarks. The code for this project is publicly available at https://github.com/peiyao-w/BaFormer.

Paper PDF Code

Code

peiyao-w/baformer officialmentioned in paper report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action Segmentation Instance Segmentation Segmentation Semantic Segmentation Temporal Action Segmentation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Action Segmentation	50 Salads	BaFormer	Acc	89.5	#3 of 28	Archive leaderboard	report
Action Segmentation	50 Salads	BaFormer	Edit	84.2	#3 of 28	Archive leaderboard	report
Action Segmentation	50 Salads	BaFormer	F1@10%	89.3	#3 of 28	Archive leaderboard	report
Action Segmentation	50 Salads	BaFormer	F1@25%	88.4	#3 of 28	Archive leaderboard	report
Action Segmentation	50 Salads	BaFormer	F1@50%	83.9	#3 of 28	Archive leaderboard	report
Action Segmentation	Breakfast	BaFormer	Acc	76.6	#6 of 37	Archive leaderboard	report
Action Segmentation	Breakfast	BaFormer	Average F1	72.4	#6 of 37	Archive leaderboard	report
Action Segmentation	Breakfast	BaFormer	Edit	77.3	#6 of 37	Archive leaderboard	report
Action Segmentation	Breakfast	BaFormer	F1@10%	79.2	#6 of 37	Archive leaderboard	report
Action Segmentation	Breakfast	BaFormer	F1@25%	74.9	#6 of 37	Archive leaderboard	report
Action Segmentation	Breakfast	BaFormer	F1@50%	63.2	#6 of 37	Archive leaderboard	report
Action Segmentation	GTEA	BaFormer	Acc	83.0	#4 of 28	Archive leaderboard	report
Action Segmentation	GTEA	BaFormer	Edit	88.7	#4 of 28	Archive leaderboard	report
Action Segmentation	GTEA	BaFormer	F1@10%	92.0	#4 of 28	Archive leaderboard	report
Action Segmentation	GTEA	BaFormer	F1@25%	91.3	#4 of 28	Archive leaderboard	report
Action Segmentation	GTEA	BaFormer	F1@50%	83.5	#4 of 28	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections