Papers › Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment

Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment

31 May 2023arXiv:2305.19478archive 2025-07-28

Quoc-Huy Tran, Ahmed Mehmood, Muhammad Ahmed, Muhammad Naufil, Anas Zafar, Andrey Konin, M. Zeeshan Zia

This paper presents an unsupervised transformer-based framework for temporal activity segmentation which leverages not only frame-level cues but also segment-level cues. This is in contrast with previous methods which often rely on frame-level information only. Our approach begins with a frame-level prediction module which estimates framewise action classes via a transformer encoder. The frame-level prediction module is trained in an unsupervised manner via temporal optimal transport. To exploit segment-level information, we utilize a segment-level prediction module and a frame-to-segment alignment module. The former includes a transformer decoder for estimating video transcripts, while the latter matches frame-level features with segment-level features, yielding permutation-aware segmentation results. Moreover, inspired by temporal optimal transport, we introduce simple-yet-effective pseudo labels for unsupervised training of the above modules. Our experiments on four public datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop Assembly show that our approach achieves comparable or better performance than previous methods in unsupervised activity segmentation.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

trquhuytin/TOT-CVPR22 officialpytorchMIT report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action Segmentation Decoder Prediction Segmentation Unsupervised Action Segmentation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Unsupervised Action Segmentation	Breakfast	UFSA	Acc	52.1	#3 of 8	Archive leaderboard	report
Unsupervised Action Segmentation	Breakfast	UFSA	F1	38	#3 of 8	Archive leaderboard	report
Unsupervised Action Segmentation	Youtube INRIA Instructional	UFSA	Acc	49.6	#5 of 8	Archive leaderboard	report
Unsupervised Action Segmentation	Youtube INRIA Instructional	UFSA	F1	32.4	#5 of 8	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections