Papers › Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation

Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation

9 Feb 2016arXiv:1602.02995archive 2025-07-28

Colin Lea, Austin Reiter, Rene Vidal, Gregory D. Hager

Joint segmentation and classification of fine-grained actions is important for applications of human-robot interaction, video surveillance, and human skill evaluation. However, despite substantial recent progress in large-scale action classification, the performance of state-of-the-art fine-grained action recognition approaches remains low. We propose a model for action segmentation which combines low-level spatiotemporal features with a high-level segmental classifier. Our spatiotemporal CNN is comprised of a spatial component that uses convolutional filters to capture information about objects and their relationships, and a temporal component that uses large 1D convolutional filters to capture information about how object relationships change across time. These features are used in tandem with a semi-Markov model that models transitions from one action to another. We introduce an efficient constrained segmental inference algorithm for this model that is orders of magnitude faster than the current approach. We highlight the effectiveness of our Segmental Spatiotemporal CNN on cooking and surgical action datasets for which we observe substantially improved performance relative to recent baseline methods.

Paper PDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action Classification Action Recognition Action Segmentation Fine-grained Action Recognition General Classification Segmentation Temporal Action Localization

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Action Segmentation	GTEA	ST-CNN	Acc	60.6	#28 of 28	Archive leaderboard	report
Action Segmentation	GTEA	ST-CNN	Edit	-	#28 of 28	Archive leaderboard	report
Action Segmentation	GTEA	ST-CNN	F1@10%	58.7	#28 of 28	Archive leaderboard	report
Action Segmentation	GTEA	ST-CNN	F1@25%	54.4	#28 of 28	Archive leaderboard	report
Action Segmentation	GTEA	ST-CNN	F1@50%	41.9	#28 of 28	Archive leaderboard	report
Action Segmentation	JIGSAWS	ST-CNN+Seg	Accuracy	74.22	#7 of 7	Archive leaderboard	report
Action Segmentation	JIGSAWS	ST-CNN+Seg	Edit Distance	66.56	#7 of 7	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections