Papers › CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition

CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition

18 Jan 2021arXiv:2101.07042archive 2025-07-28

Shreyank N Gowda, Laura Sevilla-Lara, Frank Keller, Marcus Rohrbach

Zero-shot action recognition is the task of recognizingaction classes without visual examples, only with a seman-tic embedding which relates unseen to seen classes. Theproblem can be seen as learning a function which general-izes well to instances of unseen classes without losing dis-crimination between classes. Neural networks can modelthe complex boundaries between visual classes, which ex-plains their success as supervised models. However, inzero-shot learning, these highly specialized class bound-aries may not transfer well from seen to unseen classes.In this paper we propose a centroid-based representation,which clusters visual and semantic representation, consid-ers all training samples at once, and in this way generaliz-ing well to instances from unseen classes. We optimize theclustering using Reinforcement Learning which we show iscritical for our approach to work. We call the proposedmethod CLASTER and observe that it consistently outper-forms the state-of-the-art in all standard datasets, includ-ing UCF101, HMDB51 and Olympic Sports; both in thestandard zero-shot evaluation and the generalized zero-shotlearning. Further, we show that our model performs com-petitively in the image domain as well, outperforming thestate-of-the-art in many settings.

Paper PDF

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action Recognition Clustering Generalized Zero-Shot Learning Reinforcement Learning Reinforcement Learning (RL)Zero-Shot Action Recognition Zero-Shot Learning reinforcement-learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Zero-Shot Action Recognition	HMDB51	CLASTER	Top-1 Accuracy	43.2	#14 of 29	Archive leaderboard	report
Zero-Shot Action Recognition	Olympics	CLASTER	Top-1 Accuracy	68.4	#2 of 9	Archive leaderboard	report
Zero-Shot Action Recognition	UCF101	CLASTER	Top-1 Accuracy	53.9	#19 of 35	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections