{"url":"/task/action-segmentation","name":"Action Segmentation","slug":"action-segmentation","description_markdown":"**Action Segmentation** is a challenging problem in high-level video understanding. In its simplest form, Action Segmentation aims to segment a temporally untrimmed video by time and label each segmented part with one of pre-defined action labels. The results of Action Segmentation can be further used as input to various applications, such as video-to-text and action localization.\n\n\n<span class=\"description-source\">Source: [TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation ](https://arxiv.org/abs/1705.07818)</span>","categories":[{"name":"Computer Vision","url":"/area/computer-vision"}],"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","slug_source":"archive_url"},"counts":{"papers_tagged":219,"papers_with_code":99,"benchmarks":9,"benchmark_tables_in_archive":9,"benchmark_tables_shown":9,"benchmark_tables_withheld_as_spam":0,"benchmark_definition":"a leaderboard table with at least one row; benchmark_tables_shown also counts the zero-row tables; benchmark_tables_in_archive adds the tables withheld as spam","datasets":18,"subtasks":4,"parent_tasks":1},"benchmarks":[{"leaderboard":"/sota/action-segmentation-on-breakfast-1","slug":"action-segmentation-on-breakfast-1","dataset":"Breakfast","dataset_url":"/dataset/breakfast","rows_in_archive":37,"metrics":["Average F1","F1@50%","F1@25%","F1@10%","Edit","Acc","mIoU","F1"],"first_row_in_archive_order":{"model":"AdaFocus (newly extracted I3D-features, LT-Context model)","paper_title":"Towards Weakly Supervised End-to-end Learning for Long-video Action Recognition","paper_url":"/paper/adafocus-towards-end-to-end-weakly-supervised","paper_date":"2023-11-28","arxiv_id":"2311.17118","code_links":[],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-50-salads-1","slug":"action-segmentation-on-50-salads-1","dataset":"50 Salads","dataset_url":"/dataset/50-salads","rows_in_archive":28,"metrics":["F1@50%","F1@25%","F1@10%","Acc","Edit"],"first_row_in_archive_order":{"model":"Br-Prompt+ASPnet (RGB, flow, accelerometer)","paper_title":"ASPnet: Action Segmentation With Shared-Private Representation of Multiple Data Sources","paper_url":"/paper/aspnet-action-segmentation-with-shared","paper_date":"2023-01-01","arxiv_id":null,"code_links":[],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-gtea-1","slug":"action-segmentation-on-gtea-1","dataset":"GTEA","dataset_url":"/dataset/gtea","rows_in_archive":28,"metrics":["F1@50%","F1@25%","F1@10%","Acc","Edit"],"first_row_in_archive_order":{"model":"Semantic2Graph","paper_title":"Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos","paper_url":"/paper/semantic2graph-graph-based-multi-modal","paper_date":"2022-09-13","arxiv_id":"2209.05653","code_links":[],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-coin","slug":"action-segmentation-on-coin","dataset":"COIN","dataset_url":"/dataset/coin","rows_in_archive":9,"metrics":["Frame accuracy"],"first_row_in_archive_order":{"model":"UnLoc-L","paper_title":"UnLoc: A Unified Framework for Video Localization Tasks","paper_url":"/paper/unloc-a-unified-framework-for-video","paper_date":"2023-08-21","arxiv_id":"2308.11062","code_links":[{"title":"google-research/scenic","url":"https://github.com/google-research/scenic"}],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-assembly101","slug":"action-segmentation-on-assembly101","dataset":"Assembly101","dataset_url":"/dataset/assembly101","rows_in_archive":7,"metrics":["F1@10%","F1@25%","F1@50%","Edit","MoF"],"first_row_in_archive_order":{"model":"ASQuery","paper_title":"ASQuery: A Query-based Model for Action Segmentation","paper_url":"/paper/asquery-a-query-based-model-for-action","paper_date":"2024-09-30","arxiv_id":null,"code_links":[{"title":"zlngan/ASQuery","url":"https://github.com/zlngan/ASQuery"}],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-jigsaws","slug":"action-segmentation-on-jigsaws","dataset":"JIGSAWS","dataset_url":"/dataset/jigsaws","rows_in_archive":7,"metrics":["Edit Distance","Accuracy","F1@10","F1@25","F1@50"],"first_row_in_archive_order":{"model":"MRG-Net","paper_title":"Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery","paper_url":"/paper/relational-graph-learning-on-visual-and","paper_date":"2020-11-03","arxiv_id":"2011.01619","code_links":[],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-youtube-inria","slug":"action-segmentation-on-youtube-inria","dataset":"Youtube INRIA Instructional","dataset_url":"/dataset/youtube-inria-instructional","rows_in_archive":2,"metrics":["Acc","F1"],"first_row_in_archive_order":{"model":"TSA (FINCH)","paper_title":"Leveraging triplet loss for unsupervised action segmentation","paper_url":"/paper/leveraging-triplet-loss-for-unsupervised","paper_date":"2023-04-13","arxiv_id":"2304.06403","code_links":[{"title":"elenabbbuenob/tsa-actionseg","url":"https://github.com/elenabbbuenob/tsa-actionseg"}],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-50salads","slug":"action-segmentation-on-50salads","dataset":"50Salads","dataset_url":null,"rows_in_archive":1,"metrics":["Acc","Edit","F1@10%","F1@25%","F1@50%"],"first_row_in_archive_order":{"model":"EUT","paper_title":"Do we really need temporal convolutions in action segmentation?","paper_url":"/paper/efficient-u-transformer-with-boundary-aware","paper_date":"2022-05-26","arxiv_id":"2205.13425","code_links":[{"title":"ddz16/TUT","url":"https://github.com/ddz16/TUT"}],"syntology":null}},{"leaderboard":"/sota/action-segmentation-on-mpii-cooking-2-dataset","slug":"action-segmentation-on-mpii-cooking-2-dataset","dataset":"MPII Cooking 2 Dataset","dataset_url":"/dataset/mpii-cooking-2-dataset","rows_in_archive":1,"metrics":["Accuracy","mIoU"],"first_row_in_archive_order":{"model":"Unsup. TW-FINCH (K=avg/activity)","paper_title":"Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation","paper_url":"/paper/temporally-weighted-hierarchical-clustering","paper_date":"2021-03-20","arxiv_id":"2103.11264","code_links":[{"title":"ssarfraz/FINCH-CLustering","url":"https://github.com/ssarfraz/FINCH-CLustering"}],"syntology":{"n":7,"n_ran":5,"n_unverified":2,"n_pointer_only":1}}}],"datasets":[{"url":"/dataset/breakfast","name":"Breakfast","full_name":"The Breakfast Actions Dataset","num_papers_in_archive":179},{"url":"/dataset/gtea","name":"GTEA","full_name":"Georgia Tech Egocentric Activity","num_papers_in_archive":120},{"url":"/dataset/coin","name":"COIN","full_name":"","num_papers_in_archive":105},{"url":"/dataset/jigsaws","name":"JIGSAWS","full_name":"JHU-ISI Gesture and Skill Assessment Working Set","num_papers_in_archive":105},{"url":"/dataset/assembly101","name":"Assembly101","full_name":"","num_papers_in_archive":57},{"url":"/dataset/50-salads","name":"50 Salads","full_name":"","num_papers_in_archive":35},{"url":"/dataset/hoi4d","name":"HOI4D","full_name":"","num_papers_in_archive":23},{"url":"/dataset/watch-n-patch","name":"Watch-n-Patch","full_name":"Watch-n-Patch","num_papers_in_archive":13},{"url":"/dataset/egoexolearn","name":"EgoExoLearn","full_name":"","num_papers_in_archive":12},{"url":"/dataset/egoprocel","name":"EgoProceL","full_name":"","num_papers_in_archive":9},{"url":"/dataset/youtube-inria-instructional","name":"Youtube INRIA Instructional","full_name":"Unsupervised learning from narrated instruction videos","num_papers_in_archive":8},{"url":"/dataset/tum-kitchen","name":"TUM Kitchen","full_name":"TUM Kitchen","num_papers_in_archive":7},{"url":"/dataset/mpii-cooking-2-dataset","name":"MPII Cooking 2 Dataset","full_name":"","num_papers_in_archive":5},{"url":"/dataset/ha-vid","name":"HA-ViD","full_name":"HA-ViD: A Human Assembly Video Dataset","num_papers_in_archive":1},{"url":"/dataset/inhard","name":"InHARD","full_name":"Industrial Human Action Recognition Dataset in the Context of Industrial Collaborative Robotics","num_papers_in_archive":1},{"url":"/dataset/lara","name":"LARa","full_name":"Logistic Activity Recognition Challenge","num_papers_in_archive":1},{"url":"/dataset/uw-iom","name":"UW IOM","full_name":"University of Washington Indoor Object Manipulation","num_papers_in_archive":1},{"url":"/dataset/sics-155","name":"SICS-155","full_name":"Phase Recognition in Small Incision Cataract Surgery Videos","num_papers_in_archive":0}],"subtasks":[{"url":"/task/skeleton-based-action-segmentation","name":"Skeleton Based Action Segmentation"},{"url":"/task/unsupervised-action-segmentation","name":"Unsupervised Action Segmentation"},{"url":"/task/weakly-supervised-action-segmentation","name":"Weakly Supervised Action Segmentation (Transcript)"},{"url":"/task/weakly-supervised-action-segmentation-action","name":"Weakly Supervised Action Segmentation (Action Set))"}],"parent_tasks":[{"url":"/task/action-localization","name":"Action Localization"}],"papers":{"order":"repositories listed in the archive (desc), then date (desc); the archive holds no stars","population":"papers tagged with this task that list at least one repository in the archive","shown":30,"of":99,"tagged_in_all":219,"items":[{"url":"/paper/temporal-convolutional-networks-for-action","title":"Temporal Convolutional Networks for Action Segmentation and Detection","date":"2016-11-16","arxiv_id":"1611.05267","repositories_listed":5,"syntology":{"n":21,"n_ran":2,"n_unverified":19,"n_pointer_only":0}},{"url":"/paper/end-to-end-learning-of-visual-representations","title":"End-to-End Learning of Visual Representations from Uncurated Instructional Videos","date":"2019-12-13","arxiv_id":"1912.06430","repositories_listed":4,"syntology":{"n":5,"n_ran":3,"n_unverified":2,"n_pointer_only":0}},{"url":"/paper/hierarchical-neurosymbolic-approach-for","title":"Hierarchical NeuroSymbolic Approach for Comprehensive and Explainable Action Quality Assessment","date":"2024-03-20","arxiv_id":"2403.13798","repositories_listed":3,"syntology":null},{"url":"/paper/temporal-action-segmentation-an-analysis-of","title":"Temporal Action Segmentation: An Analysis of Modern Techniques","date":"2022-10-19","arxiv_id":"2210.10352","repositories_listed":3,"syntology":{"n":4,"n_ran":2,"n_unverified":2,"n_pointer_only":0}},{"url":"/paper/logo-a-long-form-video-dataset-for-group-1","title":"LOGO: A Long-Form Video Dataset for Group Action Quality Assessment","date":"2024-04-07","arxiv_id":"2404.05029","repositories_listed":2,"syntology":{"n":4,"n_ran":4,"n_unverified":0,"n_pointer_only":4}},{"url":"/paper/unified-fully-and-timestamp-supervised","title":"Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation","date":"2022-09-01","arxiv_id":"2209.00638","repositories_listed":2,"syntology":{"n":2,"n_ran":2,"n_unverified":0,"n_pointer_only":2}},{"url":"/paper/rf-next-efficient-receptive-field-search-for","title":"RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks","date":"2022-06-14","arxiv_id":"2206.06637","repositories_listed":2,"syntology":{"n":3,"n_ran":2,"n_unverified":1,"n_pointer_only":3}},{"url":"/paper/videoclip-contrastive-pre-training-for-zero","title":"VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding","date":"2021-09-28","arxiv_id":"2109.14084","repositories_listed":2,"syntology":null},{"url":"/paper/global2local-efficient-structure-search-for","title":"Global2Local: Efficient Structure Search for Video Action Segmentation","date":"2021-01-04","arxiv_id":"2101.00910","repositories_listed":2,"syntology":{"n":3,"n_ran":3,"n_unverified":0,"n_pointer_only":3}},{"url":"/paper/univilm-a-unified-video-and-language-pre","title":"UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation","date":"2020-02-15","arxiv_id":"2002.06353","repositories_listed":2,"syntology":{"n":4,"n_ran":1,"n_unverified":3,"n_pointer_only":1}},{"url":"/paper/unsupervised-learning-of-action-classes-with","title":"Unsupervised learning of action classes with continuous temporal embedding","date":"2019-04-08","arxiv_id":"1904.04189","repositories_listed":2,"syntology":null},{"url":"/paper/ms-tcn-multi-stage-temporal-convolutional","title":"MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation","date":"2019-03-05","arxiv_id":"1903.01945","repositories_listed":2,"syntology":null},{"url":"/paper/hopadiff-holistic-partial-aware-fourier","title":"HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios","date":"2025-06-11","arxiv_id":"2506.09650","repositories_listed":1,"syntology":{"n":11,"n_ran":4,"n_unverified":7,"n_pointer_only":0}},{"url":"/paper/epfl-smart-kitchen-30-densely-annotated","title":"EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models","date":"2025-06-02","arxiv_id":"2506.01608","repositories_listed":1,"syntology":null},{"url":"/paper/cost-sensitive-learning-for-long-tailed","title":"Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentation","date":"2025-03-24","arxiv_id":"2503.18358","repositories_listed":1,"syntology":null},{"url":"/paper/mamba4d-efficient-4d-point-cloud-video","title":"Mamba4D: Efficient 4D Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models","date":"2025-01-01","arxiv_id":null,"repositories_listed":1,"syntology":null},{"url":"/paper/hierarchical-vector-quantization-for","title":"Hierarchical Vector Quantization for Unsupervised Action Segmentation","date":"2024-12-23","arxiv_id":"2412.17640","repositories_listed":1,"syntology":null},{"url":"/paper/onlinetas-an-online-baseline-for-temporal","title":"OnlineTAS: An Online Baseline for Temporal Action Segmentation","date":"2024-11-02","arxiv_id":"2411.01122","repositories_listed":1,"syntology":null},{"url":"/paper/language-assisted-skeleton-action","title":"Language-Assisted Skeleton Action Understanding for Skeleton-Based Temporal Action Segmentation","date":"2024-10-31","arxiv_id":null,"repositories_listed":1,"syntology":null},{"url":"/paper/promqa-question-answering-dataset-for","title":"ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understanding","date":"2024-10-29","arxiv_id":"2410.22211","repositories_listed":1,"syntology":null},{"url":"/paper/language-assisted-human-part-motion-learning","title":"Language-Assisted Human Part Motion Learning for Skeleton-Based Temporal Action Segmentation","date":"2024-10-08","arxiv_id":"2410.06353","repositories_listed":1,"syntology":null},{"url":"/paper/asquery-a-query-based-model-for-action","title":"ASQuery: A Query-based Model for Action Segmentation","date":"2024-09-30","arxiv_id":null,"repositories_listed":1,"syntology":null},{"url":"/paper/transformer-with-controlled-attention-for","title":"Transformer with Controlled Attention for Synchronous Motion Captioning","date":"2024-09-13","arxiv_id":"2409.09177","repositories_listed":1,"syntology":null},{"url":"/paper/3d-pose-based-temporal-action-segmentation","title":"3D Pose-Based Temporal Action Segmentation for Figure Skating: A Fine-Grained and Jump Procedure-Aware Annotation Approach","date":"2024-08-29","arxiv_id":"2408.16638","repositories_listed":1,"syntology":null},{"url":"/paper/long-tail-temporal-action-segmentation-with","title":"Long-Tail Temporal Action Segmentation with Group-wise Temporal Logit Adjustment","date":"2024-08-19","arxiv_id":"2408.09919","repositories_listed":1,"syntology":null},{"url":"/paper/a-study-of-animal-action-segmentation","title":"A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms","date":"2024-07-23","arxiv_id":"2407.16727","repositories_listed":1,"syntology":null},{"url":"/paper/efficient-temporal-action-segmentation-via","title":"Efficient Temporal Action Segmentation via Boundary-aware Query Voting","date":"2024-05-25","arxiv_id":"2405.15995","repositories_listed":1,"syntology":null},{"url":"/paper/snippet-aware-transformer-with-multiple","title":"Snippet-Aware Transformer With Multiple Action Elements for Skeleton-Based Action Segmentation","date":"2024-05-06","arxiv_id":null,"repositories_listed":1,"syntology":null},{"url":"/paper/temporally-consistent-unbalanced-optimal","title":"Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentation","date":"2024-04-01","arxiv_id":"2404.01518","repositories_listed":1,"syntology":null},{"url":"/paper/efficient-and-effective-weakly-supervised","title":"Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment","date":"2024-03-28","arxiv_id":"2403.19225","repositories_listed":1,"syntology":null}],"syntology_records":9,"syntology_note":"a paper without a record is not a recorded non-run: it may lack an arXiv id or simply be absent from the graph layer"},"description_links":{"kept":0,"unwrapped_to_text":0,"bare_urls_linked":0,"relative_images_dropped":0,"rule":"internal links are kept only when the target slug exists in the catalog"},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per-sample execution status on synthesized fixtures ('ran N of M samples'); not a correctness claim and not a ranking signal.","status_vocabulary":{"ran_honours":"ran, honoured the contract we drafted","ran_violates":"ran, violated the contract we drafted","ran_draft_wrong":"ran; our contract draft was wrong, not the code","ran_fixture":"ran; our fixture could not drive it","ran":"ran on a synthesized input","unverified":"unverified (harvested, no recorded run)"}},"not_shown":{"libraries":"the archive has no per-task library table","trend_sparklines":"the Trend column of the benchmarks table was a rendered image; it is not in the archive","social_and_latest_sorts":"stars and social signals are not in the archive"}}