{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-assisted-skeleton-action","title":"Language-Assisted Skeleton Action Understanding for Skeleton-Based Temporal Action Segmentation","arxiv_id":null,"date":"2024-10-31","proceeding":"European Conference on Computer Vision (ECCV2024) 2024 10","authors":["Haoyu Ji","Bowen Chen","Xinglong Xu","Weihong Ren","Zhiyong Wang","Honghai Liu"],"abstract":"Skeleton-based Temporal Action Segmentation (STAS) aims to densely segment and classify human actions in long, untrimmed skeletal motion sequences. Existing STAS methods primarily model spatial dependencies among joints and temporal relationships among frames to generate frame-level one-hot classifications. However, these methods overlook the deep mining of semantic relations among joints as well as actions at a linguistic level, which limits the comprehensiveness of skeleton action understanding. In this work, we propose a Language-assisted Skeleton Action Understanding (LaSA) method that leverages the language modality to assist in learning semantic relationships among joints and actions. Specifically, in terms of joint relationships, the Joint Relationships Establishment (JRE) module establishes correlations among joints in the feature sequence by applying attention between joint texts and differentiates distinct joints by embedding joint texts as positional embeddings. Regarding action relationships, the Action Relationships Supervision (ARS) module enhances the discrimination across action classes through contrastive learning of single-class action-text pairs and models the semantic associations of adjacent actions by contrasting mixed-class clip-text pairs. Performance evaluation on five public datasets demonstrates that LaSA achieves state-of-the-art results. Code is available at https://github.com/HaoyuJi/LaSA.","url_abs":"https://link.springer.com/chapter/10.1007/978-3-031-72949-2_23","url_pdf":"https://link.springer.com/content/pdf/10.1007/978-3-031-72949-2_23.pdf?pdf=inline%20link","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-assisted-skeleton-action","repo_url":"https://github.com/HaoyuJi/LaSA","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-segmentation","task_name":"Action Segmentation"},{"task_slug":"action-understanding","task_name":"Action Understanding"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"},{"task_slug":"skeleton-based-action-segmentation","task_name":"Skeleton Based Action Segmentation"},{"task_slug":"temporal-action-segmentation","task_name":"Temporal Action Segmentation"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}