{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tdsm-triplet-diffusion-for-skeleton-text","title":"TDSM: Triplet Diffusion for Skeleton-Text Matching in Zero-Shot Action Recognition","arxiv_id":"2411.10745","date":"2024-11-16","proceeding":null,"authors":["Jeonghyeok Do","Munchurl Kim"],"abstract":"We firstly present a diffusion-based action recognition with zero-shot learning for skeleton inputs. In zero-shot skeleton-based action recognition, aligning skeleton features with the text features of action labels is essential for accurately predicting unseen actions. Previous methods focus on direct alignment between skeleton and text latent spaces, but the modality gaps between these spaces hinder robust generalization learning. Motivated from the remarkable performance of text-to-image diffusion models, we leverage their alignment capabilities between different modalities mostly by focusing on the training process during reverse diffusion rather than using their generative power. Based on this, our framework is designed as a Triplet Diffusion for Skeleton-Text Matching (TDSM) method which aligns skeleton features with text prompts through reverse diffusion, embedding the prompts into the unified skeleton-text latent space to achieve robust matching. To enhance discriminative power, we introduce a novel triplet diffusion (TD) loss that encourages our TDSM to correct skeleton-text matches while pushing apart incorrect ones. Our TDSM significantly outperforms the very recent state-of-the-art methods with large margins of 2.36%-point to 13.05%-point, demonstrating superior accuracy and scalability in zero-shot settings through effective skeleton-text matching.","url_abs":"https://arxiv.org/abs/2411.10745v2","url_pdf":"https://arxiv.org/pdf/2411.10745v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tdsm-triplet-diffusion-for-skeleton-text","repo_url":"https://github.com/KAIST-VICLab/TDSM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":null,"task_name":"Triplet"},{"task_slug":"zero-shot-skeletal-action-recognition","task_name":"Zero Shot Skeletal Action Recognition"},{"task_slug":"zero-shot-action-recognition","task_name":"Zero-Shot Action Recognition"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-skeletal-action-recognition-on-ntu","task":"Zero Shot Skeletal Action Recognition","dataset":"NTU RGB+D","model":"TDSM","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy (12 unseen classes)":"56.03","Accuracy (5 unseen classes)":"86.49","Random Split Accuracy":"88.88"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-skeletal-action-recognition-on-ntu-1","task":"Zero Shot Skeletal Action Recognition","dataset":"NTU RGB+D 120","model":"TDSM","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy (10 unseen classes)":"74.15","Accuracy (24 unseen classes)":"65.06","Random Split Accuracy":"69.47"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-skeletal-action-recognition-on-pku","task":"Zero Shot Skeletal Action Recognition","dataset":"PKU-MMD","model":"TDSM","rank_in_archive_order":2,"of":7,"metrics":{"Random Split Accuracy":"70.76"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.10745","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}