{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-supervised-training-for-skeleton","title":"Generative Action Description Prompts for Skeleton-based Action Recognition","arxiv_id":"2208.05318","date":"2022-08-10","proceeding":"ICCV 2023 1","authors":["Wangmeng Xiang","Chao Li","Yuxuan Zhou","Biao Wang","Lei Zhang"],"abstract":"Skeleton-based action recognition has recently received considerable attention. Current approaches to skeleton-based action recognition are typically formulated as one-hot classification tasks and do not fully exploit the semantic relations between actions. For example, \"make victory sign\" and \"thumb up\" are two actions of hand gestures, whose major difference lies in the movement of hands. This information is agnostic from the categorical one-hot encoding of action classes but could be unveiled from the action description. Therefore, utilizing action description in training could potentially benefit representation learning. In this work, we propose a Generative Action-description Prompts (GAP) approach for skeleton-based action recognition. More specifically, we employ a pre-trained large-scale language model as the knowledge engine to automatically generate text descriptions for body parts movements of actions, and propose a multi-modal training scheme by utilizing the text encoder to generate feature vectors for different body parts and supervise the skeleton encoder for action representation learning. Experiments show that our proposed GAP method achieves noticeable improvements over various baseline models without extra computation cost at inference. GAP achieves new state-of-the-arts on popular skeleton-based action recognition benchmarks, including NTU RGB+D, NTU RGB+D 120 and NW-UCLA. The source code is available at https://github.com/MartinXM/GAP.","url_abs":"https://arxiv.org/abs/2208.05318v2","url_pdf":"https://arxiv.org/pdf/2208.05318v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-supervised-training-for-skeleton","repo_url":"https://github.com/martinxm/gap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-supervised-training-for-skeleton","repo_url":"https://github.com/martinxm/lst","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-supervised-training-for-skeleton","repo_url":"https://github.com/maclong01/STEP-CATFormer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/skeleton-based-action-recognition-on-n-ucla","task":"Skeleton Based Action Recognition","dataset":"N-UCLA","model":"LST","rank_in_archive_order":7,"of":25,"metrics":{"Accuracy":"97.2"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-ntu-rgbd","task":"Skeleton Based Action Recognition","dataset":"NTU RGB+D","model":"LST","rank_in_archive_order":23,"of":135,"metrics":{"Accuracy (CS)":"92.9","Accuracy (CV)":"97","Ensembled Modalities":"4"},"uses_additional_data":true},{"leaderboard":"/sota/skeleton-based-action-recognition-on-ntu-rgbd-1","task":"Skeleton Based Action Recognition","dataset":"NTU RGB+D 120","model":"LST","rank_in_archive_order":13,"of":83,"metrics":{"Accuracy (Cross-Setup)":"91.1","Accuracy (Cross-Subject)":"89.9","Ensembled Modalities":"4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.05318","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.05318"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/martinxm/gap","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maclong01/STEP-CATFormer","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/martinxm/lst","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"ran":1,"ran_violates":2,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"616b5e82dd2a38f9","entry":"count_params","repo":"martinxm/lst","repo_kind":"official","path":"utils.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"616b5e82dd2a38f9"}},{"code_sha256_prefix":"5426f077b824bbb4","entry":"activation_factory","repo":"martinxm/lst","repo_kind":"official","path":"model/activation.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/model/activation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5426f077b824bbb4"}},{"code_sha256_prefix":"9ec19e473ca8e62e","entry":"gen_label","repo":"martinxm/lst","repo_kind":"official","path":"tools.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/tools.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ec19e473ca8e62e"}},{"code_sha256_prefix":"ffed4f85d50832c9","entry":"import_class","repo":"martinxm/lst","repo_kind":"official","path":"model/baseline.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/model/baseline.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ffed4f85d50832c9"}},{"code_sha256_prefix":"4e946d7c92be9d41","entry":"import_class","repo":"martinxm/gap","repo_kind":"official","path":"main_multipart_ntu.py","file_url":"https://github.com/martinxm/gap/blob/HEAD/main_multipart_ntu.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e946d7c92be9d41"}},{"code_sha256_prefix":"f017532fc389cbfe","entry":"str2bool","repo":"martinxm/lst","repo_kind":"official","path":"utils.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f017532fc389cbfe"}},{"code_sha256_prefix":"a957019ca36a6747","entry":"str2bool","repo":"martinxm/lst","repo_kind":"official","path":"main_multipart_ntu.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/main_multipart_ntu.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a957019ca36a6747"}},{"code_sha256_prefix":"13c6de98a5cc764e","entry":"build_model","repo":"martinxm/lst","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13c6de98a5cc764e"}},{"code_sha256_prefix":"cad29a7a417c68d3","entry":"create_logits","repo":"martinxm/lst","repo_kind":"official","path":"tools.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cad29a7a417c68d3"}},{"code_sha256_prefix":"b91b981015bbf498","entry":"load","repo":"martinxm/lst","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/martinxm/lst/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b91b981015bbf498"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}