{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/macdiff-unified-skeleton-modeling-with-masked","title":"MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusion","arxiv_id":"2409.10473","date":"2024-09-16","proceeding":null,"authors":["Lehong Wu","Lilang Lin","Jiahang Zhang","Yiyang Ma","Jiaying Liu"],"abstract":"Self-supervised learning has proved effective for skeleton-based human action understanding. However, previous works either rely on contrastive learning that suffers false negative problems or are based on reconstruction that learns too much unessential low-level clues, leading to limited representations for downstream tasks. Recently, great advances have been made in generative learning, which is naturally a challenging yet meaningful pretext task to model the general underlying data distributions. However, the representation learning capacity of generative models is under-explored, especially for the skeletons with spacial sparsity and temporal redundancy. To this end, we propose Masked Conditional Diffusion (MacDiff) as a unified framework for human skeleton modeling. For the first time, we leverage diffusion models as effective skeleton representation learners. Specifically, we train a diffusion decoder conditioned on the representations extracted by a semantic encoder. Random masking is applied to encoder inputs to introduce a information bottleneck and remove redundancy of skeletons. Furthermore, we theoretically demonstrate that our generative objective involves the contrastive learning objective which aligns the masked and noisy views. Meanwhile, it also enforces the representation to complement for the noisy view, leading to better generalization performance. MacDiff achieves state-of-the-art performance on representation learning benchmarks while maintaining the competence for generative tasks. Moreover, we leverage the diffusion model for data augmentation, significantly enhancing the fine-tuning performance in scenarios with scarce labeled data. Our project is available at https://lehongwu.github.io/ECCV24MacDiff/.","url_abs":"https://arxiv.org/abs/2409.10473v1","url_pdf":"https://arxiv.org/pdf/2409.10473v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-understanding","task_name":"Action Understanding"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.10473","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.10473"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LehongWu/MacDiff","reach":null}],"summary":{"ran":7,"ran_fixture":1,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"c41cf977fcc5f5cb","entry":"Attention","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c41cf977fcc5f5cb"}},{"code_sha256_prefix":"d87bd702280998e1","entry":"Block","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d87bd702280998e1"}},{"code_sha256_prefix":"f035f943e5114e94","entry":"DecoderBlock","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f035f943e5114e94"}},{"code_sha256_prefix":"0096ff0f534391b0","entry":"FeatureModulation","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0096ff0f534391b0"}},{"code_sha256_prefix":"44ea42448b5bf98c","entry":"GaussianDiffusion","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44ea42448b5bf98c"}},{"code_sha256_prefix":"71aeba8837a31bba","entry":"SkeleEmbed","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71aeba8837a31bba"}},{"code_sha256_prefix":"391ceeaed087e497","entry":"UniformSampler","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"391ceeaed087e497"}},{"code_sha256_prefix":"9b6e4e070473e76f","entry":"space_timesteps","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b6e4e070473e76f"}},{"code_sha256_prefix":"9cc3dab0e22fbf67","entry":"ScheduleSampler","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9cc3dab0e22fbf67"}},{"code_sha256_prefix":"9a13449b9c6b302b","entry":"SpacedDiffusion","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a13449b9c6b302b"}},{"code_sha256_prefix":"2d91a1ea1d185ce8","entry":"Transformer","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d91a1ea1d185ce8"}},{"code_sha256_prefix":"9bb0960c615d2f3e","entry":"create_gaussian_diffusion","repo":"LehongWu/MacDiff","repo_kind":"found_in_text","path":"model/transformer_macdiff.py","file_url":"https://github.com/LehongWu/MacDiff/blob/HEAD/model/transformer_macdiff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9bb0960c615d2f3e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}