{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/motionbert-unified-pretraining-for-human","title":"MotionBERT: A Unified Perspective on Learning Human Motion Representations","arxiv_id":"2210.06551","date":"2022-10-12","proceeding":"ICCV 2023 1","authors":["Wentao Zhu","Xiaoxuan Ma","Zhaoyang Liu","Libin Liu","Wayne Wu","Yizhou Wang"],"abstract":"We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. Specifically, we propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. The motion representations acquired in this way incorporate geometric, kinematic, and physical knowledge about human motion, which can be easily transferred to multiple downstream tasks. We implement the motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. Furthermore, our proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with a simple regression head (1-2 layers), which demonstrates the versatility of the learned motion representations. Code and models are available at https://motionbert.github.io/","url_abs":"https://arxiv.org/abs/2210.06551v5","url_pdf":"https://arxiv.org/pdf/2210.06551v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"motionbert-unified-pretraining-for-human","repo_url":"https://github.com/Walter0807/MotionBERT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"3d-human-pose-estimation","task_name":"3D Human Pose Estimation"},{"task_slug":"3d-pose-estimation","task_name":"3D Pose Estimation"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"monocular-3d-human-pose-estimation","task_name":"Monocular 3D Human Pose Estimation"},{"task_slug":"one-shot-3d-action-recognition","task_name":"One-Shot 3D Action Recognition"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-human-pose-estimation-on-3dpw","task":"3D Human Pose Estimation","dataset":"3DPW","model":"MotionBERT-HybrIK","rank_in_archive_order":14,"of":119,"metrics":{"MPJPE":"68.8","MPVPE":"79.4","PA-MPJPE":"40.6"},"uses_additional_data":false},{"leaderboard":"/sota/3d-human-pose-estimation-on-3dpw","task":"3D Human Pose Estimation","dataset":"3DPW","model":"MotionBERT (Finetune)","rank_in_archive_order":37,"of":119,"metrics":{"MPJPE":"76.9","MPVPE":"88.1","PA-MPJPE":"47.2"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-full-body-parkinsons","task":"Classification","dataset":"Full-body Parkinson’s disease dataset","model":"MotionBERT","rank_in_archive_order":3,"of":7,"metrics":{"F1-score (weighted)":"0.47"},"uses_additional_data":false},{"leaderboard":"/sota/classification-on-full-body-parkinsons","task":"Classification","dataset":"Full-body Parkinson’s disease dataset","model":"MotionBERT-LITE","rank_in_archive_order":5,"of":7,"metrics":{"F1-score (weighted)":"0.43"},"uses_additional_data":false},{"leaderboard":"/sota/monocular-3d-human-pose-estimation-on-human3","task":"Monocular 3D Human Pose Estimation","dataset":"Human3.6M","model":"MotionBERT (Finetune)","rank_in_archive_order":1,"of":52,"metrics":{"2D detector":"SH","Average MPJPE (mm)":"37.5","Frames Needed":"243","Need Ground Truth 2D Pose":"No","Use Video Sequence":"Yes"},"uses_additional_data":true},{"leaderboard":"/sota/monocular-3d-human-pose-estimation-on-human3","task":"Monocular 3D Human Pose Estimation","dataset":"Human3.6M","model":"MotionBERT (Scratch)","rank_in_archive_order":4,"of":52,"metrics":{"2D detector":"SH","Average MPJPE (mm)":"39.2","Frames Needed":"243","Need Ground Truth 2D Pose":"No","Use Video Sequence":"Yes"},"uses_additional_data":false},{"leaderboard":"/sota/one-shot-3d-action-recognition-on-ntu-rgbd","task":"One-Shot 3D Action Recognition","dataset":"NTU RGB+D 120","model":"MotionBERT (Finetune)","rank_in_archive_order":3,"of":10,"metrics":{"Accuracy":"67.4%"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-ntu-rgbd","task":"Skeleton Based Action Recognition","dataset":"NTU RGB+D","model":"MotionBert (finetune)","rank_in_archive_order":21,"of":135,"metrics":{"Accuracy (CS)":"93.0","Accuracy (CV)":"97.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.06551","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.06551"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Walter0807/MotionBERT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran":3,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3ac6b7d76e8e3584","entry":"drop_path","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"lib/model/drop.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/lib/model/drop.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ac6b7d76e8e3584"}},{"code_sha256_prefix":"48a788f0938a2209","entry":"loss_mpjpe","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"lib/model/loss.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/lib/model/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48a788f0938a2209"}},{"code_sha256_prefix":"08fdd93d7a20142f","entry":"mpjpe","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"lib/model/loss.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/lib/model/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08fdd93d7a20142f"}},{"code_sha256_prefix":"e2b944da85a57144","entry":"p_mpjpe","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"lib/model/loss.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/lib/model/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2b944da85a57144"}},{"code_sha256_prefix":"02566da69866c48c","entry":"trunc_normal_","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"lib/model/DSTformer.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/lib/model/DSTformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"02566da69866c48c"}},{"code_sha256_prefix":"2e29155b32d35c84","entry":"err","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"infer_wild_mesh.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/infer_wild_mesh.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e29155b32d35c84"}},{"code_sha256_prefix":"bd08312c6f24ba04","entry":"extract_feats","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"train_action_1shot.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/train_action_1shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd08312c6f24ba04"}},{"code_sha256_prefix":"775e1419f4a8e087","entry":"solve_scale","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"infer_wild_mesh.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/infer_wild_mesh.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"775e1419f4a8e087"}},{"code_sha256_prefix":"ed6e8cfa9cd438dc","entry":"validate","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"train_action.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/train_action.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed6e8cfa9cd438dc"}},{"code_sha256_prefix":"66db75824a727d6b","entry":"validate","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"train_action_1shot.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/train_action_1shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66db75824a727d6b"}},{"code_sha256_prefix":"b91c99fd64360d1a","entry":"validate","repo":"Walter0807/MotionBERT","repo_kind":"official","path":"train_mesh.py","file_url":"https://github.com/Walter0807/MotionBERT/blob/HEAD/train_mesh.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b91c99fd64360d1a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}