{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-unified-diffusion-framework-for-scene-aware","title":"A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals","arxiv_id":"2404.04890","date":"2024-04-07","proceeding":"CVPR 2024 1","authors":["Jiangnan Tang","Jingya Wang","Kaiyang Ji","Lan Xu","Jingyi Yu","Ye Shi"],"abstract":"Estimating full-body human motion via sparse tracking signals from head-mounted displays and hand controllers in 3D scenes is crucial to applications in AR/VR. One of the biggest challenges to this task is the one-to-many mapping from sparse observations to dense full-body motions, which endowed inherent ambiguities. To help resolve this ambiguous problem, we introduce a new framework to combine rich contextual information provided by scenes to benefit full-body motion tracking from sparse observations. To estimate plausible human motions given sparse tracking signals and 3D scenes, we develop $\\text{S}^2$Fusion, a unified framework fusing \\underline{S}cene and sparse \\underline{S}ignals with a conditional dif\\underline{Fusion} model. $\\text{S}^2$Fusion first extracts the spatial-temporal relations residing in the sparse signals via a periodic autoencoder, and then produces time-alignment feature embedding as additional inputs. Subsequently, by drawing initial noisy motion from a pre-trained prior, $\\text{S}^2$Fusion utilizes conditional diffusion to fuse scene geometry and sparse tracking signals to generate full-body scene-aware motions. The sampling procedure of $\\text{S}^2$Fusion is further guided by a specially designed scene-penetration loss and phase-matching loss, which effectively regularizes the motion of the lower body even in the absence of any tracking signals, making the generated motion much more plausible and coherent. Extensive experimental results have demonstrated that our $\\text{S}^2$Fusion outperforms the state-of-the-art in terms of estimation quality and smoothness.","url_abs":"https://arxiv.org/abs/2404.04890v1","url_pdf":"https://arxiv.org/pdf/2404.04890v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-unified-diffusion-framework-for-scene-aware","repo_url":"https://github.com/jn-tang/s2fusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"motion-estimation","task_name":"Motion Estimation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.04890","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.04890"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jn-tang/s2fusion","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/jn-tang/S2Fusion","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_fixture":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1ce10587ab046d89","entry":"create_conditioning_from_conf","repo":"jn-tang/S2Fusion","repo_kind":"official","path":"model/conditioning.py","file_url":"https://github.com/jn-tang/S2Fusion/blob/HEAD/model/conditioning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ce10587ab046d89"}},{"code_sha256_prefix":"183c819df5293137","entry":"rot_mat_to_euler","repo":"jn-tang/S2Fusion","repo_kind":"official","path":"model/SMPL/lbs.py","file_url":"https://github.com/jn-tang/S2Fusion/blob/HEAD/model/SMPL/lbs.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"183c819df5293137"}},{"code_sha256_prefix":"00ccf5b4e0e83030","entry":"sinusoidal_position_embedding","repo":"jn-tang/S2Fusion","repo_kind":"official","path":"model/backbone/embedding.py","file_url":"https://github.com/jn-tang/S2Fusion/blob/HEAD/model/backbone/embedding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00ccf5b4e0e83030"}},{"code_sha256_prefix":"43ebe3b786b2d59b","entry":"to_np","repo":"jn-tang/S2Fusion","repo_kind":"official","path":"model/SMPL/lbs.py","file_url":"https://github.com/jn-tang/S2Fusion/blob/HEAD/model/SMPL/lbs.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43ebe3b786b2d59b"}},{"code_sha256_prefix":"94904a5bd2e1074f","entry":"to_tensor","repo":"jn-tang/S2Fusion","repo_kind":"official","path":"model/SMPL/lbs.py","file_url":"https://github.com/jn-tang/S2Fusion/blob/HEAD/model/SMPL/lbs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94904a5bd2e1074f"}},{"code_sha256_prefix":"88f252aa03d23f0b","entry":"create_loss_from_conf","repo":"jn-tang/S2Fusion","repo_kind":"official","path":"model/conditioning.py","file_url":"https://github.com/jn-tang/S2Fusion/blob/HEAD/model/conditioning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88f252aa03d23f0b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}