{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/compositional-human-scene-interaction","title":"Compositional Human-Scene Interaction Synthesis with Semantic Control","arxiv_id":"2207.12824","date":"2022-07-26","proceeding":null,"authors":["Kaifeng Zhao","Shaofei Wang","Yan Zhang","Thabo Beeler","Siyu Tang"],"abstract":"Synthesizing natural interactions between virtual humans and their 3D environments is critical for numerous applications, such as computer games and AR/VR experiences. Our goal is to synthesize humans interacting with a given 3D scene controlled by high-level semantic specifications as pairs of action categories and object instances, e.g., \"sit on the chair\". The key challenge of incorporating interaction semantics into the generation framework is to learn a joint representation that effectively captures heterogeneous information, including human body articulation, 3D object geometry, and the intent of the interaction. To address this challenge, we design a novel transformer-based generative model, in which the articulated 3D human body surface points and 3D objects are jointly encoded in a unified latent space, and the semantics of the interaction between the human and objects are embedded via positional encoding. Furthermore, inspired by the compositional nature of interactions that humans can simultaneously interact with multiple objects, we define interaction semantics as the composition of varying numbers of atomic action-object pairs. Our proposed generative model can naturally incorporate varying numbers of atomic interactions, which enables synthesizing compositional human-scene interactions without requiring composite interaction data. We extend the PROX dataset with interaction semantic labels and scene instance segmentation to evaluate our method and demonstrate that our method can generate realistic human-scene interactions with semantic control. Our perceptual study shows that our synthesized virtual humans can naturally interact with 3D scenes, considerably outperforming existing methods. We name our method COINS, for COmpositional INteraction Synthesis with Semantic Control. Code and data are available at https://github.com/zkf1997/COINS.","url_abs":"https://arxiv.org/abs/2207.12824v1","url_pdf":"https://arxiv.org/pdf/2207.12824v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"compositional-human-scene-interaction","repo_url":"https://github.com/zkf1997/coins","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.12824","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.12824"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zkf1997/coins","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/zkf1997/COINS","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_honours":1,"ran_fixture":1,"unverified":10},"by_repo_kind":{"official":{"samples":13,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d402177f304ff725","entry":"get_embedder","repo":"zkf1997/COINS","repo_kind":"official","path":"interaction/interaction_model.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/interaction/interaction_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d402177f304ff725"}},{"code_sha256_prefix":"ec413739d406e611","entry":"pc_normalize","repo":"zkf1997/COINS","repo_kind":"official","path":"interaction/pointnet2.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/interaction/pointnet2.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec413739d406e611"}},{"code_sha256_prefix":"580dee6761649b29","entry":"square_distance","repo":"zkf1997/COINS","repo_kind":"official","path":"interaction/pointnet2.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/interaction/pointnet2.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"580dee6761649b29"}},{"code_sha256_prefix":"eeb1f268b855ffcd","entry":"batch_sparse_dense_matmul","repo":"zkf1997/COINS","repo_kind":"official","path":"interaction/graph_layers.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/interaction/graph_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eeb1f268b855ffcd"}},{"code_sha256_prefix":"0d9a9295115ecbdd","entry":"calc_diversity","repo":"zkf1997/COINS","repo_kind":"official","path":"evaluation/eval_results.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/evaluation/eval_results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d9a9295115ecbdd"}},{"code_sha256_prefix":"e3c7e257f04448c7","entry":"eval_physical_metric","repo":"zkf1997/COINS","repo_kind":"official","path":"evaluation/posa_metric_utils.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/evaluation/posa_metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3c7e257f04448c7"}},{"code_sha256_prefix":"00f2be9bc4677fd1","entry":"evaluate_diversity","repo":"zkf1997/COINS","repo_kind":"official","path":"evaluation/eval_results.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/evaluation/eval_results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00f2be9bc4677fd1"}},{"code_sha256_prefix":"a88c9c2fb6e25051","entry":"load_posa","repo":"zkf1997/COINS","repo_kind":"official","path":"evaluation/load_results.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/evaluation/load_results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a88c9c2fb6e25051"}},{"code_sha256_prefix":"1c8bf84ad350dd01","entry":"load_results","repo":"zkf1997/COINS","repo_kind":"official","path":"evaluation/load_results.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/evaluation/load_results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c8bf84ad350dd01"}},{"code_sha256_prefix":"f467082e21a9734c","entry":"padded_idx_to_code","repo":"zkf1997/COINS","repo_kind":"official","path":"interaction/interaction_model.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/interaction/interaction_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f467082e21a9734c"}},{"code_sha256_prefix":"b5060a2fb98d78e2","entry":"read_sdf","repo":"zkf1997/COINS","repo_kind":"official","path":"evaluation/posa_metric_utils.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/evaluation/posa_metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5060a2fb98d78e2"}},{"code_sha256_prefix":"9b987af9445edb28","entry":"spmm","repo":"zkf1997/COINS","repo_kind":"official","path":"interaction/graph_layers.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/interaction/graph_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b987af9445edb28"}},{"code_sha256_prefix":"b1227ddb721e2999","entry":"timeit","repo":"zkf1997/COINS","repo_kind":"official","path":"interaction/pointnet2.py","file_url":"https://github.com/zkf1997/COINS/blob/HEAD/interaction/pointnet2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1227ddb721e2999"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}