{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/4d-panoptic-scene-graph-generation-1","title":"4D Panoptic Scene Graph Generation","arxiv_id":"2405.10305","date":"2024-05-16","proceeding":"NeurIPS 2023 11","authors":["Jingkang Yang","Jun Cen","Wenxuan Peng","Shuai Liu","Fangzhou Hong","Xiangtai Li","Kaiyang Zhou","Qifeng Chen","Ziwei Liu"],"abstract":"We are living in a three-dimensional space while moving forward through a fourth dimension: time. To allow artificial intelligence to develop a comprehensive understanding of such a 4D environment, we introduce 4D Panoptic Scene Graph (PSG-4D), a new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding. Specifically, PSG-4D abstracts rich 4D sensory data into nodes, which represent entities with precise location and status information, and edges, which capture the temporal relations. To facilitate research in this new area, we build a richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of 1M frames, each of which is labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs. To solve PSG-4D, we propose PSG4DFormer, a Transformer-based model that can predict panoptic segmentation masks, track masks along the time axis, and generate the corresponding scene graphs via a relation component. Extensive experiments on the new dataset show that our method can serve as a strong baseline for future research on PSG-4D. In the end, we provide a real-world application example to demonstrate how we can achieve dynamic scene understanding by integrating a large language model into our PSG-4D system.","url_abs":"https://arxiv.org/abs/2405.10305v1","url_pdf":"https://arxiv.org/pdf/2405.10305v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"4d-panoptic-scene-graph-generation-1","repo_url":"https://github.com/jingkang50/psg4d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"4d-panoptic-scene-graph-generation-1","repo_url":"https://github.com/Jingkang50/OpenPSG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"4d-panoptic-scene-graph-generation-1","repo_url":"https://github.com/jingkang50/openpvsg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"4d-panoptic-segmentation","task_name":"4D Panoptic Segmentation"},{"task_slug":"graph-generation","task_name":"Graph Generation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"panoptic-scene-graph-generation","task_name":"Panoptic Scene Graph Generation"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.10305","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.10305"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jingkang50/OpenPSG","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingkang50/psg4d","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingkang50/openpvsg","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":10,"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"f4c6fd89f0168001","entry":"argsort_desc","repo":"Jingkang50/OpenPSG","repo_kind":"official","path":"openpsg/evaluation/sgg_eval_util.py","file_url":"https://github.com/Jingkang50/OpenPSG/blob/HEAD/openpsg/evaluation/sgg_eval_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4c6fd89f0168001"}},{"code_sha256_prefix":"6693fd4e3fbcd72f","entry":"attention","repo":"jingkang50/psg4d","repo_kind":"official","path":"pc_seg/backbone/transformer.py","file_url":"https://github.com/jingkang50/psg4d/blob/HEAD/pc_seg/backbone/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6693fd4e3fbcd72f"}},{"code_sha256_prefix":"9a93ecf3e946d063","entry":"compute_vn","repo":"jingkang50/psg4d","repo_kind":"official","path":"pc_seg/lib/segmentator/utils.py","file_url":"https://github.com/jingkang50/psg4d/blob/HEAD/pc_seg/lib/segmentator/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a93ecf3e946d063"}},{"code_sha256_prefix":"361f4cf883997964","entry":"compute_vn_kernel","repo":"jingkang50/psg4d","repo_kind":"official","path":"pc_seg/lib/segmentator/utils.py","file_url":"https://github.com/jingkang50/psg4d/blob/HEAD/pc_seg/lib/segmentator/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"361f4cf883997964"}},{"code_sha256_prefix":"f4c579a3b51908a0","entry":"format_result_dict","repo":"Jingkang50/OpenPSG","repo_kind":"official","path":"openpsg/evaluation/sgg_eval.py","file_url":"https://github.com/Jingkang50/OpenPSG/blob/HEAD/openpsg/evaluation/sgg_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4c579a3b51908a0"}},{"code_sha256_prefix":"891b8ebab395921f","entry":"get_clones","repo":"jingkang50/psg4d","repo_kind":"official","path":"pc_seg/backbone/transformer.py","file_url":"https://github.com/jingkang50/psg4d/blob/HEAD/pc_seg/backbone/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"891b8ebab395921f"}},{"code_sha256_prefix":"4c158fc0e48afd02","entry":"initialize_optimizer","repo":"jingkang50/psg4d","repo_kind":"official","path":"pc_seg/utils/solver.py","file_url":"https://github.com/jingkang50/psg4d/blob/HEAD/pc_seg/utils/solver.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c158fc0e48afd02"}},{"code_sha256_prefix":"521ea446997228b2","entry":"initialize_scheduler","repo":"jingkang50/psg4d","repo_kind":"official","path":"pc_seg/utils/solver.py","file_url":"https://github.com/jingkang50/psg4d/blob/HEAD/pc_seg/utils/solver.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"521ea446997228b2"}},{"code_sha256_prefix":"c3e5199833bfb97a","entry":"intersect_2d","repo":"Jingkang50/OpenPSG","repo_kind":"official","path":"openpsg/evaluation/sgg_eval_util.py","file_url":"https://github.com/Jingkang50/OpenPSG/blob/HEAD/openpsg/evaluation/sgg_eval_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3e5199833bfb97a"}},{"code_sha256_prefix":"1360e4a605aad164","entry":"joint_stracks","repo":"Jingkang50/PSG4D","repo_kind":"official","path":"pc_track/unitrack/basetrack.py","file_url":"https://github.com/Jingkang50/PSG4D/blob/HEAD/pc_track/unitrack/basetrack.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1360e4a605aad164"}},{"code_sha256_prefix":"0df7ff0ca2d84435","entry":"mask_iou","repo":"Jingkang50/OpenPSG","repo_kind":"official","path":"openpsg/evaluation/sgg_metrics.py","file_url":"https://github.com/Jingkang50/OpenPSG/blob/HEAD/openpsg/evaluation/sgg_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0df7ff0ca2d84435"}},{"code_sha256_prefix":"9fc317bb1ed0030e","entry":"preprocess_panoptic_gt_split_th_st","repo":"jingkang50/openpvsg","repo_kind":"official","path":"models/mask2former/mask2former_head_split_focal.py","file_url":"https://github.com/jingkang50/openpvsg/blob/HEAD/models/mask2former/mask2former_head_split_focal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9fc317bb1ed0030e"}},{"code_sha256_prefix":"30a3c33b9572d851","entry":"sub_stracks","repo":"Jingkang50/PSG4D","repo_kind":"official","path":"pc_track/unitrack/basetrack.py","file_url":"https://github.com/Jingkang50/PSG4D/blob/HEAD/pc_track/unitrack/basetrack.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30a3c33b9572d851"}},{"code_sha256_prefix":"c9470c4dd1975f70","entry":"remove_duplicate_stracks","repo":"Jingkang50/PSG4D","repo_kind":"official","path":"pc_track/unitrack/basetrack.py","file_url":"https://github.com/Jingkang50/PSG4D/blob/HEAD/pc_track/unitrack/basetrack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9470c4dd1975f70"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}