{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cooperative-holistic-scene-understanding","title":"Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation","arxiv_id":"1810.13049","date":"2018-10-31","proceeding":"NeurIPS 2018 12","authors":["Siyuan Huang","Siyuan Qi","Yinxue Xiao","Yixin Zhu","Ying Nian Wu","Song-Chun Zhu"],"abstract":"Holistic 3D indoor scene understanding refers to jointly recovering the i)\nobject bounding boxes, ii) room layout, and iii) camera pose, all in 3D. The\nexisting methods either are ineffective or only tackle the problem partially.\nIn this paper, we propose an end-to-end model that simultaneously solves all\nthree tasks in real-time given only a single RGB image. The essence of the\nproposed method is to improve the prediction by i) parametrizing the targets\n(e.g., 3D boxes) instead of directly estimating the targets, and ii)\ncooperative training across different modules in contrast to training these\nmodules individually. Specifically, we parametrize the 3D object bounding boxes\nby the predictions from several modules, i.e., 3D camera pose and object\nattributes. The proposed method provides two major advantages: i) The\nparametrization helps maintain the consistency between the 2D image and the 3D\nworld, thus largely reducing the prediction variances in 3D coordinates. ii)\nConstraints can be imposed on the parametrization to train different modules\nsimultaneously. We call these constraints \"cooperative losses\" as they enable\nthe joint training and inference. We employ three cooperative losses for 3D\nbounding boxes, 2D projections, and physical constraints to estimate a\ngeometrically consistent and physically plausible 3D scene. Experiments on the\nSUN RGB-D dataset shows that the proposed method significantly outperforms\nprior approaches on 3D object detection, 3D layout estimation, 3D camera pose\nestimation, and holistic scene understanding.","url_abs":"http://arxiv.org/abs/1810.13049v2","url_pdf":"http://arxiv.org/pdf/1810.13049v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cooperative-holistic-scene-understanding","repo_url":"https://github.com/thusiyuan/cooperative_scene_parsing","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"camera-pose-estimation","task_name":"Camera Pose Estimation"},{"task_slug":"monocular-3d-object-detection","task_name":"Monocular 3D Object Detection"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"room-layout-estimation","task_name":"Room Layout Estimation"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/monocular-3d-object-detection-on-sun-rgb-d","task":"Monocular 3D Object Detection","dataset":"SUN RGB-D","model":"Cooperative","rank_in_archive_order":5,"of":7,"metrics":{"AP@0.15 (10 / NYU-37)":"23.65","AP@0.15 (10 / PNet-30)":"23.65","AP@0.15 (NYU-37)":"12.23"},"uses_additional_data":false},{"leaderboard":"/sota/room-layout-estimation-on-sun-rgb-d","task":"Room Layout Estimation","dataset":"SUN RGB-D","model":"Cooperative","rank_in_archive_order":5,"of":7,"metrics":{"Camera Pitch":"3.28","Camera Roll":"2.19","IoU":"56.9"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1810.13049","atlas_url":"https://app.syntology.ai/?focus=1810.13049","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1810.13049"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thusiyuan/cooperative_scene_parsing","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":17},"by_repo_kind":{"official":{"samples":18,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"d7d70eec1f26a2ee","entry":"basis_from_ori","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/net_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/net_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7d70eec1f26a2ee"}},{"code_sha256_prefix":"b3b1caa583de1ef2","entry":"conv","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/net_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/net_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3b1caa583de1ef2"}},{"code_sha256_prefix":"d00913003a1ae7d1","entry":"flip_towards_viewer","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/sunrgbd_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/sunrgbd_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d00913003a1ae7d1"}},{"code_sha256_prefix":"d070fe9a565666f7","entry":"get_corners_of_bb3d","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/net_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/net_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d070fe9a565666f7"}},{"code_sha256_prefix":"dbcddde3dfca915c","entry":"get_corners_of_bb3d","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/sunrgbd_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/sunrgbd_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbcddde3dfca915c"}},{"code_sha256_prefix":"df2844ad942a7725","entry":"get_corners_of_bb3d_no_index","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/sunrgbd_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/sunrgbd_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df2844ad942a7725"}},{"code_sha256_prefix":"a93788c49d78992f","entry":"hex_to_rgb","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/vis_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/vis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a93788c49d78992f"}},{"code_sha256_prefix":"eabf66badd22efef","entry":"plot_cuboid","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/vis_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/vis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eabf66badd22efef"}},{"code_sha256_prefix":"c5c7080f6ecf3d19","entry":"resnet18","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5c7080f6ecf3d19"}},{"code_sha256_prefix":"da793c6ec62d1b24","entry":"resnet34","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da793c6ec62d1b24"}},{"code_sha256_prefix":"e35af01f865dda39","entry":"rotation_matrix_3d_x","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/camera_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/camera_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e35af01f865dda39"}},{"code_sha256_prefix":"fcc42a597d7af788","entry":"rotation_matrix_3d_y","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/camera_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/camera_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcc42a597d7af788"}},{"code_sha256_prefix":"61e36ec80c121182","entry":"rotation_matrix_3d_z","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/camera_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/camera_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61e36ec80c121182"}},{"code_sha256_prefix":"5d5053380b78ff19","entry":"set_logger","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"config.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d5053380b78ff19"}},{"code_sha256_prefix":"2b3009efbb7c1daa","entry":"show_2dboxes","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"utils/vis_utils.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/utils/vis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b3009efbb7c1daa"}},{"code_sha256_prefix":"7f8010cad4b21210","entry":"show_2dboxes","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"preprocess/sunrgbd/sunrgbd_visualize.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/preprocess/sunrgbd/sunrgbd_visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f8010cad4b21210"}},{"code_sha256_prefix":"aa3870a02a8d9ff8","entry":"show_2dcorner","repo":"thusiyuan/cooperative_scene_parsing","repo_kind":"official","path":"preprocess/sunrgbd/sunrgbd_visualize.py","file_url":"https://github.com/thusiyuan/cooperative_scene_parsing/blob/HEAD/preprocess/sunrgbd/sunrgbd_visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa3870a02a8d9ff8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}