{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/3d-aware-visual-question-answering-about-1","title":"3D-Aware Visual Question Answering about Parts, Poses and Occlusions","arxiv_id":"2310.17914","date":"2023-10-27","proceeding":"NeurIPS 2023 11","authors":["Xingrui Wang","Wufei Ma","Zhuowan Li","Adam Kortylewski","Alan Yuille"],"abstract":"Despite rapid progress in Visual question answering (VQA), existing datasets and models mainly focus on testing reasoning in 2D. However, it is important that VQA models also understand the 3D structure of visual scenes, for example to support tasks like navigation or manipulation. This includes an understanding of the 3D object pose, their parts and occlusions. In this work, we introduce the task of 3D-aware VQA, which focuses on challenging questions that require a compositional reasoning over the 3D structure of visual scenes. We address 3D-aware VQA from both the dataset and the model perspective. First, we introduce Super-CLEVR-3D, a compositional reasoning dataset that contains questions about object parts, their 3D poses, and occlusions. Second, we propose PO3D-VQA, a 3D-aware VQA model that marries two powerful ideas: probabilistic neural symbolic program execution for reasoning and deep neural networks with 3D generative representations of objects for robust visual recognition. Our experimental results show our model PO3D-VQA outperforms existing methods significantly, but we still observe a significant performance gap compared to 2D VQA benchmarks, indicating that 3D-aware VQA remains an important open research area.","url_abs":"https://arxiv.org/abs/2310.17914v1","url_pdf":"https://arxiv.org/pdf/2310.17914v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"3d-aware-visual-question-answering-about-1","repo_url":"https://github.com/xingruiwang/3d-aware-vqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"3d-aware-visual-question-answering-about-1","repo_url":"https://github.com/XingruiWang/superclevr-3D-question","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"mdetr","method_name":"MDETR"},{"method_slug":"po3d-vqa","method_name":"PO3D-VQA"}],"datasets_introduced":[{"slug":"super-clevr-3d","name":"Super-CLEVR-3D","full_name":"Super-CLEVR-3D"}],"methods_introduced":[{"slug":"po3d-vqa","name":"PO3D-VQA","full_name":"Parts, Poses, and Occlusions in 3D Visual Question Answering"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.17914","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.17914"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xingruiwang/3d-aware-vqa","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/XingruiWang/superclevr-3D-question","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_draft_wrong":1,"unverified":17},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1},"listed":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"b3c724d540d72cce","entry":"convert_bbox","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/align_objects.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/align_objects.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3c724d540d72cce"}},{"code_sha256_prefix":"84fe49b6fbbf5f07","entry":"direction_probabilities","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/align_objects_parts_prob.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/align_objects_parts_prob.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84fe49b6fbbf5f07"}},{"code_sha256_prefix":"46aeea7280866e13","entry":"extract_bboxes","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/post_process.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/post_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46aeea7280866e13"}},{"code_sha256_prefix":"2a2da93e76d3b6b1","entry":"filter_object","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/post_process.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/post_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a2da93e76d3b6b1"}},{"code_sha256_prefix":"b24784b77cae1db3","entry":"get_attrs_clevr","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/attr_net/utils.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/attr_net/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b24784b77cae1db3"}},{"code_sha256_prefix":"a080ed5b2ef7aedb","entry":"get_iou","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/align_objects.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/align_objects.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a080ed5b2ef7aedb"}},{"code_sha256_prefix":"bc0a3b6b3b7d1cde","entry":"get_overlap","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/post_process.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/post_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc0a3b6b3b7d1cde"}},{"code_sha256_prefix":"712891b9264da608","entry":"get_trainer","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/attr_net/trainer.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/attr_net/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"712891b9264da608"}},{"code_sha256_prefix":"e6dbe2d3b6f0cc87","entry":"invert_dict","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/attr_net/utils.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/attr_net/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6dbe2d3b6f0cc87"}},{"code_sha256_prefix":"57f66e40396ced61","entry":"one_hot","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/align_objects_parts_prob.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/align_objects_parts_prob.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57f66e40396ced61"}},{"code_sha256_prefix":"1778773a6330dd75","entry":"process_pose","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/6D_pose_estimator/align_objects.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/6D_pose_estimator/align_objects.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1778773a6330dd75"}},{"code_sha256_prefix":"fee3bf99805ee6cb","entry":"add_pose","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/add_occlusion_mask/add_occlusion.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/add_occlusion_mask/add_occlusion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fee3bf99805ee6cb"}},{"code_sha256_prefix":"24cf398747aafdc0","entry":"align","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"question_generation/align_removed.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/question_generation/align_removed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24cf398747aafdc0"}},{"code_sha256_prefix":"f9b74271607a3291","entry":"check_visibility","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/super_restore_render_images_split.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/super_restore_render_images_split.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9b74271607a3291"}},{"code_sha256_prefix":"6814863b0a1e72a9","entry":"compute_all_relationships","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/super_restore_render_images.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/super_restore_render_images.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6814863b0a1e72a9"}},{"code_sha256_prefix":"a3dbae9548681fea","entry":"extract_args","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/utils.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3dbae9548681fea"}},{"code_sha256_prefix":"c576aa00026275ff","entry":"get_candidates","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"question_generation/align_removed.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/question_generation/align_removed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c576aa00026275ff"}},{"code_sha256_prefix":"80e331ff8945a5c9","entry":"get_feat_vec_clevr","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/attr_net/utils.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/attr_net/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80e331ff8945a5c9"}},{"code_sha256_prefix":"05df8fe432cb08a8","entry":"get_mask","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/restore_img2json.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/restore_img2json.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05df8fe432cb08a8"}},{"code_sha256_prefix":"5a49cf40fd3eef79","entry":"get_model","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/attr_net/model-2.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/attr_net/model-2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a49cf40fd3eef79"}},{"code_sha256_prefix":"c23434d78dbe6b73","entry":"get_model","repo":"xingruiwang/3d-aware-vqa","repo_kind":"official","path":"PO3D-VQA/attr_net/model.py","file_url":"https://github.com/xingruiwang/3d-aware-vqa/blob/HEAD/PO3D-VQA/attr_net/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c23434d78dbe6b73"}},{"code_sha256_prefix":"a37183fd64f6eb8c","entry":"load_properties_json","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/preprocess_cgpart.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/preprocess_cgpart.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a37183fd64f6eb8c"}},{"code_sha256_prefix":"5487421f055b3459","entry":"parse_args","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/utils.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5487421f055b3459"}},{"code_sha256_prefix":"967a2489572384a0","entry":"render_shadeless","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/super_restore_render_images_split.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/super_restore_render_images_split.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"967a2489572384a0"}},{"code_sha256_prefix":"2b0321ad9cda9dcb","entry":"scene_handler","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"question_generation/question_engine.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/question_generation/question_engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b0321ad9cda9dcb"}},{"code_sha256_prefix":"7ad3633d3e96f006","entry":"srgb_to_linear","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/restore_img2json.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/restore_img2json.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ad3633d3e96f006"}},{"code_sha256_prefix":"a2ed7edd94a0f20c","entry":"to_str","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"image_generation/restore_img2json.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/image_generation/restore_img2json.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2ed7edd94a0f20c"}},{"code_sha256_prefix":"515aaf0b48a01dcd","entry":"unique_handler","repo":"XingruiWang/superclevr-3D-question","repo_kind":"listed","path":"question_generation/question_engine.py","file_url":"https://github.com/XingruiWang/superclevr-3D-question/blob/HEAD/question_generation/question_engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"515aaf0b48a01dcd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}