{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/situational-awareness-matters-in-3d-vision","title":"Situational Awareness Matters in 3D Vision Language Reasoning","arxiv_id":"2406.07544","date":"2024-06-11","proceeding":"CVPR 2024 1","authors":["Yunze Man","Liang-Yan Gui","Yu-Xiong Wang"],"abstract":"Being able to carry out complicated vision language reasoning tasks in 3D space represents a significant milestone in developing household robots and human-centered embodied AI. In this work, we demonstrate that a critical and distinct challenge in 3D vision language reasoning is situational awareness, which incorporates two key components: (1) The autonomous agent grounds its self-location based on a language prompt. (2) The agent answers open-ended questions from the perspective of its calculated position. To address this challenge, we introduce SIG3D, an end-to-end Situation-Grounded model for 3D vision language reasoning. We tokenize the 3D scene into sparse voxel representation and propose a language-grounded situation estimator, followed by a situated question answering module. Experiments on the SQA3D and ScanQA datasets show that SIG3D outperforms state-of-the-art models in situation estimation and question answering by a large margin (e.g., an enhancement of over 30% on situation estimation accuracy). Subsequent analysis corroborates our architectural design choices, explores the distinct functions of visual and textual tokens, and highlights the importance of situational awareness in the domain of 3D question answering.","url_abs":"https://arxiv.org/abs/2406.07544v2","url_pdf":"https://arxiv.org/pdf/2406.07544v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"situational-awareness-matters-in-3d-vision","repo_url":"https://github.com/YunzeMan/Situation3D","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-sqa3d","task":"Question Answering","dataset":"SQA3D","model":"Situation3D","rank_in_archive_order":2,"of":7,"metrics":{"AnswerExactMatch (Question Answering)":"52.6"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.07544","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07544"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YunzeMan/Situation3D","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"ran_fixture":1},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f2988c8d779e42f8","entry":"batch_rotation_vector_to_matrix","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/models/sqa_module.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/models/sqa_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2988c8d779e42f8"}},{"code_sha256_prefix":"86a87fc28c57c91f","entry":"decode_eta","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/eta.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/eta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86a87fc28c57c91f"}},{"code_sha256_prefix":"e6aaf158a822877c","entry":"get_eta","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/eta.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/eta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6aaf158a822877c"}},{"code_sha256_prefix":"e5929ca970405ef7","entry":"poly_area","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/box_util.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/box_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5929ca970405ef7"}},{"code_sha256_prefix":"4aa9e3769f269306","entry":"poly_area_batch","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/box_util.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/box_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4aa9e3769f269306"}},{"code_sha256_prefix":"05b20b55c5918c4a","entry":"polygon_clip","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/box_util.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/box_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05b20b55c5918c4a"}},{"code_sha256_prefix":"f1fe1fca70db75e3","entry":"quaternions_to_rotation_matrices","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/models/sqa_module.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/models/sqa_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1fe1fca70db75e3"}},{"code_sha256_prefix":"25e18849d43559c1","entry":"setup_logger","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/logger.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25e18849d43559c1"}},{"code_sha256_prefix":"d474ca8ac6d98398","entry":"sqa3d_question_type","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/language_util.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/language_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d474ca8ac6d98398"}},{"code_sha256_prefix":"95c31c78be42f723","entry":"voc_ap","repo":"YunzeMan/Situation3D","repo_kind":"official","path":"situation3d/utils/eval_det.py","file_url":"https://github.com/YunzeMan/Situation3D/blob/HEAD/situation3d/utils/eval_det.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95c31c78be42f723"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}