{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bridging-the-gap-between-2d-and-3d-visual","title":"Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA","arxiv_id":"2402.15933","date":"2024-02-24","proceeding":null,"authors":["Wentao Mo","Yang Liu"],"abstract":"In 3D Visual Question Answering (3D VQA), the scarcity of fully annotated data and limited visual content diversity hampers the generalization to novel scenes and 3D concepts (e.g., only around 800 scenes are utilized in ScanQA and SQA dataset). Current approaches resort supplement 3D reasoning with 2D information. However, these methods face challenges: either they use top-down 2D views that introduce overly complex and sometimes question-irrelevant visual clues, or they rely on globally aggregated scene/image-level representations from 2D VLMs, losing the fine-grained vision-language correlations. To overcome these limitations, our approach utilizes question-conditional 2D view selection procedure, pinpointing semantically relevant 2D inputs for crucial visual clues. We then integrate this 2D knowledge into the 3D-VQA system via a two-branch Transformer structure. This structure, featuring a Twin-Transformer design, compactly combines 2D and 3D modalities and captures fine-grained correlations between modalities, allowing them mutually augmenting each other. Integrating proposed mechanisms above, we present BridgeQA, that offers a fresh perspective on multi-modal transformer-based architectures for 3D-VQA. Experiments validate that BridgeQA achieves state-of-the-art on 3D-VQA datasets and significantly outperforms existing solutions. Code is available at $\\href{https://github.com/matthewdm0816/BridgeQA}{\\text{this URL}}$.","url_abs":"https://arxiv.org/abs/2402.15933v1","url_pdf":"https://arxiv.org/pdf/2402.15933v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bridging-the-gap-between-2d-and-3d-visual","repo_url":"https://github.com/matthewdm0816/bridgeqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"3d-question-answering-3d-qa","task_name":"3D Question Answering (3D-QA)"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"BridgeQA","rank_in_archive_order":1,"of":18,"metrics":{"BLEU-1":"34.49","BLEU-4":"24.06","CIDEr":"83.75","Exact Match":"31.29","METEOR":"16.51","ROUGE":"43.26"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.15933","atlas_url":"https://app.syntology.ai/?focus=2402.15933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.15933"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/matthewdm0816/bridgeqa","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":5,"ran_honours":1,"ran_draft_wrong":4,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"c1a3575f07494487","entry":"construct_bbox_corners","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/eval_helper.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/eval_helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c1a3575f07494487"}},{"code_sha256_prefix":"d209e4ea51338ac8","entry":"create_enet","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/enet.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/enet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d209e4ea51338ac8"}},{"code_sha256_prefix":"eddd23cbbb297813","entry":"flip_axis_to_camera","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/ap_helper.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/ap_helper.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eddd23cbbb297813"}},{"code_sha256_prefix":"1fa1f4c088979cd4","entry":"get_answer_score","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/dataset.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1fa1f4c088979cd4"}},{"code_sha256_prefix":"a0958108a7e9ed33","entry":"is_dummy","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"eval_scene_best_views.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/eval_scene_best_views.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a0958108a7e9ed33"}},{"code_sha256_prefix":"e1e4a142a595bcf2","entry":"l1_loss","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/loss.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/loss.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1e4a142a595bcf2"}},{"code_sha256_prefix":"a9ad469e43133558","entry":"prep_ans","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/qa_helper.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/qa_helper.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9ad469e43133558"}},{"code_sha256_prefix":"1f4e29c084d8fb7a","entry":"process_digit_article","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/qa_helper.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/qa_helper.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f4e29c084d8fb7a"}},{"code_sha256_prefix":"ad323281cc450ace","entry":"process_punctuation","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/qa_helper.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/qa_helper.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad323281cc450ace"}},{"code_sha256_prefix":"fe7609e5999f907d","entry":"smoothl1_loss","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/loss.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fe7609e5999f907d"}},{"code_sha256_prefix":"f9f6ab5de42cc2d4","entry":"create_enet_for_3d","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/enet.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/enet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f9f6ab5de42cc2d4"}},{"code_sha256_prefix":"d638b0201b9a9ece","entry":"flip_axis_to_depth","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/ap_helper.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/ap_helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d638b0201b9a9ece"}},{"code_sha256_prefix":"1a034a72c15f9903","entry":"softmax","repo":"matthewdm0816/bridgeqa","repo_kind":"official","path":"lib/ap_helper.py","file_url":"https://github.com/matthewdm0816/bridgeqa/blob/HEAD/lib/ap_helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a034a72c15f9903"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}