{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/viewrefer-grasp-the-multi-view-knowledge-for","title":"ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidance","arxiv_id":"2303.16894","date":"2023-03-29","proceeding":null,"authors":["Zoey Guo","Yiwen Tang","Ray Zhang","Dong Wang","Zhigang Wang","Bin Zhao","Xuelong Li"],"abstract":"Understanding 3D scenes from multi-view inputs has been proven to alleviate the view discrepancy issue in 3D visual grounding. However, existing methods normally neglect the view cues embedded in the text modality and fail to weigh the relative importance of different views. In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities. For the text branch, ViewRefer leverages the diverse linguistic knowledge of large-scale language models, e.g., GPT, to expand a single grounding text to multiple geometry-consistent descriptions. Meanwhile, in the 3D modality, a transformer fusion module with inter-view attention is introduced to boost the interaction of objects across views. On top of that, we further present a set of learnable multi-view prototypes, which memorize scene-agnostic knowledge for different views, and enhance the framework from two perspectives: a view-guided attention module for more robust text features, and a view-guided scoring strategy during the final prediction. With our designed paradigm, ViewRefer achieves superior performance on three benchmarks and surpasses the second-best by +2.8%, +1.5%, and +1.35% on Sr3D, Nr3D, and ScanRefer. Code is released at https://github.com/Ivan-Tang-3D/ViewRefer3D.","url_abs":"https://arxiv.org/abs/2303.16894v4","url_pdf":"https://arxiv.org/pdf/2303.16894v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"viewrefer-grasp-the-multi-view-knowledge-for","repo_url":"https://github.com/ivan-tang-3d/viewrefer3d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"viewrefer-grasp-the-multi-view-knowledge-for","repo_url":"https://github.com/ziyuguo99/viewrefer3d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"viewrefer-grasp-the-multi-view-knowledge-for","repo_url":"https://github.com/EvenJoker/Point-PEFT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"viewrefer-grasp-the-multi-view-knowledge-for","repo_url":"https://github.com/even-jk/peft-3d","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"viewrefer-grasp-the-multi-view-knowledge-for","repo_url":"https://github.com/ivan-tang-3d/any2point","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"viewrefer-grasp-the-multi-view-knowledge-for","repo_url":"https://github.com/ivan-tang-3d/peft-3d","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"viewrefer-grasp-the-multi-view-knowledge-for","repo_url":"https://github.com/ivan-tang-3d/point-peft","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-visual-grounding","task_name":"3D visual grounding"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.16894","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16894"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ivan-tang-3d/any2point","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ivan-tang-3d/peft-3d","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EvenJoker/Point-PEFT","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziyuguo99/viewrefer3d","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ivan-tang-3d/viewrefer3d","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ivan-tang-3d/point-peft","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/even-jk/peft-3d","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_draft_wrong":1,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1},"listed":{"samples":7,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"ac34f59de765420a","entry":"Block","repo":"ivan-tang-3d/point-peft","repo_kind":"listed","path":"MAE/models/Point_MAE_cp.py","file_url":"https://github.com/ivan-tang-3d/point-peft/blob/HEAD/MAE/models/Point_MAE_cp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac34f59de765420a"}},{"code_sha256_prefix":"2b00cd3626cdb9f2","entry":"Encoder","repo":"ivan-tang-3d/point-peft","repo_kind":"listed","path":"MAE/models/Point_MAE_cp.py","file_url":"https://github.com/ivan-tang-3d/point-peft/blob/HEAD/MAE/models/Point_MAE_cp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b00cd3626cdb9f2"}},{"code_sha256_prefix":"5ccd1714b5d56d00","entry":"MLP","repo":"ivan-tang-3d/viewrefer3d","repo_kind":"official","path":"referit3d/models/referit3d_net.py","file_url":"https://github.com/ivan-tang-3d/viewrefer3d/blob/HEAD/referit3d/models/referit3d_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ccd1714b5d56d00"}},{"code_sha256_prefix":"0f53e769c956656b","entry":"TransformerDecoderLayer_","repo":"ivan-tang-3d/viewrefer3d","repo_kind":"official","path":"referit3d/models/referit3d_net.py","file_url":"https://github.com/ivan-tang-3d/viewrefer3d/blob/HEAD/referit3d/models/referit3d_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f53e769c956656b"}},{"code_sha256_prefix":"182c25ba5bdfe2d5","entry":"TransformerEncoder","repo":"ivan-tang-3d/point-peft","repo_kind":"listed","path":"MAE/models/Point_MAE_cp.py","file_url":"https://github.com/ivan-tang-3d/point-peft/blob/HEAD/MAE/models/Point_MAE_cp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"182c25ba5bdfe2d5"}},{"code_sha256_prefix":"6a5e7a1ee115ebdd","entry":"_get_clones","repo":"ivan-tang-3d/viewrefer3d","repo_kind":"official","path":"referit3d/models/referit3d_net.py","file_url":"https://github.com/ivan-tang-3d/viewrefer3d/blob/HEAD/referit3d/models/referit3d_net.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a5e7a1ee115ebdd"}},{"code_sha256_prefix":"fc7209d0a815505b","entry":"_translate","repo":"ivan-tang-3d/viewrefer3d","repo_kind":"official","path":"referit3d/models/referit3d_net.py","file_url":"https://github.com/ivan-tang-3d/viewrefer3d/blob/HEAD/referit3d/models/referit3d_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc7209d0a815505b"}},{"code_sha256_prefix":"140122b970061273","entry":"euler2mat","repo":"ivan-tang-3d/any2point","repo_kind":"listed","path":"Any2Point_CLIP_Lang/models/peft_module/mv_utils.py","file_url":"https://github.com/ivan-tang-3d/any2point/blob/HEAD/Any2Point_CLIP_Lang/models/peft_module/mv_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"140122b970061273"}},{"code_sha256_prefix":"36c0073a9f252f29","entry":"MaskTransformer","repo":"ivan-tang-3d/point-peft","repo_kind":"listed","path":"MAE/models/Point_MAE_cp.py","file_url":"https://github.com/ivan-tang-3d/point-peft/blob/HEAD/MAE/models/Point_MAE_cp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36c0073a9f252f29"}},{"code_sha256_prefix":"0e13f074f4c5db2f","entry":"PCViews","repo":"ivan-tang-3d/any2point","repo_kind":"listed","path":"Any2Point_CLIP_Lang/models/peft_module/mv_utils.py","file_url":"https://github.com/ivan-tang-3d/any2point/blob/HEAD/Any2Point_CLIP_Lang/models/peft_module/mv_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e13f074f4c5db2f"}},{"code_sha256_prefix":"a417a1744fad155d","entry":"ReferIt3DNet_transformer","repo":"ivan-tang-3d/viewrefer3d","repo_kind":"official","path":"referit3d/models/referit3d_net.py","file_url":"https://github.com/ivan-tang-3d/viewrefer3d/blob/HEAD/referit3d/models/referit3d_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a417a1744fad155d"}},{"code_sha256_prefix":"5bf636008d86d226","entry":"points2pos","repo":"ivan-tang-3d/any2point","repo_kind":"listed","path":"Any2Point_CLIP_Lang/models/peft_module/mv_utils.py","file_url":"https://github.com/ivan-tang-3d/any2point/blob/HEAD/Any2Point_CLIP_Lang/models/peft_module/mv_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bf636008d86d226"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}