{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-foundation-models-for-3d-vision-how","title":"Towards Foundation Models for 3D Vision: How Close Are We?","arxiv_id":"2410.10799","date":"2024-10-14","proceeding":null,"authors":["Yiming Zuo","Karhan Kayan","Maggie Wang","Kevin Jeon","Jia Deng","Thomas L. Griffiths"],"abstract":"Building a foundation model for 3D vision is a complex challenge that remains unsolved. Towards that goal, it is important to understand the 3D reasoning capabilities of current models as well as identify the gaps between these models and humans. Therefore, we construct a new 3D visual understanding benchmark named UniQA-3D. UniQA-3D covers fundamental 3D vision tasks in the Visual Question Answering (VQA) format. We evaluate state-of-the-art Vision-Language Models (VLMs), specialized models, and human subjects on it. Our results show that VLMs generally perform poorly, while the specialized models are accurate but not robust, failing under geometric perturbations. In contrast, human vision continues to be the most reliable 3D visual system. We further demonstrate that neural networks align more closely with human 3D vision mechanisms compared to classical computer vision methods, and Transformer-based networks such as ViT align more closely with human 3D vision mechanisms than CNNs. We hope our study will benefit the future development of foundation models for 3D vision. Code is available at https://github.com/princeton-vl/UniQA-3D .","url_abs":"https://arxiv.org/abs/2410.10799v2","url_pdf":"https://arxiv.org/pdf/2410.10799v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-foundation-models-for-3d-vision-how","repo_url":"https://github.com/princeton-vl/uniqa-3d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"towards-foundation-models-for-3d-vision-how","repo_url":"https://github.com/princeton-vl/Princeton365","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.10799","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.10799"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-vl/uniqa-3d","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-vl/Princeton365","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":9,"ran_draft_wrong":1,"unverified":19},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1},"listed":{"samples":16,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16822eba1e76ac6c","entry":"calculate_acc","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/relative_camera_pose/evaluate_mturk.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/relative_camera_pose/evaluate_mturk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"16822eba1e76ac6c"}},{"code_sha256_prefix":"c3c486bf7368e6dd","entry":"compare_answers","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/clevr_vqa/utils.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/clevr_vqa/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c3c486bf7368e6dd"}},{"code_sha256_prefix":"26f9efe7accdcfe5","entry":"convert_depth_click_raw","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/relative_camera_pose/evaluate_mturk.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/relative_camera_pose/evaluate_mturk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"26f9efe7accdcfe5"}},{"code_sha256_prefix":"b864e6061f32b2a5","entry":"create_bins","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/clevr_vqa/utils.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/clevr_vqa/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b864e6061f32b2a5"}},{"code_sha256_prefix":"f41cb1a19b154297","entry":"encode_image","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"LLM_evaluations/clevr_vqa/generate_gpt_response.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/LLM_evaluations/clevr_vqa/generate_gpt_response.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f41cb1a19b154297"}},{"code_sha256_prefix":"c6596d9a9fde47de","entry":"find_bin_number","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/clevr_vqa/utils.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/clevr_vqa/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c6596d9a9fde47de"}},{"code_sha256_prefix":"a708f802b11577e8","entry":"get_answer_from_depth","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/relative_depth/utils.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/relative_depth/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a708f802b11577e8"}},{"code_sha256_prefix":"fd604ee0fb02592a","entry":"inside_box","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/relative_camera_pose/evaluate_mturk.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/relative_camera_pose/evaluate_mturk.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fd604ee0fb02592a"}},{"code_sha256_prefix":"b7b4d1f83d4d65fd","entry":"read_pfm","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/relative_depth/utils.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/relative_depth/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b7b4d1f83d4d65fd"}},{"code_sha256_prefix":"e201c2c212680682","entry":"retrieve_metadata","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"Human_Study/relative_depth/utils.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/Human_Study/relative_depth/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e201c2c212680682"}},{"code_sha256_prefix":"dd80736bb1d636c2","entry":"calculate_bic","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_depth.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_depth.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"dd80736bb1d636c2"}},{"code_sha256_prefix":"8c191d61503f8e12","entry":"calculate_flow_auc","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/evaluation.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8c191d61503f8e12"}},{"code_sha256_prefix":"d8ce07490601448a","entry":"compress_folder","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/upload_submission.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/upload_submission.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d8ce07490601448a"}},{"code_sha256_prefix":"16d8cfdce0ca9089","entry":"create_board","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/board_generator.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/board_generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"16d8cfdce0ca9089"}},{"code_sha256_prefix":"5b4e7fee01cf2b32","entry":"detect_pose","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_aruco.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_aruco.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5b4e7fee01cf2b32"}},{"code_sha256_prefix":"b4a96fd355a29c5b","entry":"estimate_relative_pose","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_aruco.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_aruco.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b4a96fd355a29c5b"}},{"code_sha256_prefix":"f9cb044d350d4f6f","entry":"generate_charuco_boards","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/board_generator.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/board_generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f9cb044d350d4f6f"}},{"code_sha256_prefix":"f609b882d3318e8a","entry":"get_response","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"LLM_evaluations/clevr_vqa/generate_gpt_response.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/LLM_evaluations/clevr_vqa/generate_gpt_response.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f609b882d3318e8a"}},{"code_sha256_prefix":"864f8f7c8a83f0b3","entry":"get_response","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"LLM_evaluations/relative_camera_pose/generate_gpt4v_response.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/LLM_evaluations/relative_camera_pose/generate_gpt4v_response.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"864f8f7c8a83f0b3"}},{"code_sha256_prefix":"42bf1823205a9ad2","entry":"get_response","repo":"princeton-vl/uniqa-3d","repo_kind":"official","path":"LLM_evaluations/relative_depth/generate_gpt4v_response.py","file_url":"https://github.com/princeton-vl/uniqa-3d/blob/HEAD/LLM_evaluations/relative_depth/generate_gpt4v_response.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"42bf1823205a9ad2"}},{"code_sha256_prefix":"624504bd4df1f096","entry":"get_result_save_path","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_io.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"624504bd4df1f096"}},{"code_sha256_prefix":"ea1138983c9e7b12","entry":"median_edge_selection","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_graph.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_graph.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ea1138983c9e7b12"}},{"code_sha256_prefix":"5566a27037c99eb4","entry":"multiboard_pnp","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_aruco.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_aruco.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5566a27037c99eb4"}},{"code_sha256_prefix":"42b0f493d1cb2f37","entry":"numpy_to_list","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_io.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"42b0f493d1cb2f37"}},{"code_sha256_prefix":"8e1c7e85e2a677ba","entry":"preprocess_depth_frame","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_depth.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_depth.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8e1c7e85e2a677ba"}},{"code_sha256_prefix":"cca70d6ea8336990","entry":"read_rotation_from_json","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_io.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cca70d6ea8336990"}},{"code_sha256_prefix":"201e1778250b09d6","entry":"read_zed_depth","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_depth.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_depth.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"201e1778250b09d6"}},{"code_sha256_prefix":"9211482a65400287","entry":"snap_coordinate_systems","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_graph.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_graph.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9211482a65400287"}},{"code_sha256_prefix":"8eee5df111605343","entry":"update_edge_transformations","repo":"princeton-vl/Princeton365","repo_kind":"listed","path":"princeton365/utils/utils_graph.py","file_url":"https://github.com/princeton-vl/Princeton365/blob/HEAD/princeton365/utils/utils_graph.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8eee5df111605343"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}