{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/view-selection-for-3d-captioning-via","title":"View Selection for 3D Captioning via Diffusion Ranking","arxiv_id":"2404.07984","date":"2024-04-11","proceeding":null,"authors":["Tiange Luo","Justin Johnson","Honglak Lee"],"abstract":"Scalable annotation approaches are crucial for constructing extensive 3D-text datasets, facilitating a broader range of applications. However, existing methods sometimes lead to the generation of hallucinated captions, compromising caption quality. This paper explores the issue of hallucination in 3D object captioning, with a focus on Cap3D method, which renders 3D objects into 2D views for captioning using pre-trained models. We pinpoint a major challenge: certain rendered views of 3D objects are atypical, deviating from the training data of standard image captioning models and causing hallucinations. To tackle this, we present DiffuRank, a method that leverages a pre-trained text-to-3D model to assess the alignment between 3D objects and their 2D rendered views, where the view with high alignment closely represent the object's characteristics. By ranking all rendered views and feeding the top-ranked ones into GPT4-Vision, we enhance the accuracy and detail of captions, enabling the correction of 200k captions in the Cap3D dataset and extending it to 1 million captions across Objaverse and Objaverse-XL datasets. Additionally, we showcase the adaptability of DiffuRank by applying it to pre-trained text-to-image models for a Visual Question Answering task, where it outperforms the CLIP model.","url_abs":"https://arxiv.org/abs/2404.07984v1","url_pdf":"https://arxiv.org/pdf/2404.07984v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"view-selection-for-3d-captioning-via","repo_url":"https://github.com/chenguolin/DiffSplat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"view-selection-for-3d-captioning-via","repo_url":"https://github.com/crockwell/cap3d","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-object-captioning","task_name":"3D Object Captioning"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-to-3d","task_name":"Text to 3D"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.07984","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.07984"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/crockwell/cap3d","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chenguolin/DiffSplat","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"7b9471aa6c1ba723","entry":"load_npz_file","repo":"crockwell/cap3d","repo_kind":"listed","path":"text-to-3D/shap-e/npz2ply.py","file_url":"https://github.com/crockwell/cap3d/blob/HEAD/text-to-3D/shap-e/npz2ply.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b9471aa6c1ba723"}},{"code_sha256_prefix":"92a6a02dabc5cc0e","entry":"summarize_captions_gpt35","repo":"crockwell/cap3d","repo_kind":"listed","path":"captioning_pipeline/caption_clip_gpt.py","file_url":"https://github.com/crockwell/cap3d/blob/HEAD/captioning_pipeline/caption_clip_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92a6a02dabc5cc0e"}},{"code_sha256_prefix":"ab4e0aab713b00cc","entry":"summarize_captions_gpt4","repo":"crockwell/cap3d","repo_kind":"listed","path":"captioning_pipeline/caption_clip_gpt.py","file_url":"https://github.com/crockwell/cap3d/blob/HEAD/captioning_pipeline/caption_clip_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab4e0aab713b00cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}