{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tod3cap-towards-3d-dense-captioning-in","title":"TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes","arxiv_id":"2403.19589","date":"2024-03-28","proceeding":null,"authors":["Bu Jin","Yupeng Zheng","Pengfei Li","Weize Li","Yuhang Zheng","Sujie Hu","Xinyu Liu","Jinwei Zhu","Zhijie Yan","Haiyang Sun","Kun Zhan","Peng Jia","Xiaoxiao Long","Yilun Chen","Hao Zhao"],"abstract":"3D dense captioning stands as a cornerstone in achieving a comprehensive understanding of 3D scenes through natural language. It has recently witnessed remarkable achievements, particularly in indoor settings. However, the exploration of 3D dense captioning in outdoor scenes is hindered by two major challenges: 1) the domain gap between indoor and outdoor scenes, such as dynamics and sparse visual inputs, makes it difficult to directly adapt existing indoor methods; 2) the lack of data with comprehensive box-caption pair annotations specifically tailored for outdoor scenes. To this end, we introduce the new task of outdoor 3D dense captioning. As input, we assume a LiDAR point cloud and a set of RGB images captured by the panoramic camera rig. The expected output is a set of object boxes with captions. To tackle this task, we propose the TOD3Cap network, which leverages the BEV representation to generate object box proposals and integrates Relation Q-Former with LLaMA-Adapter to generate rich captions for these objects. We also introduce the TOD3Cap dataset, the largest one to our knowledge for 3D dense captioning in outdoor scenes, which contains 2.3M descriptions of 64.3K outdoor objects from 850 scenes. Notably, our TOD3Cap network can effectively localize and caption 3D objects in outdoor scenes, which outperforms baseline methods by a significant margin (+9.6 CiDEr@0.5IoU). Code, data, and models are publicly available at https://github.com/jxbbb/TOD3Cap.","url_abs":"https://arxiv.org/abs/2403.19589v2","url_pdf":"https://arxiv.org/pdf/2403.19589v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tod3cap-towards-3d-dense-captioning-in","repo_url":"https://github.com/jxbbb/tod3cap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-dense-captioning","task_name":"3D dense captioning"},{"task_slug":"dense-captioning","task_name":"Dense Captioning"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.19589","atlas_url":"https://app.syntology.ai/?focus=2403.19589","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.19589"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jxbbb/tod3cap","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran":5,"ran_fixture":2,"ran_violates":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":16,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"bd3578806ecf5ac2","entry":"adjust_learning_rate","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/util/lr_sched.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/util/lr_sched.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd3578806ecf5ac2"}},{"code_sha256_prefix":"fdb1265b615d25b9","entry":"all_reduce_mean","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/util/misc.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdb1265b615d25b9"}},{"code_sha256_prefix":"b47d48e431b34acd","entry":"apply_rotary_emb","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/llama.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b47d48e431b34acd"}},{"code_sha256_prefix":"7006a7479b6f348f","entry":"format_prompt","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/utils.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7006a7479b6f348f"}},{"code_sha256_prefix":"de80f1711a0ddf21","entry":"position_embedding","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/GPT2.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/GPT2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de80f1711a0ddf21"}},{"code_sha256_prefix":"a247fcd78818b192","entry":"position_embedding","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/llama_adapter_gpt.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/llama_adapter_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a247fcd78818b192"}},{"code_sha256_prefix":"14a84c2cbfebc413","entry":"precompute_freqs_cis","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/llama.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/llama.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14a84c2cbfebc413"}},{"code_sha256_prefix":"e76a491553cf9fcf","entry":"process_checkpoint","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/mmdetection3d/.dev_scripts/gather_models.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/mmdetection3d/.dev_scripts/gather_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e76a491553cf9fcf"}},{"code_sha256_prefix":"70bf6ebaafd266c4","entry":"reshape_for_broadcast","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/llama.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70bf6ebaafd266c4"}},{"code_sha256_prefix":"8845976729f4c4ee","entry":"sample_top_p","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/utils.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8845976729f4c4ee"}},{"code_sha256_prefix":"b40701674badc944","entry":"add_weight_decay","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/util/misc.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/util/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b40701674badc944"}},{"code_sha256_prefix":"90f6ae7209cde36f","entry":"get_final_epoch","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/mmdetection3d/.dev_scripts/gather_models.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/mmdetection3d/.dev_scripts/gather_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90f6ae7209cde36f"}},{"code_sha256_prefix":"da059252c88ecf40","entry":"get_model_dataset","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/mmdetection3d/.dev_scripts/gather_models.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/mmdetection3d/.dev_scripts/gather_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da059252c88ecf40"}},{"code_sha256_prefix":"825615e23fa39149","entry":"load","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/llama_adapter.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/llama_adapter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"825615e23fa39149"}},{"code_sha256_prefix":"16bd604b0433607b","entry":"load","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/llama_adapter_back.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/llama_adapter_back.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16bd604b0433607b"}},{"code_sha256_prefix":"ca22408e380b541a","entry":"token_llama2gpt2","repo":"jxbbb/tod3cap","repo_kind":"official","path":"tod3cap_camera/llama/GPT2.py","file_url":"https://github.com/jxbbb/tod3cap/blob/HEAD/tod3cap_camera/llama/GPT2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca22408e380b541a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}