{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scan2cap-context-aware-dense-captioning-in","title":"Scan2Cap: Context-aware Dense Captioning in RGB-D Scans","arxiv_id":"2012.02206","date":"2020-12-03","proceeding":"CVPR 2021 1","authors":["Dave Zhenyu Chen","Ali Gholami","Matthias Nießner","Angel X. Chang"],"abstract":"We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors. As input, we assume a point cloud of a 3D scene; the expected output is the bounding boxes along with the descriptions for the underlying objects. To address the 3D object detection and description problems, we propose Scan2Cap, an end-to-end trained method, to detect objects in the input scene and describe them in natural language. We use an attention mechanism that generates descriptive tokens while referring to the related components in the local context. To reflect object relations (i.e. relative spatial relations) in the generated captions, we use a message passing graph module to facilitate learning object relation features. Our method can effectively localize and describe 3D objects in scenes from the ScanRefer dataset, outperforming 2D baseline methods by a significant margin (27.61% CiDEr@0.5IoUimprovement).","url_abs":"https://arxiv.org/abs/2012.02206v1","url_pdf":"https://arxiv.org/pdf/2012.02206v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"3d-question-answering-3d-qa","task_name":"3D Question Answering (3D-QA)"},{"task_slug":"3d-dense-captioning","task_name":"3D dense captioning"},{"task_slug":"dense-captioning","task_name":"Dense Captioning"},{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-question-answering-3d-qa-on-sqa3d","task":"3D Question Answering (3D-QA)","dataset":"SQA3D","model":"Scan2Cap","rank_in_archive_order":11,"of":13,"metrics":{"Exact Match":"41.0"},"uses_additional_data":false},{"leaderboard":"/sota/3d-dense-captioning-on-nr3d","task":"3D dense captioning","dataset":"Nr3D","model":"Scan2Cap","rank_in_archive_order":10,"of":10,"metrics":{"BLEU-4":"17.24","CIDEr":"27.47","METEOR":"21.80","ROUGE-L":"49.06"},"uses_additional_data":false},{"leaderboard":"/sota/3d-dense-captioning-on-scanrefer-dataset","task":"3D dense captioning","dataset":"ScanRefer Dataset","model":"Scan2Cap","rank_in_archive_order":9,"of":12,"metrics":{"BLEU-4":"34.25","CIDEr":"53.73","METEOR":"26.14","ROUGE-L":"54.95"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2012.02206","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}