{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/graspcot-integrating-physical-property","title":"GraspCoT: Integrating Physical Property Reasoning for 6-DoF Grasping under Flexible Language Instructions","arxiv_id":"2503.16013","date":"2025-03-20","proceeding":null,"authors":["Xiaomeng Chu","Jiajun Deng","Guoliang You","Wei Liu","Xingchen Li","Jianmin Ji","Yanyong Zhang"],"abstract":"Flexible instruction-guided 6-DoF grasping is a significant yet challenging task for real-world robotic systems. Existing methods utilize the contextual understanding capabilities of the large language models (LLMs) to establish mappings between expressions and targets, allowing robots to comprehend users' intentions in the instructions. However, the LLM's knowledge about objects' physical properties remains underexplored despite its tight relevance to grasping. In this work, we propose GraspCoT, a 6-DoF grasp detection framework that integrates a Chain-of-Thought (CoT) reasoning mechanism oriented to physical properties, guided by auxiliary question-answering (QA) tasks. Particularly, we design a set of QA templates to enable hierarchical reasoning that includes three stages: target parsing, physical property analysis, and grasp action selection. Moreover, GraspCoT presents a unified multimodal LLM architecture, which encodes multi-view observations of 3D scenes into 3D-aware visual tokens, and then jointly embeds these visual tokens with CoT-derived textual tokens within LLMs to generate grasp pose predictions. Furthermore, we present IntentGrasp, a large-scale benchmark that fills the gap in public datasets for multi-object grasp detection under diverse and indirect verbal commands. Extensive experiments on IntentGrasp demonstrate the superiority of our method, with additional validation in real-world robotic applications confirming its practicality. Codes and data will be released.","url_abs":"https://arxiv.org/abs/2503.16013v1","url_pdf":"https://arxiv.org/pdf/2503.16013v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.16013","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.16013"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/cxmomo/GraspCoT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_fixture":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/mm_utils.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"4f25740ddf1557a5","entry":"fp16_clamp","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/model/match_cost.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/model/match_cost.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f25740ddf1557a5"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/mm_utils.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/mm_utils.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"7606525af238fb64","entry":"unpad_image","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/model/llava_arch.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/model/llava_arch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7606525af238fb64"}},{"code_sha256_prefix":"c36f5853e184effd","entry":"bbox_overlaps","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/model/match_cost.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/model/match_cost.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c36f5853e184effd"}},{"code_sha256_prefix":"22d8dc932e8b7892","entry":"denormalize_grasp","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/model/nms_free_coder.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/model/nms_free_coder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22d8dc932e8b7892"}},{"code_sha256_prefix":"86069c4dc82688fe","entry":"normalize_bbox","repo":"cxmomo/GraspCoT","repo_kind":"found_in_text","path":"llava/model/hungarian_assigner_3d.py","file_url":"https://github.com/cxmomo/GraspCoT/blob/HEAD/llava/model/hungarian_assigner_3d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86069c4dc82688fe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}