{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/details-make-a-difference-object-state","title":"Details Make a Difference: Object State-Sensitive Neurorobotic Task Planning","arxiv_id":"2406.09988","date":"2024-06-14","proceeding":null,"authors":["Xiaowen Sun","Xufeng Zhao","Jae Hee Lee","Wenhao Lu","Matthias Kerzel","Stefan Wermter"],"abstract":"The state of an object reflects its current status or condition and is important for a robot's task planning and manipulation. However, detecting an object's state and generating a state-sensitive plan for robots is challenging. Recently, pre-trained Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown impressive capabilities in generating plans. However, to the best of our knowledge, there is hardly any investigation on whether LLMs or VLMs can also generate object state-sensitive plans. To study this, we introduce an Object State-Sensitive Agent (OSSA), a task-planning agent empowered by pre-trained neural networks. We propose two methods for OSSA: (i) a modular model consisting of a pre-trained vision processing module (dense captioning model, DCM) and a natural language processing model (LLM), and (ii) a monolithic model consisting only of a VLM. To quantitatively evaluate the performances of the two methods, we use tabletop scenarios where the task is to clear the table. We contribute a multimodal benchmark dataset that takes object states into consideration. Our results show that both methods can be used for object state-sensitive tasks, but the monolithic approach outperforms the modular approach. The code for OSSA is available at https://github.com/Xiao-wen-Sun/OSSA","url_abs":"https://arxiv.org/abs/2406.09988v2","url_pdf":"https://arxiv.org/pdf/2406.09988v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"details-make-a-difference-object-state","repo_url":"https://github.com/xiao-wen-sun/ossa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"dense-captioning","task_name":"Dense Captioning"},{"task_slug":"object","task_name":"Object"},{"task_slug":"task-planning","task_name":"Task Planning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.09988","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.09988"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiao-wen-sun/ossa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e9b355b1c2a3ff0d","entry":"get_deep_speed_config","repo":"xiao-wen-sun/ossa","repo_kind":"official","path":"grit/train_deepspeed.py","file_url":"https://github.com/xiao-wen-sun/ossa/blob/HEAD/grit/train_deepspeed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9b355b1c2a3ff0d"}},{"code_sha256_prefix":"edb9869f131321d2","entry":"get_rel_pos","repo":"xiao-wen-sun/ossa","repo_kind":"official","path":"grit/grit/modeling/backbone/utils.py","file_url":"https://github.com/xiao-wen-sun/ossa/blob/HEAD/grit/grit/modeling/backbone/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"edb9869f131321d2"}},{"code_sha256_prefix":"08bf7ecdb2fffe84","entry":"get_vit_lr_decay_rate","repo":"xiao-wen-sun/ossa","repo_kind":"official","path":"grit/grit/custom_solver.py","file_url":"https://github.com/xiao-wen-sun/ossa/blob/HEAD/grit/grit/custom_solver.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08bf7ecdb2fffe84"}},{"code_sha256_prefix":"750facb8c52e8864","entry":"window_partition","repo":"xiao-wen-sun/ossa","repo_kind":"official","path":"grit/grit/modeling/backbone/utils.py","file_url":"https://github.com/xiao-wen-sun/ossa/blob/HEAD/grit/grit/modeling/backbone/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"750facb8c52e8864"}},{"code_sha256_prefix":"535942f820542fb9","entry":"window_unpartition","repo":"xiao-wen-sun/ossa","repo_kind":"official","path":"grit/grit/modeling/backbone/utils.py","file_url":"https://github.com/xiao-wen-sun/ossa/blob/HEAD/grit/grit/modeling/backbone/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"535942f820542fb9"}},{"code_sha256_prefix":"1e90c2ddcf6921e2","entry":"encode_image","repo":"xiao-wen-sun/ossa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/xiao-wen-sun/ossa/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e90c2ddcf6921e2"}},{"code_sha256_prefix":"4dfe8592fa13dca7","entry":"get_image_files","repo":"xiao-wen-sun/ossa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/xiao-wen-sun/ossa/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4dfe8592fa13dca7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}