{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/remembr-building-and-reasoning-over-long","title":"ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation","arxiv_id":"2409.13682","date":"2024-09-20","proceeding":null,"authors":["Abrar Anwar","John Welsh","Joydeep Biswas","Soha Pouya","Yan Chang"],"abstract":"Navigating and understanding complex environments over extended periods of time is a significant challenge for robots. People interacting with the robot may want to ask questions like where something happened, when it occurred, or how long ago it took place, which would require the robot to reason over a long history of their deployment. To address this problem, we introduce a Retrieval-augmented Memory for Embodied Robots, or ReMEmbR, a system designed for long-horizon video question answering for robot navigation. To evaluate ReMEmbR, we introduce the NaVQA dataset where we annotate spatial, temporal, and descriptive questions to long-horizon robot navigation videos. ReMEmbR employs a structured approach involving a memory building and a querying phase, leveraging temporal information, spatial information, and images to efficiently handle continuously growing robot histories. Our experiments demonstrate that ReMEmbR outperforms LLM and VLM baselines, allowing ReMEmbR to achieve effective long-horizon reasoning with low latency. Additionally, we deploy ReMEmbR on a robot and show that our approach can handle diverse queries. The dataset, code, videos, and other material can be found at the following link: https://nvidia-ai-iot.github.io/remembr","url_abs":"https://arxiv.org/abs/2409.13682v1","url_pdf":"https://arxiv.org/pdf/2409.13682v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"remembr-building-and-reasoning-over-long","repo_url":"https://github.com/NVIDIA-AI-IOT/remembr","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"robot-navigation","task_name":"Robot Navigation"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.13682","atlas_url":"https://app.syntology.ai/?focus=2409.13682","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.13682"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVIDIA-AI-IOT/remembr","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"51638e4706c299f9","entry":"get_frames","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/utils/util.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"51638e4706c299f9"}},{"code_sha256_prefix":"a7bee88c1c7fd6a3","entry":"image_parser","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/captioners/vila_captioner.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/captioners/vila_captioner.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a7bee88c1c7fd6a3"}},{"code_sha256_prefix":"2afd732da80b1c00","entry":"inspect","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/agents/remembr_agent.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/agents/remembr_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2afd732da80b1c00"}},{"code_sha256_prefix":"20b5922e237d1de7","entry":"load_image","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/captioners/vila_captioner.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/captioners/vila_captioner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"20b5922e237d1de7"}},{"code_sha256_prefix":"7f2d6adb6e2cddf2","entry":"load_images","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/captioners/vila_captioner.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/captioners/vila_captioner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7f2d6adb6e2cddf2"}},{"code_sha256_prefix":"6414d8dbc4b69fa3","entry":"np_image_to_base64","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/agents/vlm_non_agent.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/agents/vlm_non_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6414d8dbc4b69fa3"}},{"code_sha256_prefix":"d221abac9d966ccf","entry":"similarity_search_with_score_by_vector","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/memory/milvus_memory.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/memory/milvus_memory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d221abac9d966ccf"}},{"code_sha256_prefix":"acd3a0b23e28c84a","entry":"file_to_string","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/utils/util.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"acd3a0b23e28c84a"}},{"code_sha256_prefix":"6785d3e60dfb7ca5","entry":"parse_json","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/agents/non_agent.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/agents/non_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6785d3e60dfb7ca5"}},{"code_sha256_prefix":"383192b3ef21d786","entry":"parse_json","repo":"NVIDIA-AI-IOT/remembr","repo_kind":"official","path":"remembr/agents/remembr_agent.py","file_url":"https://github.com/NVIDIA-AI-IOT/remembr/blob/HEAD/remembr/agents/remembr_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"383192b3ef21d786"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}