{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memory-benchmark-robots-a-benchmark-for","title":"Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning","arxiv_id":"2502.10550","date":"2025-02-14","proceeding":null,"authors":["Egor Cherepanov","Nikita Kachaev","Alexey K. Kovalev","Aleksandr I. Panov"],"abstract":"Memory is crucial for enabling agents to tackle complex tasks with temporal and spatial dependencies. While many reinforcement learning (RL) algorithms incorporate memory, the field lacks a universal benchmark to assess an agent's memory capabilities across diverse scenarios. This gap is particularly evident in tabletop robotic manipulation, where memory is essential for solving tasks with partial observability and ensuring robust performance, yet no standardized benchmarks exist. To address this, we introduce MIKASA (Memory-Intensive Skills Assessment Suite for Agents), a comprehensive benchmark for memory RL, with three key contributions: (1) we propose a comprehensive classification framework for memory-intensive RL tasks, (2) we collect MIKASA-Base -- a unified benchmark that enables systematic evaluation of memory-enhanced agents across diverse scenarios, and (3) we develop MIKASA-Robo (pip install mikasa-robo-suite) -- a novel benchmark of 32 carefully designed memory-intensive tasks that assess memory capabilities in tabletop robotic manipulation. Our work introduces a unified framework to advance memory RL research, enabling more robust systems for real-world use. MIKASA is available at https://tinyurl.com/membenchrobots.","url_abs":"https://arxiv.org/abs/2502.10550v2","url_pdf":"https://arxiv.org/pdf/2502.10550v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memory-benchmark-robots-a-benchmark-for","repo_url":"https://github.com/CognitiveAISystems/MIKASA-Robo","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"skills-assessment","task_name":"Skills Assessment"}],"methods":[],"datasets_introduced":[{"slug":"mikasa-robo-dataset","name":"MIKASA-Robo Dataset","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.10550","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.10550"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CognitiveAISystems/MIKASA-Robo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"beb42d440ccf2925","entry":"layer_init","repo":"CognitiveAISystems/MIKASA-Robo","repo_kind":"official","path":"mikasa_robo_suite/rl/dataset_collectors/get_dataset_collectors_ckpt.py","file_url":"https://github.com/CognitiveAISystems/MIKASA-Robo/blob/HEAD/mikasa_robo_suite/rl/dataset_collectors/get_dataset_collectors_ckpt.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"beb42d440ccf2925"}},{"code_sha256_prefix":"ec6d7f193559c315","entry":"env_info","repo":"CognitiveAISystems/MIKASA-Robo","repo_kind":"official","path":"mikasa_robo_suite/rl/dataset_collectors/get_mikasa_robo_datasets.py","file_url":"https://github.com/CognitiveAISystems/MIKASA-Robo/blob/HEAD/mikasa_robo_suite/rl/dataset_collectors/get_mikasa_robo_datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec6d7f193559c315"}},{"code_sha256_prefix":"32810daed3dbbdad","entry":"get_env_config","repo":"CognitiveAISystems/MIKASA-Robo","repo_kind":"official","path":"mikasa_robo_suite/rl/dataset_collectors/get_dataset_collectors_ckpt.py","file_url":"https://github.com/CognitiveAISystems/MIKASA-Robo/blob/HEAD/mikasa_robo_suite/rl/dataset_collectors/get_dataset_collectors_ckpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32810daed3dbbdad"}},{"code_sha256_prefix":"edb09234403168e1","entry":"get_list_of_all_checkpoints_available","repo":"CognitiveAISystems/MIKASA-Robo","repo_kind":"official","path":"mikasa_robo_suite/rl/dataset_collectors/get_mikasa_robo_datasets.py","file_url":"https://github.com/CognitiveAISystems/MIKASA-Robo/blob/HEAD/mikasa_robo_suite/rl/dataset_collectors/get_mikasa_robo_datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edb09234403168e1"}},{"code_sha256_prefix":"a9173814c0f84ef0","entry":"load_benchmark_tasks","repo":"CognitiveAISystems/MIKASA-Robo","repo_kind":"official","path":"mikasa_robo_suite/vla/benchmarking.py","file_url":"https://github.com/CognitiveAISystems/MIKASA-Robo/blob/HEAD/mikasa_robo_suite/vla/benchmarking.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9173814c0f84ef0"}},{"code_sha256_prefix":"f5c634c5cd641e9e","entry":"make_run_dir","repo":"CognitiveAISystems/MIKASA-Robo","repo_kind":"official","path":"mikasa_robo_suite/vla/benchmarking.py","file_url":"https://github.com/CognitiveAISystems/MIKASA-Robo/blob/HEAD/mikasa_robo_suite/vla/benchmarking.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5c634c5cd641e9e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}