{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decision-mamba-a-multi-grained-state-space","title":"Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL","arxiv_id":"2406.05427","date":"2024-06-08","proceeding":null,"authors":["Qi Lv","Xiang Deng","Gongwei Chen","Michael Yu Wang","Liqiang Nie"],"abstract":"While the conditional sequence modeling with the transformer architecture has demonstrated its effectiveness in dealing with offline reinforcement learning (RL) tasks, it is struggle to handle out-of-distribution states and actions. Existing work attempts to address this issue by data augmentation with the learned policy or adding extra constraints with the value-based RL algorithm. However, these studies still fail to overcome the following challenges: (1) insufficiently utilizing the historical temporal information among inter-steps, (2) overlooking the local intrastep relationships among return-to-gos (RTGs), states, and actions, (3) overfitting suboptimal trajectories with noisy labels. To address these challenges, we propose Decision Mamba (DM), a novel multi-grained state space model (SSM) with a self-evolving policy learning strategy. DM explicitly models the historical hidden state to extract the temporal information by using the mamba architecture. To capture the relationship among RTG-state-action triplets, a fine-grained SSM module is designed and integrated into the original coarse-grained SSM in mamba, resulting in a novel mamba architecture tailored for offline RL. Finally, to mitigate the overfitting issue on noisy trajectories, a self-evolving policy is proposed by using progressive regularization. The policy evolves by using its own past knowledge to refine the suboptimal actions, thus enhancing its robustness on noisy demonstrations. Extensive experiments on various tasks show that DM outperforms other baselines substantially.","url_abs":"https://arxiv.org/abs/2406.05427v3","url_pdf":"https://arxiv.org/pdf/2406.05427v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decision-mamba-a-multi-grained-state-space","repo_url":"https://github.com/aopolin-lv/DecisionMamba","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.05427","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.05427"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/kzl/decision-transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aopolin-lv/DecisionMamba","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran":4,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1},"community":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0161b27cbe3cc58d","entry":"discount_cumsum","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"experiment-d4rl/experiment.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/experiment-d4rl/experiment.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0161b27cbe3cc58d"}},{"code_sha256_prefix":"f219cb4953f6a929","entry":"add_hook","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"experiment-d4rl/visualize.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/experiment-d4rl/visualize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f219cb4953f6a929"}},{"code_sha256_prefix":"167ed5cb9b55d213","entry":"add_hook_group","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"experiment-d4rl/visualize.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/experiment-d4rl/visualize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"167ed5cb9b55d213"}},{"code_sha256_prefix":"0b9a2957d531f968","entry":"evaluate_episode_rtg","repo":"Toshihiro-Ota/decision-mamba","repo_kind":"community","path":"gym/util/trainer.py","file_url":"https://github.com/Toshihiro-Ota/decision-mamba/blob/HEAD/gym/util/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b9a2957d531f968"}},{"code_sha256_prefix":"c7fc1727fb00a10b","entry":"get_optimizer","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"experiment-d4rl/utils.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/experiment-d4rl/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7fc1727fb00a10b"}},{"code_sha256_prefix":"7b11f2a122671aa8","entry":"str_to_bool","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"experiment-d4rl/utils.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/experiment-d4rl/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b11f2a122671aa8"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"aopolin-lv/DecisionMamba","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/aopolin-lv/DecisionMamba/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}