{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-mamba-compatible-with-trajectory","title":"Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?","arxiv_id":"2405.12094","date":"2024-05-20","proceeding":null,"authors":["Yang Dai","Oubo Ma","Longfei Zhang","Xingxing Liang","Shengchao Hu","Mengzhu Wang","Shouling Ji","Jincai Huang","Li Shen"],"abstract":"Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL). Yet, it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a promising new linear-time sequence model, offers performance on par with transformers while delivering substantially fewer parameters on long sequences. As it remains unclear whether Mamba is compatible with trajectory optimization, this work aims to conduct comprehensive experiments to explore the potential of Decision Mamba (dubbed DeMa) in offline RL from the aspect of data structures and essential components with the following insights: (1) Long sequences impose a significant computational burden without contributing to performance improvements since DeMa's focus on sequences diminishes approximately exponentially. Consequently, we introduce a Transformer-like DeMa as opposed to an RNN-like DeMa. (2) For the components of DeMa, we identify the hidden attention mechanism as a critical factor in its success, which can also work well with other residual structures and does not require position embedding. Extensive evaluations demonstrate that our specially designed DeMa is compatible with trajectory optimization and surpasses previous methods, outperforming Decision Transformer (DT) with higher performance while using 30\\% fewer parameters in Atari, and exceeding DT with only a quarter of the parameters in MuJoCo.","url_abs":"https://arxiv.org/abs/2405.12094v2","url_pdf":"https://arxiv.org/pdf/2405.12094v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-mamba-compatible-with-trajectory","repo_url":"https://github.com/AndssY/DeMa","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.12094","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.12094"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AndssY/DeMa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0161b27cbe3cc58d","entry":"discount_cumsum","repo":"AndssY/DeMa","repo_kind":"official","path":"gym/experiment.py","file_url":"https://github.com/AndssY/DeMa/blob/HEAD/gym/experiment.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0161b27cbe3cc58d"}},{"code_sha256_prefix":"1a75a8f54c83db13","entry":"param_to_module","repo":"AndssY/DeMa","repo_kind":"official","path":"atari/run_atari.py","file_url":"https://github.com/AndssY/DeMa/blob/HEAD/atari/run_atari.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a75a8f54c83db13"}},{"code_sha256_prefix":"c42d7e5324e5aa6f","entry":"param_to_module","repo":"AndssY/DeMa","repo_kind":"official","path":"gym/experiment.py","file_url":"https://github.com/AndssY/DeMa/blob/HEAD/gym/experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c42d7e5324e5aa6f"}},{"code_sha256_prefix":"080c11e246335440","entry":"build_vssm_model","repo":"AndssY/DeMa","repo_kind":"official","path":"atari/models/mamba.py","file_url":"https://github.com/AndssY/DeMa/blob/HEAD/atari/models/mamba.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"080c11e246335440"}},{"code_sha256_prefix":"8d664522a1ec25eb","entry":"report_parameters","repo":"AndssY/DeMa","repo_kind":"official","path":"atari/run_atari.py","file_url":"https://github.com/AndssY/DeMa/blob/HEAD/atari/run_atari.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d664522a1ec25eb"}},{"code_sha256_prefix":"2065374886dcaf50","entry":"report_parameters","repo":"AndssY/DeMa","repo_kind":"official","path":"gym/experiment.py","file_url":"https://github.com/AndssY/DeMa/blob/HEAD/gym/experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2065374886dcaf50"}},{"code_sha256_prefix":"97bfe84c97730104","entry":"report_parameters","repo":"AndssY/DeMa","repo_kind":"official","path":"gym/merge_exp.py","file_url":"https://github.com/AndssY/DeMa/blob/HEAD/gym/merge_exp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97bfe84c97730104"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}