{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mutual-information-regularized-offline-1","title":"Mutual Information Regularized Offline Reinforcement Learning","arxiv_id":"2210.07484","date":"2022-10-14","proceeding":"NeurIPS 2023 11","authors":["Xiao Ma","Bingyi Kang","Zhongwen Xu","Min Lin","Shuicheng Yan"],"abstract":"The major challenge of offline RL is the distribution shift that appears when out-of-distribution actions are queried, which makes the policy improvement direction biased by extrapolation errors. Most existing methods address this problem by penalizing the policy or value for deviating from the behavior policy during policy improvement or evaluation. In this work, we propose a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset by directly constraining the policy improvement direction. MISA constructs lower bounds of mutual information parameterized by the policy and Q-values. We show that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset. Hence, we constrain the policy improvement direction to lie in the data manifold. The resulting algorithm simultaneously augments the policy evaluation and improvement by adding mutual information regularizations. MISA is a general framework that unifies conservative Q-learning (CQL) and behavior regularization methods (e.g., TD3+BC) as special cases. We introduce 3 different variants of MISA, and empirically demonstrate that tighter mutual information lower bound gives better offline RL performance. In addition, our extensive experiments show MISA significantly outperforms a wide range of baselines on various tasks of the D4RL benchmark,e.g., achieving 742.9 total points on gym-locomotion tasks. Our code is available at https://github.com/sail-sg/MISA.","url_abs":"https://arxiv.org/abs/2210.07484v3","url_pdf":"https://arxiv.org/pdf/2210.07484v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mutual-information-regularized-offline-1","repo_url":"https://github.com/sail-sg/misa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.07484","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07484"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sail-sg/MISA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/misa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":9},"by_repo_kind":{"official":{"samples":13,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d2f199c948f2a9ed","entry":"extend_and_repeat","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/jax_utils.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/jax_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2f199c948f2a9ed"}},{"code_sha256_prefix":"c8b2fc80a3fc7424","entry":"index_batch","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/replay_buffer.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8b2fc80a3fc7424"}},{"code_sha256_prefix":"cc35af8b18b7dc58","entry":"mse_loss","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/jax_utils.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/jax_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc35af8b18b7dc58"}},{"code_sha256_prefix":"3f55f8769dd45c20","entry":"parition_batch_train_test","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/replay_buffer.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f55f8769dd45c20"}},{"code_sha256_prefix":"bed19deb2a987950","entry":"compute_returns","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/traj_dataset.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/traj_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bed19deb2a987950"}},{"code_sha256_prefix":"db7a0efd9adcf84c","entry":"create_exp_name","repo":"sail-sg/MISA","repo_kind":"official","path":"viskit/logging.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/viskit/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db7a0efd9adcf84c"}},{"code_sha256_prefix":"8cada88c51a2a6f0","entry":"dict_to_safe_json","repo":"sail-sg/MISA","repo_kind":"official","path":"viskit/logging.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/viskit/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cada88c51a2a6f0"}},{"code_sha256_prefix":"d97e89c7f1953b3a","entry":"prefix_metrics","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/utils.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d97e89c7f1953b3a"}},{"code_sha256_prefix":"07afff96928bd142","entry":"safe_json","repo":"sail-sg/MISA","repo_kind":"official","path":"viskit/logging.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/viskit/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07afff96928bd142"}},{"code_sha256_prefix":"fd06199912f135e9","entry":"simple_separated_format","repo":"sail-sg/MISA","repo_kind":"official","path":"viskit/tabulate.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/viskit/tabulate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd06199912f135e9"}},{"code_sha256_prefix":"b5cd4f789150335c","entry":"split_into_trajectories","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/traj_dataset.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/traj_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5cd4f789150335c"}},{"code_sha256_prefix":"528710698bd3cfc6","entry":"update_target_network","repo":"sail-sg/MISA","repo_kind":"official","path":"algos/model.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/algos/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"528710698bd3cfc6"}},{"code_sha256_prefix":"6f4355a6f1314166","entry":"value_and_multi_grad","repo":"sail-sg/MISA","repo_kind":"official","path":"utilities/jax_utils.py","file_url":"https://github.com/sail-sg/MISA/blob/HEAD/utilities/jax_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f4355a6f1314166"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}