{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncertainty-based-offline-variational","title":"Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions","arxiv_id":"2411.00465","date":"2024-11-01","proceeding":null,"authors":["Rui Yang","Jie Wang","Guoping Wu","Bin Li"],"abstract":"Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due to sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existing methods struggle to learn robust agents under high uncertainty caused by the diverse corrupted data (i.e., corrupted states, actions, rewards, and dynamics), leading to performance degradation in clean environments. To tackle this problem, we propose a novel robust variational Bayesian inference for offline RL (TRACER). It introduces Bayesian inference for the first time to capture the uncertainty via offline data for robustness against all types of data corruptions. Specifically, TRACER first models all corruptions as the uncertainty in the action-value function. Then, to capture such uncertainty, it uses all offline data as the observations to approximate the posterior distribution of the action-value function under a Bayesian inference framework. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure, since corrupted data often induces higher uncertainty and entropy. Based on the aforementioned measure, TRACER can regulate the loss associated with corrupted data to reduce its influence, thereby enhancing robustness and performance in clean environments. Experiments demonstrate that TRACER significantly outperforms several state-of-the-art approaches across both individual and simultaneous data corruptions.","url_abs":"https://arxiv.org/abs/2411.00465v1","url_pdf":"https://arxiv.org/pdf/2411.00465v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncertainty-based-offline-variational","repo_url":"https://github.com/MIRALab-USTC/RL-TRACER","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.00465","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.00465"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MIRALab-USTC/RL-TRACER","reach":null}],"summary":{"ran":2,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":13,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a84f652328ba915e","entry":"DistributionalCritic","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a84f652328ba915e"}},{"code_sha256_prefix":"5fc3f968515d4b15","entry":"TanhNormal","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fc3f968515d4b15"}},{"code_sha256_prefix":"d52eb4ad208682cb","entry":"huber_loss","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d52eb4ad208682cb"}},{"code_sha256_prefix":"53830e4c511d8058","entry":"qr_loss","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53830e4c511d8058"}},{"code_sha256_prefix":"a31194f645bf5030","entry":"BaseAlgorithm","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a31194f645bf5030"}},{"code_sha256_prefix":"ff597882fbf77ebc","entry":"DistributionalValueFunction","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff597882fbf77ebc"}},{"code_sha256_prefix":"051a7b2ec2158476","entry":"GaussianActor","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"051a7b2ec2158476"}},{"code_sha256_prefix":"57ef107ba6ccf74c","entry":"ObservationModel","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57ef107ba6ccf74c"}},{"code_sha256_prefix":"59a9261b91f00a77","entry":"QuantQFunction","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59a9261b91f00a77"}},{"code_sha256_prefix":"a0c6df6edba0cd30","entry":"TRACER","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0c6df6edba0cd30"}},{"code_sha256_prefix":"793f92e53c138d13","entry":"TanhGaussianActor","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"793f92e53c138d13"}},{"code_sha256_prefix":"7f00e1ce623d2217","entry":"log","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f00e1ce623d2217"}},{"code_sha256_prefix":"30067fe61c13c8bf","entry":"update_params","repo":"MIRALab-USTC/RL-TRACER","repo_kind":"official","path":"algo/model_free/tracer.py","file_url":"https://github.com/MIRALab-USTC/RL-TRACER/blob/HEAD/algo/model_free/tracer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30067fe61c13c8bf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}