{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dmc-vb-a-benchmark-for-representation","title":"DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors","arxiv_id":"2409.18330","date":"2024-09-26","proceeding":null,"authors":["Joseph Ortiz","Antoine Dedieu","Wolfgang Lehrach","Swaroop Guntupalli","Carter Wendelken","Ahmad Humayun","Guangyao Zhou","Sivaramakrishnan Swaminathan","Miguel Lázaro-Gredilla","Kevin Murphy"],"abstract":"Learning from previously collected data via behavioral cloning or offline reinforcement learning (RL) is a powerful recipe for scaling generalist agents by avoiding the need for expensive online learning. Despite strong generalization in some respects, agents are often remarkably brittle to minor visual variations in control-irrelevant factors such as the background or camera viewpoint. In this paper, we present theDeepMind Control Visual Benchmark (DMC-VB), a dataset collected in the DeepMind Control Suite to evaluate the robustness of offline RL agents for solving continuous control tasks from visual input in the presence of visual distractors. In contrast to prior works, our dataset (a) combines locomotion and navigation tasks of varying difficulties, (b) includes static and dynamic visual variations, (c) considers data generated by policies with different skill levels, (d) systematically returns pairs of state and pixel observation, (e) is an order of magnitude larger, and (f) includes tasks with hidden goals. Accompanying our dataset, we propose three benchmarks to evaluate representation learning methods for pretraining, and carry out experiments on several recently proposed methods. First, we find that pretrained representations do not help policy learning on DMC-VB, and we highlight a large representation gap between policies learned on pixel observations and on states. Second, we demonstrate when expert data is limited, policy learning can benefit from representations pretrained on (a) suboptimal data, and (b) tasks with stochastic hidden goals. Our dataset and benchmark code to train and evaluate agents are available at: https://github.com/google-deepmind/dmc_vision_benchmark.","url_abs":"https://arxiv.org/abs/2409.18330v1","url_pdf":"https://arxiv.org/pdf/2409.18330v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dmc-vb-a-benchmark-for-representation","repo_url":"https://github.com/google-deepmind/dmc_vision_benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2409.18330","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.18330"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/google-deepmind/dmc_vision_benchmark","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"57aa333588d7ab69","entry":"blend_to_background","repo":"google-deepmind/dmc_vision_benchmark","repo_kind":"official","path":"dmc_vision_benchmark/environments/distracting_suite.py","file_url":"https://github.com/google-deepmind/dmc_vision_benchmark/blob/HEAD/dmc_vision_benchmark/environments/distracting_suite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57aa333588d7ab69"}},{"code_sha256_prefix":"a970a840b1d30f83","entry":"get_actor_critic_trunk_default","repo":"google-deepmind/dmc_vision_benchmark","repo_kind":"official","path":"dmc_vision_benchmark/rep_learn/agent_modules.py","file_url":"https://github.com/google-deepmind/dmc_vision_benchmark/blob/HEAD/dmc_vision_benchmark/rep_learn/agent_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a970a840b1d30f83"}},{"code_sha256_prefix":"b3b72ef7e5a72b71","entry":"get_state_encoder_default","repo":"google-deepmind/dmc_vision_benchmark","repo_kind":"official","path":"dmc_vision_benchmark/rep_learn/agent_modules.py","file_url":"https://github.com/google-deepmind/dmc_vision_benchmark/blob/HEAD/dmc_vision_benchmark/rep_learn/agent_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b3b72ef7e5a72b71"}},{"code_sha256_prefix":"38dc185c55ec44fb","entry":"pack_videos","repo":"google-deepmind/dmc_vision_benchmark","repo_kind":"official","path":"dmc_vision_benchmark/evals/visualize.py","file_url":"https://github.com/google-deepmind/dmc_vision_benchmark/blob/HEAD/dmc_vision_benchmark/evals/visualize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38dc185c55ec44fb"}},{"code_sha256_prefix":"36b718be95cf2934","entry":"scale_video","repo":"google-deepmind/dmc_vision_benchmark","repo_kind":"official","path":"dmc_vision_benchmark/evals/visualize.py","file_url":"https://github.com/google-deepmind/dmc_vision_benchmark/blob/HEAD/dmc_vision_benchmark/evals/visualize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36b718be95cf2934"}},{"code_sha256_prefix":"ffa2dd6d2495b8ab","entry":"size_and_flatten","repo":"google-deepmind/dmc_vision_benchmark","repo_kind":"official","path":"dmc_vision_benchmark/environments/distracting_suite.py","file_url":"https://github.com/google-deepmind/dmc_vision_benchmark/blob/HEAD/dmc_vision_benchmark/environments/distracting_suite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ffa2dd6d2495b8ab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}