{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pre-training-contextualized-world-models-with-1","title":"Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning","arxiv_id":"2305.18499","date":"2023-05-29","proceeding":"NeurIPS 2023 11","authors":["Jialong Wu","Haoyu Ma","Chaoyi Deng","Mingsheng Long"],"abstract":"Unsupervised pre-training methods utilizing large and diverse datasets have achieved tremendous success across a range of domains. Recent work has investigated such unsupervised pre-training methods for model-based reinforcement learning (MBRL) but is limited to domain-specific or simulated data. In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of downstream visual control tasks. However, in-the-wild videos are complicated with various contextual factors, such as intricate backgrounds and textured appearance, which precludes a world model from extracting shared world knowledge to generalize better. To tackle this issue, we introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling to overcome the complexity and diversity of in-the-wild videos and facilitate knowledge transfer between distinct scenes. Specifically, a contextualized extension of the latent dynamics model is elaborately realized by incorporating a context encoder to retain contextual information and empower the image decoder, which encourages the latent dynamics model to concentrate on essential temporal variations. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of MBRL in various domains, including robotic manipulation, locomotion, and autonomous driving. Code is available at this repository: https://github.com/thuml/ContextWM.","url_abs":"https://arxiv.org/abs/2305.18499v2","url_pdf":"https://arxiv.org/pdf/2305.18499v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pre-training-contextualized-world-models-with-1","repo_url":"https://github.com/thuml/ContextWM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"model-based-reinforcement-learning","task_name":"Model-based Reinforcement Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.18499","atlas_url":"https://app.syntology.ai/?focus=2305.18499","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.18499"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thuml/ContextWM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ed3c4acfebf1e198","entry":"count_episodes","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/replay.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/replay.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed3c4acfebf1e198"}},{"code_sha256_prefix":"0e570e9f59796944","entry":"dict_apply","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/torch_utils.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/torch_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e570e9f59796944"}},{"code_sha256_prefix":"adf5449f8b15f04e","entry":"dict_detach","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/torch_utils.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/torch_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"adf5449f8b15f04e"}},{"code_sha256_prefix":"6ba3e24b6a006d72","entry":"dict_to_device","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/torch_utils.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/torch_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ba3e24b6a006d72"}},{"code_sha256_prefix":"26640a58b1b695f6","entry":"lambda_return","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/other.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/other.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26640a58b1b695f6"}},{"code_sha256_prefix":"d4e79d3bcc3f8363","entry":"load_episodes","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/replay.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/replay.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4e79d3bcc3f8363"}},{"code_sha256_prefix":"1bf97c0a08cb2642","entry":"save_episode","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/replay.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/replay.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bf97c0a08cb2642"}},{"code_sha256_prefix":"2e07fb68b4f246c1","entry":"schedule","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/other.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/other.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e07fb68b4f246c1"}},{"code_sha256_prefix":"d7df2f57994415fa","entry":"sequence_scan","repo":"thuml/ContextWM","repo_kind":"official","path":"wmlib/core/other.py","file_url":"https://github.com/thuml/ContextWM/blob/HEAD/wmlib/core/other.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7df2f57994415fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}