{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generalizable-visual-reinforcement-learning","title":"Generalizable Visual Reinforcement Learning with Segment Anything Model","arxiv_id":"2312.17116","date":"2023-12-28","proceeding":null,"authors":["Ziyu Wang","Yanjie Ze","Yifei Sun","Zhecheng Yuan","Huazhe Xu"],"abstract":"Learning policies that can generalize to unseen environments is a fundamental challenge in visual reinforcement learning (RL). While most current methods focus on acquiring robust visual representations through auxiliary supervision, pre-training, or data augmentation, the potential of modern vision foundation models remains underleveraged. In this work, we introduce Segment Anything Model for Generalizable visual RL (SAM-G), a novel framework that leverages the promptable segmentation ability of Segment Anything Model (SAM) to enhance the generalization capabilities of visual RL agents. We utilize image features from DINOv2 and SAM to find correspondence as point prompts to SAM, and then SAM produces high-quality masked images for agents directly. Evaluated across 8 DMControl tasks and 3 Adroit tasks, SAM-G significantly improves the visual generalization ability without altering the RL agents' architecture but merely their observations. Notably, SAM-G achieves 44% and 29% relative improvements on the challenging video hard setting on DMControl and Adroit respectively, compared to state-of-the-art methods. Video and code: https://yanjieze.com/SAM-G/","url_abs":"https://arxiv.org/abs/2312.17116v1","url_pdf":"https://arxiv.org/pdf/2312.17116v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generalizable-visual-reinforcement-learning","repo_url":"https://github.com/wadiuvatzy/sam-g","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"model","task_name":"model"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.17116","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.17116"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wadiuvatzy/sam-g","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":2,"ran_fixture":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bb3d4f05a88ee013","entry":"load_image","repo":"wadiuvatzy/sam-g","repo_kind":"official","path":"efficientvit/demo_sam_model.py","file_url":"https://github.com/wadiuvatzy/sam-g/blob/HEAD/efficientvit/demo_sam_model.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb3d4f05a88ee013"}},{"code_sha256_prefix":"052cf8121d95cddc","entry":"bbox_xywh_to_xyxy","repo":"wadiuvatzy/sam-g","repo_kind":"official","path":"efficientvit/eval_sam_coco.py","file_url":"https://github.com/wadiuvatzy/sam-g/blob/HEAD/efficientvit/eval_sam_coco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"052cf8121d95cddc"}},{"code_sha256_prefix":"799ae994e666d035","entry":"cat_images","repo":"wadiuvatzy/sam-g","repo_kind":"official","path":"efficientvit/demo_sam_model.py","file_url":"https://github.com/wadiuvatzy/sam-g/blob/HEAD/efficientvit/demo_sam_model.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"799ae994e666d035"}},{"code_sha256_prefix":"066e85f370a04fd7","entry":"draw_binary_mask","repo":"wadiuvatzy/sam-g","repo_kind":"official","path":"efficientvit/demo_sam_model.py","file_url":"https://github.com/wadiuvatzy/sam-g/blob/HEAD/efficientvit/demo_sam_model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"066e85f370a04fd7"}},{"code_sha256_prefix":"1b14059bb75ef19e","entry":"iou","repo":"wadiuvatzy/sam-g","repo_kind":"official","path":"efficientvit/eval_sam_coco.py","file_url":"https://github.com/wadiuvatzy/sam-g/blob/HEAD/efficientvit/eval_sam_coco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b14059bb75ef19e"}},{"code_sha256_prefix":"bdef24507a4ccfa7","entry":"mlp","repo":"wadiuvatzy/sam-g","repo_kind":"official","path":"algos/drq.py","file_url":"https://github.com/wadiuvatzy/sam-g/blob/HEAD/algos/drq.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdef24507a4ccfa7"}},{"code_sha256_prefix":"f1ba293af9a573fd","entry":"make_agent","repo":"wadiuvatzy/sam-g","repo_kind":"official","path":"carlatrain.py","file_url":"https://github.com/wadiuvatzy/sam-g/blob/HEAD/carlatrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1ba293af9a573fd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}