{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simple-but-effective-clip-embeddings-for","title":"Simple but Effective: CLIP Embeddings for Embodied AI","arxiv_id":"2111.09888","date":"2021-11-18","proceeding":"CVPR 2022 1","authors":["Apoorv Khandelwal","Luca Weihs","Roozbeh Mottaghi","Aniruddha Kembhavi"],"abstract":"Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation. We investigate the effectiveness of CLIP visual backbones for Embodied AI tasks. We build incredibly simple baselines, named EmbCLIP, with no task specific architectures, inductive biases (such as the use of semantic maps), auxiliary tasks during training, or depth maps -- yet we find that our improved baselines perform very well across a range of tasks and simulators. EmbCLIP tops the RoboTHOR ObjectNav leaderboard by a huge margin of 20 pts (Success Rate). It tops the iTHOR 1-Phase Rearrangement leaderboard, beating the next best submission, which employs Active Neural Mapping, and more than doubling the % Fixed Strict metric (0.08 to 0.17). It also beats the winners of the 2021 Habitat ObjectNav Challenge, which employ auxiliary tasks, depth maps, and human demonstrations, and those of the 2019 Habitat PointNav Challenge. We evaluate the ability of CLIP's visual representations at capturing semantic information about input observations -- primitives that are useful for navigation-heavy embodied tasks -- and find that CLIP's representations encode these primitives more effectively than ImageNet-pretrained backbones. Finally, we extend one of our baselines, producing an agent capable of zero-shot object navigation that can navigate to objects that were not used as targets during training. Our code and models are available at https://github.com/allenai/embodied-clip","url_abs":"https://arxiv.org/abs/2111.09888v2","url_pdf":"https://arxiv.org/pdf/2111.09888v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simple-but-effective-clip-embeddings-for","repo_url":"https://github.com/allenai/embodied-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"simple-but-effective-clip-embeddings-for","repo_url":"https://github.com/allenai/ai2thor-rearrangement","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-manipulation","task_name":"Image Manipulation"},{"task_slug":"navigate","task_name":"Navigate"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.09888","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.09888"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/ai2thor-rearrangement","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/embodied-clip","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"79edb1cd88ed5dbc","entry":"class_mask","repo":"allenai/embodied-clip","repo_kind":"official","path":"primitive_probing/generate_data/thor_image_features.py","file_url":"https://github.com/allenai/embodied-clip/blob/HEAD/primitive_probing/generate_data/thor_image_features.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79edb1cd88ed5dbc"}},{"code_sha256_prefix":"8a526704af115184","entry":"freeze_model","repo":"allenai/embodied-clip","repo_kind":"official","path":"primitive_probing/generate_data/reachable_image_features.py","file_url":"https://github.com/allenai/embodied-clip/blob/HEAD/primitive_probing/generate_data/reachable_image_features.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a526704af115184"}},{"code_sha256_prefix":"e9cc7279106a055c","entry":"obj_presence","repo":"allenai/embodied-clip","repo_kind":"official","path":"primitive_probing/generate_data/thor_image_features.py","file_url":"https://github.com/allenai/embodied-clip/blob/HEAD/primitive_probing/generate_data/thor_image_features.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9cc7279106a055c"}},{"code_sha256_prefix":"73f1ea7b9675f75a","entry":"thor_id_to_class","repo":"allenai/embodied-clip","repo_kind":"official","path":"primitive_probing/generate_data/reachable_metadata.py","file_url":"https://github.com/allenai/embodied-clip/blob/HEAD/primitive_probing/generate_data/reachable_metadata.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73f1ea7b9675f75a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}