{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/embodied-language-grounding-with-implicit-3d","title":"Embodied Language Grounding with 3D Visual Feature Representations","arxiv_id":"1910.01210","date":"2019-10-02","proceeding":"CVPR 2020 6","authors":["Mihir Prabhudesai","Hsiao-Yu Fish Tung","Syed Ashar Javed","Maximilian Sieb","Adam W. Harley","Katerina Fragkiadaki"],"abstract":"We propose associating language utterances to 3D visual abstractions of the scene they describe. The 3D visual abstractions are encoded as 3-dimensional visual feature maps. We infer these 3D visual scene feature maps from RGB images of the scene via view prediction: when the generated 3D scene feature map is neurally projected from a camera viewpoint, it should match the corresponding RGB image. We present generative models that condition on the dependency tree of an utterance and generate a corresponding visual 3D feature map as well as reason about its plausibility, and detector models that condition on both the dependency tree of an utterance and a related image and localize the object referents in the 3D feature map inferred from the image. Our model outperforms models of language and vision that associate language with 2D CNN activations or 2D images by a large margin in a variety of tasks, such as, classifying plausibility of utterances, detecting referential expressions, and supplying rewards for trajectory optimization of object placement policies from language instructions. We perform numerous ablations and show the improved performance of our detectors is due to its better generalization across camera viewpoints and lack of object interferences in the inferred 3D feature space, and the improved performance of our generators is due to their ability to spatially reason about objects and their configurations in 3D when mapping from language to scenes.","url_abs":"https://arxiv.org/abs/1910.01210v3","url_pdf":"https://arxiv.org/pdf/1910.01210v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"embodied-language-grounding-with-implicit-3d","repo_url":"https://github.com/mihirp1998/EmbLang","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.01210","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.01210"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mihirp1998/EmbLang","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"606d49130380a0d8","entry":"l2loss_numpy","repo":"mihirp1998/EmbLang","repo_kind":"listed","path":"vis_imagine_static_voxels/evaluate.py","file_url":"https://github.com/mihirp1998/EmbLang/blob/HEAD/vis_imagine_static_voxels/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"606d49130380a0d8"}},{"code_sha256_prefix":"a99e461874240c89","entry":"make_border_black","repo":"mihirp1998/EmbLang","repo_kind":"listed","path":"vis_imagine_static_voxels/evaluate.py","file_url":"https://github.com/mihirp1998/EmbLang/blob/HEAD/vis_imagine_static_voxels/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a99e461874240c89"}},{"code_sha256_prefix":"8631816d1677d752","entry":"subsample_embs_2D","repo":"mihirp1998/EmbLang","repo_kind":"listed","path":"vis_imagine_static_voxels/evaluate.py","file_url":"https://github.com/mihirp1998/EmbLang/blob/HEAD/vis_imagine_static_voxels/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8631816d1677d752"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}