{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/wordepth-variational-language-prior-for","title":"WorDepth: Variational Language Prior for Monocular Depth Estimation","arxiv_id":"2404.03635","date":"2024-04-04","proceeding":"CVPR 2024 1","authors":["Ziyao Zeng","Daniel Wang","Fengyu Yang","Hyoungseob Park","Yangchao Wu","Stefano Soatto","Byung-Woo Hong","Dong Lao","Alex Wong"],"abstract":"Three-dimensional (3D) reconstruction from a single image is an ill-posed problem with inherent ambiguities, i.e. scale. Predicting a 3D scene from text description(s) is similarly ill-posed, i.e. spatial arrangements of objects described. We investigate the question of whether two inherently ambiguous modalities can be used in conjunction to produce metric-scaled reconstructions. To test this, we focus on monocular depth estimation, the problem of predicting a dense depth map from a single image, but with an additional text caption describing the scene. To this end, we begin by encoding the text caption as a mean and standard deviation; using a variational framework, we learn the distribution of the plausible metric reconstructions of 3D scenes corresponding to the text captions as a prior. To \"select\" a specific reconstruction or depth map, we encode the given image through a conditional sampler that samples from the latent space of the variational text encoder, which is then decoded to the output depth map. Our approach is trained alternatingly between the text and image branches: in one optimization step, we predict the mean and standard deviation from the text description and sample from a standard Gaussian, and in the other, we sample using a (image) conditional sampler. Once trained, we directly predict depth from the encoded text using the conditional sampler. We demonstrate our approach on indoor (NYUv2) and outdoor (KITTI) scenarios, where we show that language can consistently improve performance in both.","url_abs":"https://arxiv.org/abs/2404.03635v4","url_pdf":"https://arxiv.org/pdf/2404.03635v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"wordepth-variational-language-prior-for","repo_url":"https://github.com/adonis-galaxy/wordepth","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-reconstruction","task_name":"3D Reconstruction"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.03635","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.03635"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adonis-galaxy/wordepth","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Adonis-galaxy/WorDepth","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran":12,"ran_draft_wrong":2,"ran_honours":1,"unverified":10},"by_repo_kind":{"official":{"samples":26,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":26,"samples":[{"code_sha256_prefix":"144d10b49baeb8a6","entry":"window_partition","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/networks/swin_transformer.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/networks/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"144d10b49baeb8a6"}},{"code_sha256_prefix":"78636f3e5d41d22f","entry":"EpsLayer","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78636f3e5d41d22f"}},{"code_sha256_prefix":"e42969b7bf86747e","entry":"MetricLayer","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e42969b7bf86747e"}},{"code_sha256_prefix":"96be5d7e36e0d6f8","entry":"OutConv","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96be5d7e36e0d6f8"}},{"code_sha256_prefix":"22c1941ae236e18b","entry":"Refine","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22c1941ae236e18b"}},{"code_sha256_prefix":"f78085ec460691c1","entry":"SILogLoss","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f78085ec460691c1"}},{"code_sha256_prefix":"0ec8b081015d6151","entry":"Text_Encoder","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec8b081015d6151"}},{"code_sha256_prefix":"ca0383a1bf504494","entry":"Up","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca0383a1bf504494"}},{"code_sha256_prefix":"51d3a176d1206333","entry":"_load_checkpoint","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51d3a176d1206333"}},{"code_sha256_prefix":"e87fd998291bd040","entry":"compute_errors","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e87fd998291bd040"}},{"code_sha256_prefix":"a91567db5e6c35b4","entry":"get_dist_info","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a91567db5e6c35b4"}},{"code_sha256_prefix":"050c56fb7282eb88","entry":"get_num_lines","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"050c56fb7282eb88"}},{"code_sha256_prefix":"a1b8bb2acb22a0ec","entry":"get_torchvision_models","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1b8bb2acb22a0ec"}},{"code_sha256_prefix":"8a05ea4d5e917bf4","entry":"normalize_result","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a05ea4d5e917bf4"}},{"code_sha256_prefix":"46c295513e59ec5a","entry":"preprocessing_transforms","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/dataloaders/dataloader.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/dataloaders/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46c295513e59ec5a"}},{"code_sha256_prefix":"61bf152e6a42a184","entry":"window_reverse","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/networks/swin_transformer.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/networks/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61bf152e6a42a184"}},{"code_sha256_prefix":"4474d70757f8bbdf","entry":"BasicLayer","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4474d70757f8bbdf"}},{"code_sha256_prefix":"533554e0f3ae3b96","entry":"DoubleConv","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"533554e0f3ae3b96"}},{"code_sha256_prefix":"0ba2498ffe0b4454","entry":"SwinTransformer","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ba2498ffe0b4454"}},{"code_sha256_prefix":"895009095cba7a32","entry":"WorDepth","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"895009095cba7a32"}},{"code_sha256_prefix":"9b9dd29ad0d626a0","entry":"is_module_wrapper","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b9dd29ad0d626a0"}},{"code_sha256_prefix":"99a1fadf82ab1b67","entry":"load_checkpoint","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99a1fadf82ab1b67"}},{"code_sha256_prefix":"b9bb350524a8fa9f","entry":"load_state_dict","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b9bb350524a8fa9f"}},{"code_sha256_prefix":"3b1a511338023426","entry":"load_url_dist","repo":"Adonis-galaxy/WorDepth","repo_kind":"official","path":"src/networks/wordepth.py","file_url":"https://github.com/Adonis-galaxy/WorDepth/blob/HEAD/src/networks/wordepth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b1a511338023426"}},{"code_sha256_prefix":"92dad73ee855852f","entry":"load_url_dist","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/networks/utils.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/networks/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92dad73ee855852f"}},{"code_sha256_prefix":"9012d3fdd4cbaea1","entry":"resize","repo":"adonis-galaxy/wordepth","repo_kind":"official","path":"src/networks/utils.py","file_url":"https://github.com/adonis-galaxy/wordepth/blob/HEAD/src/networks/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9012d3fdd4cbaea1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}