{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/real-time-joint-semantic-segmentation-and","title":"Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations","arxiv_id":"1809.04766","date":"2018-09-13","proceeding":null,"authors":["Vladimir Nekrasov","Thanuja Dharmasiri","Andrew Spek","Tom Drummond","Chunhua Shen","Ian Reid"],"abstract":"Deployment of deep learning models in robotics as sensory information\nextractors can be a daunting task to handle, even using generic GPU cards.\nHere, we address three of its most prominent hurdles, namely, i) the adaptation\nof a single model to perform multiple tasks at once (in this work, we consider\ndepth estimation and semantic segmentation crucial for acquiring geometric and\nsemantic understanding of the scene), while ii) doing it in real-time, and iii)\nusing asymmetric datasets with uneven numbers of annotations per each modality.\nTo overcome the first two issues, we adapt a recently proposed real-time\nsemantic segmentation network, making changes to further reduce the number of\nfloating point operations. To approach the third issue, we embrace a simple\nsolution based on hard knowledge distillation under the assumption of having\naccess to a powerful `teacher' network. We showcase how our system can be\neasily extended to handle more tasks, and more datasets, all at once,\nperforming depth estimation and segmentation both indoors and outdoors with a\nsingle model. Quantitatively, we achieve results equivalent to (or better than)\ncurrent state-of-the-art approaches with one forward pass costing just 13ms and\n6.5 GFLOPs on 640x480 inputs. This efficiency allows us to directly incorporate\nthe raw predictions of our network into the SemanticFusion framework for dense\n3D semantic reconstruction of the scene.","url_abs":"http://arxiv.org/abs/1809.04766v2","url_pdf":"http://arxiv.org/pdf/1809.04766v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"real-time-joint-semantic-segmentation-and","repo_url":"https://github.com/DrSleep/multi-task-refinenet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"real-time-joint-semantic-segmentation-and","repo_url":"https://github.com/AleksBanbur/EE8204---Real-Time-Multi-Task-Learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"real-time-joint-semantic-segmentation-and","repo_url":"https://github.com/AndreiMoraru123/Hydra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"real-time-joint-semantic-segmentation-and","repo_url":"https://github.com/MindSpore-paper-code-2/code2/tree/main/RefineNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"},{"task_slug":"real-time-semantic-segmentation","task_name":"Real-Time Semantic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"surface-normals-estimation","task_name":"Surface Normals Estimation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/monocular-depth-estimation-on-nyu-depth-v2","task":"Monocular Depth Estimation","dataset":"NYU-Depth V2","model":"Multi-Task Light-Weight-RefineNet","rank_in_archive_order":79,"of":85,"metrics":{"RMSE":"0.565"},"uses_additional_data":false},{"leaderboard":"/sota/real-time-semantic-segmentation-on-nyu-depth-1","task":"Real-Time Semantic Segmentation","dataset":"NYU Depth v2","model":"Multi-Task Light-Weight-RefineNet","rank_in_archive_order":6,"of":10,"metrics":{"Speed(ms/f)":"13","mIoU":"42.0"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"Multi-Task Light-Weight-RefineNet","rank_in_archive_order":104,"of":121,"metrics":{"Mean IoU":"42.0%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1809.04766","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.04766"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AndreiMoraru123/Hydra","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AleksBanbur/EE8204---Real-Time-Multi-Task-Learning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DrSleep/multi-task-refinenet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-paper-code-2/code2/tree/main/RefineNet","reach":null}],"summary":{"ran_draft_wrong":6},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"b0c7a956ef5db771","entry":"batch_norm","repo":"AleksBanbur/EE8204---Real-Time-Multi-Task-Learning","repo_kind":"listed","path":"model.py","file_url":"https://github.com/AleksBanbur/EE8204---Real-Time-Multi-Task-Learning/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0c7a956ef5db771"}},{"code_sha256_prefix":"f14815b1ec63be18","entry":"batchnorm","repo":"DrSleep/multi-task-refinenet","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/DrSleep/multi-task-refinenet/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f14815b1ec63be18"}},{"code_sha256_prefix":"1a9fefd2a89349fa","entry":"conv1x1","repo":"AleksBanbur/EE8204---Real-Time-Multi-Task-Learning","repo_kind":"listed","path":"model.py","file_url":"https://github.com/AleksBanbur/EE8204---Real-Time-Multi-Task-Learning/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a9fefd2a89349fa"}},{"code_sha256_prefix":"6dd066d1daaec9ba","entry":"conv1x1","repo":"DrSleep/multi-task-refinenet","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/DrSleep/multi-task-refinenet/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6dd066d1daaec9ba"}},{"code_sha256_prefix":"2fe0da8bae58287c","entry":"conv3x3","repo":"AleksBanbur/EE8204---Real-Time-Multi-Task-Learning","repo_kind":"listed","path":"model.py","file_url":"https://github.com/AleksBanbur/EE8204---Real-Time-Multi-Task-Learning/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fe0da8bae58287c"}},{"code_sha256_prefix":"5615983908c7aa6c","entry":"conv3x3","repo":"DrSleep/multi-task-refinenet","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/DrSleep/multi-task-refinenet/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5615983908c7aa6c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}