{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/every-pixel-counts-joint-learning-of-geometry","title":"Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding","arxiv_id":"1810.06125","date":"2018-10-14","proceeding":null,"authors":["Chenxu Luo","Zhenheng Yang","Peng Wang","Yang Wang","Wei Xu","Ram Nevatia","Alan Yuille"],"abstract":"Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One typical assumption of the existing depth estimation methods is that the scenes contain no independent moving objects. while object moving could be easily modeled using optical flow. In this paper, we propose to address the two tasks as a whole, i.e. to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as \"Every Pixel Counts++\" or \"EPC++\". Specifically, during training, given two consecutive frames from a video, we adopt three parallel networks to predict the camera motion (MotionNet), dense depth map (DepthNet), and per-pixel optical flow between two frames (OptFlowNet) respectively. The three types of information are fed into a holistic 3D motion parser (HMP), and per-pixel 3D motion of both rigid background and moving objects are disentangled and recovered. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods. Code will be available at: https://github.com/chenxuluo/EPC.","url_abs":"https://arxiv.org/abs/1810.06125v2","url_pdf":"https://arxiv.org/pdf/1810.06125v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"every-pixel-counts-joint-learning-of-geometry","repo_url":"https://github.com/chenxuluo/EPC","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-geometry","task_name":"3D geometry"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"scene-flow-estimation","task_name":"Scene Flow Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/scene-flow-estimation-on-kitti-2015-scene","task":"Scene Flow Estimation","dataset":"KITTI 2015 Scene Flow Training","model":"EPC++","rank_in_archive_order":1,"of":4,"metrics":{" Runtime (s)":"0.05","D1-all":"23.84","D2-all":" 60.32","Fl-all":"19.64","SF-all":"(>60.32)"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1810.06125","atlas_url":"https://app.syntology.ai/?focus=1810.06125","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1810.06125"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chenxuluo/EPC","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a15c2c0e0aedaa22","entry":"resize_like","repo":"chenxuluo/EPC","repo_kind":"official","path":"nets.py","file_url":"https://github.com/chenxuluo/EPC/blob/HEAD/nets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a15c2c0e0aedaa22"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}