{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/m-fuse-multi-frame-fusion-for-scene-flow","title":"M-FUSE: Multi-frame Fusion for Scene Flow Estimation","arxiv_id":"2207.05704","date":"2022-07-12","proceeding":null,"authors":["Lukas Mehl","Azin Jahedi","Jenny Schmalfuss","Andrés Bruhn"],"abstract":"Recently, neural network for scene flow estimation show impressive results on automotive data such as the KITTI benchmark. However, despite of using sophisticated rigidity assumptions and parametrizations, such networks are typically limited to only two frame pairs which does not allow them to exploit temporal information. In our paper we address this shortcoming by proposing a novel multi-frame approach that considers an additional preceding stereo pair. To this end, we proceed in two steps: Firstly, building upon the recent RAFT-3D approach, we develop an improved two-frame baseline by incorporating an advanced stereo method. Secondly, and even more importantly, exploiting the specific modeling concepts of RAFT-3D, we propose a U-Net architecture that performs a fusion of forward and backward flow estimates and hence allows to integrate temporal information on demand. Experiments on the KITTI benchmark do not only show that the advantages of the improved baseline and the temporal fusion approach complement each other, they also demonstrate that the computed scene flow is highly accurate. More precisely, our approach ranks second overall and first for the even more challenging foreground objects, in total outperforming the original RAFT-3D method by more than 16%. Code is available at https://github.com/cv-stuttgart/M-FUSE.","url_abs":"https://arxiv.org/abs/2207.05704v2","url_pdf":"https://arxiv.org/pdf/2207.05704v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"m-fuse-multi-frame-fusion-for-scene-flow","repo_url":"https://github.com/cv-stuttgart/m-fuse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"scene-flow-estimation","task_name":"Scene Flow Estimation"}],"methods":[{"method_slug":"concatenated-skip-connection","method_name":"Concatenated Skip Connection"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"u-net","method_name":"U-Net"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/scene-flow-estimation-on-spring","task":"Scene Flow Estimation","dataset":"Spring","model":"M-FUSE (F)","rank_in_archive_order":1,"of":6,"metrics":{"1px total":"34.896"},"uses_additional_data":false},{"leaderboard":"/sota/scene-flow-estimation-on-spring","task":"Scene Flow Estimation","dataset":"Spring","model":"M-FUSE (K)","rank_in_archive_order":4,"of":6,"metrics":{"1px total":"62.490"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.05704","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}