{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/combining-events-and-frames-using-recurrent","title":"Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction","arxiv_id":"2102.09320","date":"2021-02-18","proceeding":null,"authors":["Daniel Gehrig","Michelle Rüegg","Mathias Gehrig","Javier Hidalgo Carrio","Davide Scaramuzza"],"abstract":"Event cameras are novel vision sensors that report per-pixel brightness changes as a stream of asynchronous \"events\". They offer significant advantages compared to standard cameras due to their high temporal resolution, high dynamic range and lack of motion blur. However, events only measure the varying component of the visual signal, which limits their ability to encode scene context. By contrast, standard cameras measure absolute intensity frames, which capture a much richer representation of the scene. Both sensors are thus complementary. However, due to the asynchronous nature of events, combining them with synchronous images remains challenging, especially for learning-based methods. This is because traditional recurrent neural networks (RNNs) are not designed for asynchronous and irregular data from additional sensors. To address this challenge, we introduce Recurrent Asynchronous Multimodal (RAM) networks, which generalize traditional RNNs to handle asynchronous and irregular data from multiple sensors. Inspired by traditional RNNs, RAM networks maintain a hidden state that is updated asynchronously and can be queried at any time to generate a prediction. We apply this novel architecture to monocular depth estimation with events and frames where we show an improvement over state-of-the-art methods by up to 30% in terms of mean absolute depth error. To enable further research on multimodal learning with events, we release EventScape, a new dataset with events, intensity frames, semantic labels, and depth maps recorded in the CARLA simulator.","url_abs":"https://arxiv.org/abs/2102.09320v1","url_pdf":"https://arxiv.org/pdf/2102.09320v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"combining-events-and-frames-using-recurrent","repo_url":"https://github.com/uzh-rpg/rpg_ramnet","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"depth-prediction","task_name":"Depth Prediction"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"},{"task_slug":"object-detection","task_name":"Object Detection"}],"methods":[{"method_slug":"carla","method_name":"CARLA"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-on-dsec","task":"Object Detection","dataset":"DSEC","model":"RAMNet","rank_in_archive_order":12,"of":12,"metrics":{"mAP":"17.6"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-pku-ddd17-car","task":"Object Detection","dataset":"PKU-DDD17-Car","model":"RAMNet","rank_in_archive_order":13,"of":14,"metrics":{"mAP50":"79.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.09320","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.09320"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uzh-rpg/rpg_ramnet","reach":null}],"summary":{"ran_fixture":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"991179b2336238bf","entry":"prepare_depth_data","repo":"uzh-rpg/rpg_ramnet","repo_kind":"official","path":"RAM_Net/evaluation.py","file_url":"https://github.com/uzh-rpg/rpg_ramnet/blob/HEAD/RAM_Net/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"991179b2336238bf"}},{"code_sha256_prefix":"309766318635ebf3","entry":"rescale_by_the_median","repo":"uzh-rpg/rpg_ramnet","repo_kind":"official","path":"RAM_Net/evaluation.py","file_url":"https://github.com/uzh-rpg/rpg_ramnet/blob/HEAD/RAM_Net/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"309766318635ebf3"}},{"code_sha256_prefix":"afac895121a47138","entry":"eval_metrics","repo":"uzh-rpg/rpg_ramnet","repo_kind":"official","path":"RAM_Net/evaluation.py","file_url":"https://github.com/uzh-rpg/rpg_ramnet/blob/HEAD/RAM_Net/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"afac895121a47138"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}