{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-bimodal-learning-approach-to-assist-multi","title":"A Bimodal Learning Approach to Assist Multi-sensory Effects Synchronization","arxiv_id":"1804.10822","date":"2018-04-28","proceeding":null,"authors":["Raphael Abreu","Joel dos Santos","Eduardo Bezerra"],"abstract":"In mulsemedia applications, traditional media content (text, image, audio,\nvideo, etc.) can be related to media objects that target other human senses\n(e.g., smell, haptics, taste). Such applications aim at bridging the virtual\nand real worlds through sensors and actuators. Actuators are responsible for\nthe execution of sensory effects (e.g., wind, heat, light), which produce\nsensory stimulations on the users. In these applications sensory stimulation\nmust happen in a timely manner regarding the other traditional media content\nbeing presented. For example, at the moment in which an explosion is presented\nin the audiovisual content, it may be adequate to activate actuators that\nproduce heat and light. It is common to use some declarative multimedia\nauthoring language to relate the timestamp in which each media object is to be\npresented to the execution of some sensory effect. One problem in this setting\nis that the synchronization of media objects and sensory effects is done\nmanually by the author(s) of the application, a process which is time-consuming\nand error prone. In this paper, we present a bimodal neural network\narchitecture to assist the synchronization task in mulsemedia applications. Our\napproach is based on the idea that audio and video signals can be used\nsimultaneously to identify the timestamps in which some sensory effect should\nbe executed. Our learning architecture combines audio and video signals for the\nprediction of scene components. For evaluation purposes, we construct a dataset\nbased on Google's AudioSet. We provide experiments to validate our bimodal\narchitecture. Our results show that the bimodal approach produces better\nresults when compared to several variants of unimodal architectures.","url_abs":"http://arxiv.org/abs/1804.10822v1","url_pdf":"http://arxiv.org/pdf/1804.10822v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-bimodal-learning-approach-to-assist-multi","repo_url":"https://github.com/MLRG-CEFET-RJ/bimodal_audioset","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}