{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-sound-of-motions","title":"The Sound of Motions","arxiv_id":"1904.05979","date":"2019-04-11","proceeding":"ICCV 2019 10","authors":["Hang Zhao","Chuang Gan","Wei-Chiu Ma","Antonio Torralba"],"abstract":"Sounds originate from object motions and vibrations of surrounding air.\nInspired by the fact that humans is capable of interpreting sound sources from\nhow objects move visually, we propose a novel system that explicitly captures\nsuch motion cues for the task of sound localization and separation. Our system\nis composed of an end-to-end learnable model called Deep Dense Trajectory\n(DDT), and a curriculum learning scheme. It exploits the inherent coherence of\naudio-visual signals from a large quantities of unlabeled videos. Quantitative\nand qualitative evaluations show that comparing to previous models that rely on\nvisual appearance cues, our motion based system improves performance in\nseparating musical instrument sounds. Furthermore, it separates sound\ncomponents from duets of the same category of instruments, a challenging\nproblem that has not been addressed before.","url_abs":"http://arxiv.org/abs/1904.05979v1","url_pdf":"http://arxiv.org/pdf/1904.05979v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-sound-of-motions","repo_url":"https://github.com/sheldontsui/minus-plus-network","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.05979","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}