{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/traumerai-dreaming-music-with-stylegan","title":"TräumerAI: Dreaming Music with StyleGAN","arxiv_id":"2102.04680","date":"2021-02-09","proceeding":null,"authors":["Dasaem Jeong","Seungheon Doh","Taegyun Kwon"],"abstract":"The goal of this paper to generate a visually appealing video that responds to music with a neural network so that each frame of the video reflects the musical characteristics of the corresponding audio clip. To achieve the goal, we propose a neural music visualizer directly mapping deep music embeddings to style embeddings of StyleGAN, named Tr\\\"aumerAI, which consists of a music auto-tagging model using short-chunk CNN and StyleGAN2 pre-trained on WikiArt dataset. Rather than establishing an objective metric between musical and visual semantics, we manually labeled the pairs in a subjective manner. An annotator listened to 100 music clips of 10 seconds long and selected an image that suits the music among the 200 StyleGAN-generated examples. Based on the collected data, we trained a simple transfer function that converts an audio embedding to a style embedding. The generated examples show that the mapping between audio and video makes a certain level of intra-segment similarity and inter-segment dissimilarity.","url_abs":"https://arxiv.org/abs/2102.04680v1","url_pdf":"https://arxiv.org/pdf/2102.04680v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"traumerai-dreaming-music-with-stylegan","repo_url":"https://github.com/jdasam/traeumerAI","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"music-auto-tagging","task_name":"Music Auto-Tagging"}],"methods":[{"method_slug":"adaptive-instance-normalization","method_name":"Adaptive Instance Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"path-length-regularization","method_name":"Path Length Regularization"},{"method_slug":"r1-regularization","method_name":"R1 Regularization"},{"method_slug":"stylegan","method_name":"StyleGAN"},{"method_slug":"stylegan2","method_name":"StyleGAN2"},{"method_slug":"weight-demodulation","method_name":"Weight Demodulation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/music-auto-tagging-on-timetravel","task":"Music Auto-Tagging","dataset":"TimeTravel","model":"Fellini","rank_in_archive_order":1,"of":1,"metrics":{"0..5sec":"5"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2102.04680","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}