{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/music-transformer","title":"Music Transformer","arxiv_id":"1809.04281","date":"2018-09-12","proceeding":"ICLR 2019 5","authors":["Cheng-Zhi Anna Huang","Ashish Vaswani","Jakob Uszkoreit","Noam Shazeer","Ian Simon","Curtis Hawthorne","Andrew M. Dai","Matthew D. Hoffman","Monica Dinculescu","Douglas Eck"],"abstract":"Music relies heavily on repetition to build structure and meaning.\nSelf-reference occurs on multiple timescales, from motifs to phrases to reusing\nof entire sections of music, such as in pieces with ABA structure. The\nTransformer (Vaswani et al., 2017), a sequence model based on self-attention,\nhas achieved compelling results in many generation tasks that require\nmaintaining long-range coherence. This suggests that self-attention might also\nbe well-suited to modeling music. In musical composition and performance,\nhowever, relative timing is critically important. Existing approaches for\nrepresenting relative positional information in the Transformer modulate\nattention based on pairwise distance (Shaw et al., 2018). This is impractical\nfor long sequences such as musical compositions since their memory complexity\nfor intermediate relative information is quadratic in the sequence length. We\npropose an algorithm that reduces their intermediate memory requirement to\nlinear in the sequence length. This enables us to demonstrate that a\nTransformer with our modified relative attention mechanism can generate\nminute-long compositions (thousands of steps, four times the length modeled in\nOore et al., 2018) with compelling structure, generate continuations that\ncoherently elaborate on a given motif, and in a seq2seq setup generate\naccompaniments conditioned on melodies. We evaluate the Transformer with our\nrelative attention mechanism on two datasets, JSB Chorales and\nPiano-e-Competition, and obtain state-of-the-art results on the latter.","url_abs":"http://arxiv.org/abs/1809.04281v3","url_pdf":"http://arxiv.org/pdf/1809.04281v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"music-transformer","repo_url":"https://github.com/Chatha-Sphere/pno-ai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"music-transformer","repo_url":"https://github.com/Jesplar/LSTM-MusicGenerator","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"music-transformer","repo_url":"https://github.com/VasanthManiVasi/MusicTransformer.jl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"music-transformer","repo_url":"https://github.com/dvruette/figaro","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"music-transformer","repo_url":"https://github.com/harryboos/Auto-Music-Generation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"music-transformer","repo_url":"https://github.com/jason9693/musictransformer-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"music-transformer","repo_url":"https://github.com/jason9693/musictransformer-tensorflow2.0","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"music-transformer","repo_url":"https://github.com/ololo123321/maestro","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"music-transformer","repo_url":"https://github.com/scpark20/Music-GPT-2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"music-transformer","repo_url":"https://github.com/vvvm23/TchAIkovsky-Legacy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"music-transformer","repo_url":"https://github.com/MindSpore-scientific/code-13/tree/main/text-to-music","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"music-transformer","repo_url":"https://github.com/Natooz/MidiTok","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"music-generation","task_name":"Music Generation"},{"task_slug":"music-modeling","task_name":"Music Modeling"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"seq2seq","method_name":"Seq2Seq"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/music-modeling-on-jsb-chorales","task":"Music Modeling","dataset":"JSB Chorales","model":"Music Transformer","rank_in_archive_order":3,"of":10,"metrics":{"NLL":"0.335"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1809.04281","atlas_url":"https://app.syntology.ai/?focus=1809.04281","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.04281"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jason9693/musictransformer-tensorflow2.0","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/harryboos/Auto-Music-Generation","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/scpark20/Music-GPT-2","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/VasanthManiVasi/MusicTransformer.jl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dvruette/figaro","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jason9693/musictransformer-pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-13/tree/main/text-to-music","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vvvm23/TchAIkovsky-Legacy","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Natooz/MidiTok","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ololo123321/maestro","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Chatha-Sphere/pno-ai","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jesplar/LSTM-MusicGenerator","reach":{"status":"ok"}}],"summary":{"ran_honours":2,"unverified":2},"by_repo_kind":{"listed":{"samples":4,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"4f853e28e0aae7cf","entry":"get_model_data","repo":"ololo123321/maestro","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/ololo123321/maestro/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f853e28e0aae7cf"}},{"code_sha256_prefix":"3868efd42e3f901a","entry":"sinusoid","repo":"jason9693/musictransformer-pytorch","repo_kind":"listed","path":"custom/layers.py","file_url":"https://github.com/jason9693/musictransformer-pytorch/blob/HEAD/custom/layers.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3868efd42e3f901a"}},{"code_sha256_prefix":"b7acbaed890bca30","entry":"dict2params","repo":"jason9693/musictransformer-pytorch","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/jason9693/musictransformer-pytorch/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7acbaed890bca30"}},{"code_sha256_prefix":"96ae2a6c81eb86dc","entry":"find_files_by_extensions","repo":"jason9693/musictransformer-pytorch","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/jason9693/musictransformer-pytorch/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96ae2a6c81eb86dc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}