{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/musicbert-symbolic-music-understanding-with","title":"MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training","arxiv_id":"2106.05630","date":"2021-06-10","proceeding":"Findings (ACL) 2021 8","authors":["Mingliang Zeng","Xu Tan","Rui Wang","Zeqian Ju","Tao Qin","Tie-Yan Liu"],"abstract":"Symbolic music understanding, which refers to the understanding of music from the symbolic data (e.g., MIDI format, but not audio), covers many music applications such as genre classification, emotion classification, and music pieces matching. While good music representations are beneficial for these applications, the lack of training data hinders representation learning. Inspired by the success of pre-training models in natural language processing, in this paper, we develop MusicBERT, a large-scale pre-trained model for music understanding. To this end, we construct a large-scale symbolic music corpus that contains more than 1 million music songs. Since symbolic music contains more structural (e.g., bar, position) and diverse information (e.g., tempo, instrument, and pitch), simply adopting the pre-training techniques from NLP to symbolic music only brings marginal gains. Therefore, we design several mechanisms, including OctupleMIDI encoding and bar-level masking strategy, to enhance pre-training with symbolic music data. Experiments demonstrate the advantages of MusicBERT on four music understanding tasks, including melody completion, accompaniment suggestion, genre classification, and style classification. Ablation studies also verify the effectiveness of our designs of OctupleMIDI encoding and bar-level masking strategy in MusicBERT.","url_abs":"https://arxiv.org/abs/2106.05630v1","url_pdf":"https://arxiv.org/pdf/2106.05630v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"musicbert-symbolic-music-understanding-with","repo_url":"https://github.com/microsoft/muzic/tree/main/musicbert","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"musicbert-symbolic-music-understanding-with","repo_url":"https://github.com/Natooz/MidiTok","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"emotion-classification","task_name":"Emotion Classification"},{"task_slug":"genre-classification","task_name":"Genre classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.05630","atlas_url":"https://app.syntology.ai/?focus=2106.05630","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.05630"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Natooz/MidiTok","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/muzic/tree/main/musicbert","reach":null}],"summary":{"ran_honours":2,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"080d0c696a0a9941","entry":"b2e","repo":"microsoft/muzic","repo_kind":"official","path":"musicbert/preprocess.py","file_url":"https://github.com/microsoft/muzic/blob/HEAD/musicbert/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"080d0c696a0a9941"}},{"code_sha256_prefix":"21c2e5101ac354b8","entry":"time_signature_reduce","repo":"microsoft/muzic","repo_kind":"official","path":"musicbert/preprocess.py","file_url":"https://github.com/microsoft/muzic/blob/HEAD/musicbert/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21c2e5101ac354b8"}},{"code_sha256_prefix":"1aa2ba272297146c","entry":"v2e","repo":"microsoft/muzic","repo_kind":"official","path":"musicbert/preprocess.py","file_url":"https://github.com/microsoft/muzic/blob/HEAD/musicbert/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1aa2ba272297146c"}},{"code_sha256_prefix":"eff932c4fe4749ec","entry":"MIDI_to_encoding","repo":"microsoft/muzic","repo_kind":"official","path":"musicbert/preprocess.py","file_url":"https://github.com/microsoft/muzic/blob/HEAD/musicbert/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eff932c4fe4749ec"}},{"code_sha256_prefix":"7b272edf71fd390f","entry":"d2e","repo":"microsoft/muzic","repo_kind":"official","path":"musicbert/preprocess.py","file_url":"https://github.com/microsoft/muzic/blob/HEAD/musicbert/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b272edf71fd390f"}},{"code_sha256_prefix":"f3d2948c891c6d3b","entry":"e2t","repo":"microsoft/muzic","repo_kind":"official","path":"musicbert/preprocess.py","file_url":"https://github.com/microsoft/muzic/blob/HEAD/musicbert/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3d2948c891c6d3b"}},{"code_sha256_prefix":"4b737a6cdd8ef10d","entry":"t2e","repo":"microsoft/muzic","repo_kind":"official","path":"musicbert/preprocess.py","file_url":"https://github.com/microsoft/muzic/blob/HEAD/musicbert/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b737a6cdd8ef10d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}