{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/midicaps-a-large-scale-midi-dataset-with-text","title":"MidiCaps: A large-scale MIDI dataset with text captions","arxiv_id":"2406.02255","date":"2024-06-04","proceeding":null,"authors":["Jan Melechovsky","Abhinaba Roy","Dorien Herremans"],"abstract":"Generative models guided by text prompts are increasingly becoming more popular. However, no text-to-MIDI models currently exist due to the lack of a captioned MIDI dataset. This work aims to enable research that combines LLMs with symbolic music by presenting, the first openly available large-scale MIDI dataset with text captions. MIDI (Musical Instrument Digital Interface) files are widely used for encoding musical information and can capture the nuances of musical composition. They are widely used by music producers, composers, musicologists, and performers alike. Inspired by recent advancements in captioning techniques, we present a curated dataset of over 168k MIDI files with textual descriptions. Each MIDI caption describes the musical content, including tempo, chord progression, time signature, instruments, genre, and mood, thus facilitating multi-modal exploration and analysis. The dataset encompasses various genres, styles, and complexities, offering a rich data source for training and evaluating models for tasks such as music information retrieval, music understanding, and cross-modal translation. We provide detailed statistics about the dataset and have assessed the quality of the captions in an extensive listening study. We anticipate that this resource will stimulate further research at the intersection of music and natural language processing, fostering advancements in both fields.","url_abs":"https://arxiv.org/abs/2406.02255v2","url_pdf":"https://arxiv.org/pdf/2406.02255v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"midicaps-a-large-scale-midi-dataset-with-text","repo_url":"https://github.com/AMAAI-Lab/MidiCaps","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"midicaps-a-large-scale-midi-dataset-with-text","repo_url":"https://github.com/amaai-lab/t2m-inferalign","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"midicaps-a-large-scale-midi-dataset-with-text","repo_url":"https://github.com/amaai-lab/text2midi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"music-information-retrieval","task_name":"Music Information Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"midicaps","name":"MidiCaps","full_name":"MidiCaps"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.02255","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02255"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amaai-lab/t2m-inferalign","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AMAAI-Lab/MidiCaps","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amaai-lab/text2midi","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":10,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"df9410de11944207","entry":"custom_collate_fn","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"Text2midi/model/transformer_model.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/Text2midi/model/transformer_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df9410de11944207"}},{"code_sha256_prefix":"24ced75a01bea6eb","entry":"apply_rotary_emb","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"Text2midi/model/transformer_model.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/Text2midi/model/transformer_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24ced75a01bea6eb"}},{"code_sha256_prefix":"f8789c7ce4c9a34d","entry":"cal_diameter","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"key_reward.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/key_reward.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8789c7ce4c9a34d"}},{"code_sha256_prefix":"266c755d6259282f","entry":"collate_fn","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"Text2midi/model/train_accelerate.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/Text2midi/model/train_accelerate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"266c755d6259282f"}},{"code_sha256_prefix":"22b77295b3aaec86","entry":"collate_fn","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"Text2midi/model/train_hf.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/Text2midi/model/train_hf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22b77295b3aaec86"}},{"code_sha256_prefix":"129801b1d1f45a57","entry":"get_mtg_tags","repo":"AMAAI-Lab/MidiCaps","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/AMAAI-Lab/MidiCaps/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"129801b1d1f45a57"}},{"code_sha256_prefix":"2139cdef6aa3fa31","entry":"largest_distance","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"key_reward.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/key_reward.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2139cdef6aa3fa31"}},{"code_sha256_prefix":"81d44a261c82e3a3","entry":"piano_roll_to_ce","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"key_reward.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/key_reward.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81d44a261c82e3a3"}},{"code_sha256_prefix":"22480b5c1cf849b8","entry":"precompute_freqs_cis","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"Text2midi/model/transformer_model.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/Text2midi/model/transformer_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22480b5c1cf849b8"}},{"code_sha256_prefix":"71c1ded06c1801c2","entry":"preprocess_logits","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"Text2midi/model/train_hf.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/Text2midi/model/train_hf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71c1ded06c1801c2"}},{"code_sha256_prefix":"a8ee7e2003a27b10","entry":"save_wav","repo":"amaai-lab/t2m-inferalign","repo_kind":"listed","path":"Text2midi/utils/midi_to_wav.py","file_url":"https://github.com/amaai-lab/t2m-inferalign/blob/HEAD/Text2midi/utils/midi_to_wav.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8ee7e2003a27b10"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}