{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluation-of-pretrained-language-models-on","title":"Evaluation of pretrained language models on music understanding","arxiv_id":"2409.11449","date":"2024-09-17","proceeding":null,"authors":["Yannis Vasilakis","Rachel Bittner","Johan Pauwels"],"abstract":"Music-text multimodal systems have enabled new approaches to Music Information Research (MIR) applications such as audio-to-text and text-to-audio retrieval, text-based song generation, and music captioning. Despite the reported success, little effort has been put into evaluating the musical knowledge of Large Language Models (LLM). In this paper, we demonstrate that LLMs suffer from 1) prompt sensitivity, 2) inability to model negation (e.g. 'rock song without guitar'), and 3) sensitivity towards the presence of specific words. We quantified these properties as a triplet-based accuracy, evaluating the ability to model the relative similarity of labels in a hierarchical ontology. We leveraged the Audioset ontology to generate triplets consisting of an anchor, a positive (relevant) label, and a negative (less relevant) label for the genre and instruments sub-tree. We evaluated the triplet-based musical knowledge for six general-purpose Transformer-based models. The triplets obtained through this methodology required filtering, as some were difficult to judge and therefore relatively uninformative for evaluation purposes. Despite the relatively high accuracy reported, inconsistencies are evident in all six models, suggesting that off-the-shelf LLMs need adaptation to music before use.","url_abs":"https://arxiv.org/abs/2409.11449v1","url_pdf":"https://arxiv.org/pdf/2409.11449v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluation-of-pretrained-language-models-on","repo_url":"https://github.com/YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"music-captioning","task_name":"Music Captioning"},{"task_slug":"negation","task_name":"Negation"},{"task_slug":"sensitivity","task_name":"Sensitivity"},{"task_slug":"text-to-audio-retrieval","task_name":"Text to Audio Retrieval"},{"task_slug":null,"task_name":"Triplet"}],"methods":[{"method_slug":"ontology","method_name":"Ontology"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.11449","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.11449"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"f3b5a7dc06dd2ab8","entry":"find_distance_to_root","repo":"YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding","repo_kind":"official","path":"build_triplets.py","file_url":"https://github.com/YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding/blob/HEAD/build_triplets.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3b5a7dc06dd2ab8"}},{"code_sha256_prefix":"c3fadf73a0d29244","entry":"flatten_ontology","repo":"YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding","repo_kind":"official","path":"build_triplets.py","file_url":"https://github.com/YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding/blob/HEAD/build_triplets.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3fadf73a0d29244"}},{"code_sha256_prefix":"f97f441d32a1da99","entry":"shortest_distance","repo":"YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding","repo_kind":"official","path":"build_triplets.py","file_url":"https://github.com/YannisBilly/Evaluation-of-pretrained-language-models-on-music-understanding/blob/HEAD/build_triplets.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f97f441d32a1da99"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}