{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluation-of-cnn-based-automatic-music","title":"Evaluation of CNN-based Automatic Music Tagging Models","arxiv_id":"2006.00751","date":"2020-06-01","proceeding":null,"authors":["Minz Won","Andres Ferraro","Dmitry Bogdanov","Xavier Serra"],"abstract":"Recent advances in deep learning accelerated the development of content-based automatic music tagging systems. Music information retrieval (MIR) researchers proposed various architecture designs, mainly based on convolutional neural networks (CNNs), that achieve state-of-the-art results in this multi-label binary classification task. However, due to the differences in experimental setups followed by researchers, such as using different dataset splits and software versions for evaluation, it is difficult to compare the proposed architectures directly with each other. To facilitate further research, in this paper we conduct a consistent evaluation of different music tagging models on three datasets (MagnaTagATune, Million Song Dataset, and MTG-Jamendo) and provide reference results using common evaluation metrics (ROC-AUC and PR-AUC). Furthermore, all the models are evaluated with perturbed inputs to investigate the generalization capabilities concerning time stretch, pitch shift, dynamic range compression, and addition of white noise. For reproducibility, we provide the PyTorch implementations with the pre-trained models.","url_abs":"https://arxiv.org/abs/2006.00751v1","url_pdf":"https://arxiv.org/pdf/2006.00751v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"evaluation-of-cnn-based-automatic-music","repo_url":"https://github.com/minzwon/sota-music-tagging-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"evaluation-of-cnn-based-automatic-music","repo_url":"https://github.com/Dohppak/Music_DeepEmbedding_Extractor","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"evaluation-of-cnn-based-automatic-music","repo_url":"https://github.com/HephaestusProject/pytorch-FCN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"evaluation-of-cnn-based-automatic-music","repo_url":"https://github.com/cgaroufis/msspt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"evaluation-of-cnn-based-automatic-music","repo_url":"https://github.com/jaehwlee/tf2-music-tagging-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"evaluation-of-cnn-based-automatic-music","repo_url":"https://github.com/minzwon/tag-based-music-retrieval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"evaluation-of-cnn-based-automatic-music","repo_url":"https://github.com/pxaris/ccml","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/music-auto-tagging-on-magnatagatune-clean","task":"Music Auto-Tagging","dataset":"MagnaTagATune (clean)","model":"Short-chunk CNN + Res","rank_in_archive_order":1,"of":3,"metrics":{"PR-AUC":"46.14","ROC-AUC":"91.29"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2006.00751","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.00751"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HephaestusProject/pytorch-FCN","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cgaroufis/msspt","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/minzwon/tag-based-music-retrieval","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jaehwlee/tf2-music-tagging-models","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/minzwon/sota-music-tagging-models","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Dohppak/Music_DeepEmbedding_Extractor","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pxaris/ccml","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1},"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"minzwon/sota-music-tagging-models","repo_kind":"official","path":"training/attention_modules.py","file_url":"https://github.com/minzwon/sota-music-tagging-models/blob/HEAD/training/attention_modules.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"3315c75383bc102e","entry":"get_config","repo":"HephaestusProject/pytorch-FCN","repo_kind":"listed","path":"evaluate.py","file_url":"https://github.com/HephaestusProject/pytorch-FCN/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3315c75383bc102e"}},{"code_sha256_prefix":"39b25bbed46ffd73","entry":"crop_audio","repo":"HephaestusProject/pytorch-FCN","repo_kind":"listed","path":"infer.py","file_url":"https://github.com/HephaestusProject/pytorch-FCN/blob/HEAD/infer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39b25bbed46ffd73"}},{"code_sha256_prefix":"a2b6a8d6296c3474","entry":"get_audio_loader","repo":"minzwon/sota-music-tagging-models","repo_kind":"official","path":"training/data_loader/jamendo_loader.py","file_url":"https://github.com/minzwon/sota-music-tagging-models/blob/HEAD/training/data_loader/jamendo_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2b6a8d6296c3474"}},{"code_sha256_prefix":"e5021099627a13b1","entry":"get_audio_loader","repo":"minzwon/sota-music-tagging-models","repo_kind":"official","path":"training/data_loader/msd_loader.py","file_url":"https://github.com/minzwon/sota-music-tagging-models/blob/HEAD/training/data_loader/msd_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5021099627a13b1"}},{"code_sha256_prefix":"3665d5f930e76b94","entry":"get_audio_loader","repo":"minzwon/sota-music-tagging-models","repo_kind":"official","path":"training/data_loader/mtat_loader.py","file_url":"https://github.com/minzwon/sota-music-tagging-models/blob/HEAD/training/data_loader/mtat_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3665d5f930e76b94"}},{"code_sha256_prefix":"2ee009ed5eb70422","entry":"hz_to_midi","repo":"minzwon/sota-music-tagging-models","repo_kind":"official","path":"training/modules.py","file_url":"https://github.com/minzwon/sota-music-tagging-models/blob/HEAD/training/modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ee009ed5eb70422"}},{"code_sha256_prefix":"c1c8cd1745fa5a3e","entry":"midi_to_hz","repo":"minzwon/sota-music-tagging-models","repo_kind":"official","path":"training/modules.py","file_url":"https://github.com/minzwon/sota-music-tagging-models/blob/HEAD/training/modules.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1c8cd1745fa5a3e"}},{"code_sha256_prefix":"8aa1dec7ce5bdeda","entry":"read_file","repo":"minzwon/sota-music-tagging-models","repo_kind":"official","path":"training/robustness.py","file_url":"https://github.com/minzwon/sota-music-tagging-models/blob/HEAD/training/robustness.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aa1dec7ce5bdeda"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}