{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/versa-a-versatile-evaluation-toolkit-for","title":"VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music","arxiv_id":"2412.17667","date":"2024-12-23","proceeding":null,"authors":["Jiatong Shi","Hye-jin Shim","Jinchuan Tian","Siddhant Arora","Haibin Wu","Darius Petermann","Jia Qi Yip","You Zhang","Yuxun Tang","Wangyou Zhang","Dareen Safar Alharthi","Yichen Huang","Koichi Saito","Jionghao Han","Yiwen Zhao","Chris Donahue","Shinji Watanabe"],"abstract":"In this work, we introduce VERSA, a unified and standardized evaluation toolkit designed for various speech, audio, and music signals. The toolkit features a Pythonic interface with flexible configuration and dependency control, making it user-friendly and efficient. With full installation, VERSA offers 65 metrics with 729 metric variations based on different configurations. These metrics encompass evaluations utilizing diverse external resources, including matching and non-matching reference audio, text transcriptions, and text captions. As a lightweight yet comprehensive toolkit, VERSA is versatile to support the evaluation of a wide range of downstream scenarios. To demonstrate its capabilities, this work highlights example use cases for VERSA, including audio coding, speech synthesis, speech enhancement, singing synthesis, and music generation. The toolkit is available at https://github.com/wavlab-speech/versa.","url_abs":"https://arxiv.org/abs/2412.17667v2","url_pdf":"https://arxiv.org/pdf/2412.17667v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"versa-a-versatile-evaluation-toolkit-for","repo_url":"https://github.com/shinjiwlab/versa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"versa-a-versatile-evaluation-toolkit-for","repo_url":"https://github.com/wavlab-speech/versa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.17667","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.17667"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shinjiwlab/versa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wavlab-speech/versa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"32cf51053a658a30","entry":"clap_score_setup","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/corpus_metrics/clap_score.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/corpus_metrics/clap_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32cf51053a658a30"}},{"code_sha256_prefix":"de3889319ca5781b","entry":"configure_huggingface_cache","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/huggingface_cache.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/huggingface_cache.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de3889319ca5781b"}},{"code_sha256_prefix":"872a9c1646a41b23","entry":"configure_metric_cache_dirs","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/scorer_shared.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/scorer_shared.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"872a9c1646a41b23"}},{"code_sha256_prefix":"0c9a871210ca3944","entry":"espnet_wer_setup","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/corpus_metrics/espnet_wer.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/corpus_metrics/espnet_wer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c9a871210ca3944"}},{"code_sha256_prefix":"57457986d497f352","entry":"get_hf_cache_dir","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/huggingface_cache.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/huggingface_cache.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57457986d497f352"}},{"code_sha256_prefix":"eaa5e87b8b601541","entry":"load_metric_symbol","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/metric_registry.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/metric_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eaa5e87b8b601541"}},{"code_sha256_prefix":"61f7a391270c67fe","entry":"load_metric_symbols","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/metric_registry.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/metric_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"61f7a391270c67fe"}},{"code_sha256_prefix":"b9f51a2672f708c7","entry":"model_snapshot_has_file","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/huggingface_cache.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/huggingface_cache.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9f51a2672f708c7"}},{"code_sha256_prefix":"0e28ba1be918cb2d","entry":"resample_audio","repo":"shinjiwlab/versa","repo_kind":"official","path":"versa/audio_utils.py","file_url":"https://github.com/shinjiwlab/versa/blob/HEAD/versa/audio_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e28ba1be918cb2d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}