{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2408-00005","title":"Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polish","arxiv_id":"2408.00005","date":"2024-07-18","proceeding":null,"authors":["Michał Junczyk"],"abstract":"Speech datasets available in the public domain are often underutilized because of challenges in discoverability and interoperability. A comprehensive framework has been designed to survey, catalog, and curate available speech datasets, which allows replicable evaluation of automatic speech recognition (ASR) systems. A case study focused on the Polish language was conducted; the framework was applied to curate more than 24 datasets and evaluate 25 combinations of ASR systems and models. This research constitutes the most extensive comparison to date of both commercial and free ASR systems for the Polish language. It draws insights from 600 system-model-test set evaluations, marking a significant advancement in both scale and comprehensiveness. The results of surveys and performance comparisons are available as interactive dashboards (https://huggingface.co/spaces/amu-cai/pl-asr-leaderboard) along with curated datasets (https://huggingface.co/datasets/amu-cai/pl-asr-bigos-v2, https://huggingface.co/datasets/pelcra/pl-asr-pelcra-for-bigos) and the open challenge call (https://poleval.pl/tasks/task3). Tools used for evaluation are open-sourced (https://github.com/goodmike31/pl-asr-bigos-tools), facilitating replication and adaptation for other languages, as well as continuous expansion with new datasets and systems.","url_abs":"https://arxiv.org/abs/2408.00005v1","url_pdf":"https://arxiv.org/pdf/2408.00005v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2408-00005","repo_url":"https://github.com/goodmike31/pl-asr-bigos-tools","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"automatic-speech-recognition-2","task_name":"Automatic Speech Recognition"},{"task_slug":"automatic-speech-recognition","task_name":"Automatic Speech Recognition (ASR)"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.00005","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.00005"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mlcommons/croissant","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/goodmike31/pl-asr-bigos-tools","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e423ba6be5105f77","entry":"find_unique_name","repo":"mlcommons/croissant","repo_kind":"found_in_text","path":"editor/core/names.py","file_url":"https://github.com/mlcommons/croissant/blob/HEAD/editor/core/names.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e423ba6be5105f77"}},{"code_sha256_prefix":"90cd1aff1fc1b103","entry":"hash_file_path","repo":"mlcommons/croissant","repo_kind":"found_in_text","path":"editor/core/files.py","file_url":"https://github.com/mlcommons/croissant/blob/HEAD/editor/core/files.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90cd1aff1fc1b103"}},{"code_sha256_prefix":"5bf8e3ac4b615151","entry":"validate_json","repo":"mlcommons/croissant","repo_kind":"found_in_text","path":"health/croissant-validator-neurips/hf-gradio-app/validation.py","file_url":"https://github.com/mlcommons/croissant/blob/HEAD/health/croissant-validator-neurips/hf-gradio-app/validation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bf8e3ac4b615151"}},{"code_sha256_prefix":"5d75db6cfccb9203","entry":"PAST_PROJECTS_PATH","repo":"mlcommons/croissant","repo_kind":"found_in_text","path":"editor/core/constants.py","file_url":"https://github.com/mlcommons/croissant/blob/HEAD/editor/core/constants.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d75db6cfccb9203"}},{"code_sha256_prefix":"b3ed4f03d921e5e0","entry":"code_to_index","repo":"mlcommons/croissant","repo_kind":"found_in_text","path":"editor/core/files.py","file_url":"https://github.com/mlcommons/croissant/blob/HEAD/editor/core/files.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b3ed4f03d921e5e0"}},{"code_sha256_prefix":"06f7abfbd5210f6b","entry":"name_to_code","repo":"mlcommons/croissant","repo_kind":"found_in_text","path":"editor/core/files.py","file_url":"https://github.com/mlcommons/croissant/blob/HEAD/editor/core/files.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06f7abfbd5210f6b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}