{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-zero-resource-speech-benchmark-2021","title":"The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling","arxiv_id":"2011.11588","date":"2020-11-23","proceeding":null,"authors":["Tu Anh Nguyen","Maureen de Seyssel","Patricia Rozé","Morgane Rivière","Evgeny Kharitonov","Alexei Baevski","Ewan Dunbar","Emmanuel Dupoux"],"abstract":"We introduce a new unsupervised task, spoken language modeling: the learning of linguistic representations from raw audio signals without any labels, along with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot metrics probing for the quality of the learned models at 4 linguistic levels: phonetics, lexicon, syntax and semantics. We present the results and analyses of a composite baseline made of the concatenation of three unsupervised systems: self-supervised contrastive representation learning (CPC), clustering (k-means) and language modeling (LSTM or BERT). The language models learn on the basis of the pseudo-text derived from clustering the learned representations. This simple pipeline shows better than chance performance on all four metrics, demonstrating the feasibility of spoken language modeling from raw speech. It also yields worse performance compared to text-based 'topline' systems trained on the same data, delineating the space to be explored by more sophisticated end-to-end models.","url_abs":"https://arxiv.org/abs/2011.11588v2","url_pdf":"https://arxiv.org/pdf/2011.11588v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-zero-resource-speech-benchmark-2021","repo_url":"https://github.com/bootphon/zerospeech2021_baseline","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"the-zero-resource-speech-benchmark-2021","repo_url":"https://github.com/zerospeech/zerospeech2021_baseline","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2011.11588","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2011.11588"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zerospeech/zerospeech2021_baseline","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bootphon/zerospeech2021_baseline","reach":{"status":"unanswered"}}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"listed":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"742ee8f06d3b0ce8","entry":"quantize_file","repo":"zerospeech/zerospeech2021_baseline","repo_kind":"listed","path":"scripts/quantize_audio.py","file_url":"https://github.com/zerospeech/zerospeech2021_baseline/blob/HEAD/scripts/quantize_audio.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"742ee8f06d3b0ce8"}},{"code_sha256_prefix":"6ba9e5c2207d36b0","entry":"parseArgs","repo":"zerospeech/zerospeech2021_baseline","repo_kind":"listed","path":"scripts/quantize_audio.py","file_url":"https://github.com/zerospeech/zerospeech2021_baseline/blob/HEAD/scripts/quantize_audio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ba9e5c2207d36b0"}},{"code_sha256_prefix":"811d25182ae201a2","entry":"parseArgs","repo":"zerospeech/zerospeech2021_baseline","repo_kind":"listed","path":"scripts/compute_proba_BERT.py","file_url":"https://github.com/zerospeech/zerospeech2021_baseline/blob/HEAD/scripts/compute_proba_BERT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"811d25182ae201a2"}},{"code_sha256_prefix":"92fedaf30665778b","entry":"parseArgs","repo":"zerospeech/zerospeech2021_baseline","repo_kind":"listed","path":"scripts/build_BERT_features.py","file_url":"https://github.com/zerospeech/zerospeech2021_baseline/blob/HEAD/scripts/build_BERT_features.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92fedaf30665778b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}