{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fine-tuning-self-supervised-learning-models","title":"Fine-Tuning Self-Supervised Learning Models for End-to-End Pronunciation Scoring","arxiv_id":null,"date":"2023-09-19","proceeding":"IEEE Access 2023 9","authors":["Ahmed I. Zahran","Aly A. Fahmy","Khaled T. Wassif","Hanaa Bayomi"],"abstract":"Automatic pronunciation assessment models are regularly used in language learning applications. Common methodologies for pronunciation assessment use feature-based approaches, such as the Goodness-of-Pronunciation (GOP) approach, or deep learning speech recognition models to perform speech assessment. With the rise of transformers, pre-trained self-supervised learning (SSL) models have been utilized to extract contextual speech representations, showing improvements in various downstream tasks. In this study, we propose the end-to-end regressor (E2E-R) model for pronunciation scoring. E2E-R is trained using a two-step training process. In the first step, the pre-trained SSL model is fine-tuned on a phoneme recognition task to obtain better representations for the pronounced phonemes. In the second step, transfer learning is used to build a pronunciation scoring model that uses a Siamese neural network to compare the pronounced phoneme representations to embeddings of the canonical phonemes and produce the final pronunciation scores. E2E-R achieves a Pearson correlation coefficient (PCC) of 0.68, which is almost similar to the state-of-the-art GOPT-PAII model while eliminating the need for training on additional native speech data, feature engineering, or external forced alignment modules. To our knowledge, this work presents the first utilization of a pre-trained SSL model for end-to-end phoneme-level pronunciation scoring on raw speech waveforms.","url_abs":"https://ieeexplore.ieee.org/document/10255657","url_pdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10255657","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fine-tuning-self-supervised-learning-models","repo_url":"https://github.com/ai-zahran/E2E-R","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"feature-engineering","task_name":"Feature Engineering"},{"task_slug":"phone-level-pronunciation-scoring","task_name":"Phone-level pronunciation scoring"},{"task_slug":"phoneme-recognition","task_name":"Phoneme Recognition"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/phone-level-pronunciation-scoring-on","task":"Phone-level pronunciation scoring","dataset":"speechocean762","model":"E2E-R","rank_in_archive_order":4,"of":8,"metrics":{"Pearson correlation coefficient (PCC)":"0.68"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}