{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/one-teacher-and-multiple-student-knowledge","title":"One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification","arxiv_id":null,"date":"2022-10-01","proceeding":"COLING 2022 10","authors":["Xiaoqin Chang","Sophia Yat Mei Lee","Suyang Zhu","Shoushan Li","Guodong Zhou"],"abstract":"Knowledge distillation is an effective method to transfer knowledge from a large pre-trained teacher model to a compacted student model. However, in previous studies, the distilled student models are still large and remain impractical in highly speed-sensitive systems (e.g., an IR system). In this study, we aim to distill a deep pre-trained model into an extremely compacted shallow model like CNN. Specifically, we propose a novel one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning. Moreover, we leverage large-scale unlabeled data to improve the performance of students. Empirical studies on three sentiment classification tasks demonstrate that our approach achieves better results with much fewer parameters (0.9%-18%) and extremely high speedup ratios (100X-1000X).","url_abs":"https://aclanthology.org/2022.coling-1.614","url_pdf":"https://aclanthology.org/2022.coling-1.614.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"one-teacher-and-multiple-student-knowledge","repo_url":"https://github.com/strive-hhh/otms-kd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"ensemble-learning","task_name":"Ensemble Learning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"sentiment-classification","task_name":"Sentiment Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}