{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/text-classification-with-word-embedding","title":"Text classification with word embedding regularization and soft similarity measure","arxiv_id":"2003.05019","date":"2020-03-10","proceeding":null,"authors":["Vít Novotný","Eniafe Festus Ayetiran","Michal Štefánik","Petr Sojka"],"abstract":"Since the seminal work of Mikolov et al., word embeddings have become the preferred word representations for many natural language processing tasks. Document similarity measures extracted from word embeddings, such as the soft cosine measure (SCM) and the Word Mover's Distance (WMD), were reported to achieve state-of-the-art performance on semantic text similarity and text classification. Despite the strong performance of the WMD on text classification and semantic text similarity, its super-cubic average time complexity is impractical. The SCM has quadratic worst-case time complexity, but its performance on text classification has never been compared with the WMD. Recently, two word embedding regularization techniques were shown to reduce storage and memory costs, and to improve training speed, document processing speed, and task performance on word analogy, word similarity, and semantic text similarity. However, the effect of these techniques on text classification has not yet been studied. In our work, we investigate the individual and joint effect of the two word embedding regularization techniques on the document processing speed and the task performance of the SCM and the WMD on text classification. For evaluation, we use the $k$NN classifier and six standard datasets: BBCSPORT, TWITTER, OHSUMED, REUTERS-21578, AMAZON, and 20NEWS. We show 39% average $k$NN test error reduction with regularized word embeddings compared to non-regularized word embeddings. We describe a practical procedure for deriving such regularized embeddings through Cholesky factorization. We also show that the SCM with regularized word embeddings significantly outperforms the WMD on text classification and is over 10,000 times faster.","url_abs":"https://arxiv.org/abs/2003.05019v1","url_pdf":"https://arxiv.org/pdf/2003.05019v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"text-classification-with-word-embedding","repo_url":"https://github.com/MIR-MU/regularized-embeddings","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"document-classification","task_name":"Document Classification"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"},{"task_slug":"word-similarity","task_name":"Word Similarity"},{"task_slug":"text-similarity","task_name":"text similarity"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/document-classification-on-amazon","task":"Document Classification","dataset":"Amazon","model":"Orthogonalized Soft VSM","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy":"93.42"},"uses_additional_data":false},{"leaderboard":"/sota/document-classification-on-bbcsport","task":"Document Classification","dataset":"BBCSport","model":"Orthogonalized Soft VSM","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy":"97.73"},"uses_additional_data":false},{"leaderboard":"/sota/document-classification-on-reuters-21578","task":"Document Classification","dataset":"Reuters-21578","model":"Orthogonalized Soft VSM","rank_in_archive_order":3,"of":8,"metrics":{"Accuracy":"92.65"},"uses_additional_data":false},{"leaderboard":"/sota/document-classification-on-twitter","task":"Document Classification","dataset":"Twitter","model":"Orthogonalized Soft VSM","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"69.21"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-20news","task":"Text Classification","dataset":"20NEWS","model":"Orthogonalized Soft VSM","rank_in_archive_order":16,"of":16,"metrics":{"Accuracy":"70.28"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}