{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-preprocessing-of-text-really-worth-your","title":"Is preprocessing of text really worth your time for online comment classification?","arxiv_id":"1806.02908","date":"2018-06-07","proceeding":null,"authors":["Fahim Mohammad"],"abstract":"A large proportion of online comments present on public domains are\nconstructive, however a significant proportion are toxic in nature. The\ncomments contain lot of typos which increases the number of features manifold,\nmaking the ML model difficult to train. Considering the fact that the data\nscientists spend approximately 80% of their time in collecting, cleaning and\norganizing their data [1], we explored how much effort should we invest in the\npreprocessing (transformation) of raw comments before feeding it to the\nstate-of-the-art classification models. With the help of four models on Jigsaw\ntoxic comment classification data, we demonstrated that the training of model\nwithout any transformation produce relatively decent model. Applying even basic\ntransformations, in some cases, lead to worse performance and should be applied\nwith caution.","url_abs":"http://arxiv.org/abs/1806.02908v2","url_pdf":"http://arxiv.org/pdf/1806.02908v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-preprocessing-of-text-really-worth-your","repo_url":"https://github.com/ifahim/toxic-preprocess","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"toxic-comment-classification","task_name":"Toxic Comment Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}