{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/low-resource-text-classification-with-ulmfit","title":"Low Resource Text Classification with ULMFit and Backtranslation","arxiv_id":"1903.09244","date":"2019-03-21","proceeding":null,"authors":["Sam Shleifer"],"abstract":"In computer vision, virtually every state-of-the-art deep learning system is\ntrained with data augmentation. In text classification, however, data\naugmentation is less widely practiced because it must be performed before\ntraining and risks introducing label noise. We augment the IMDB movie reviews\ndataset with examples generated by two families of techniques: random token\nperturbations introduced by Wei and Zou [2019] and backtranslation --\ntranslating to a second language then back to English. In low resource\nenvironments, backtranslation generates significant improvement on top of the\nstate of-the-art ULMFit model. A ULMFit model pretrained on wikitext103 and\nthen fine-tuned on only 50 IMDB examples and 500 synthetic examples generated\nby backtranslation achieves 80.6% accuracy, an 8.1% improvement over the\naugmentation-free baseline with only 9 minutes of additional training time.\nRandom token perturbations do not yield any improvements but incur equivalent\ncomputational cost. The benefits of training with backtranslated examples\ndecreases with the size of the available training data. On the full dataset,\nneither augmentation technique improves upon ULMFit's state of the art\nperformance. We address this by using backtranslations as a form of test time\naugmentation as well as ensembling ULMFit with other models, and achieve small\nimprovements.","url_abs":"http://arxiv.org/abs/1903.09244v2","url_pdf":"http://arxiv.org/pdf/1903.09244v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"low-resource-text-classification-with-ulmfit","repo_url":"https://github.com/oraby8/TextDataAug","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"awd-lstm","method_name":"AWD-LSTM"},{"method_slug":"activation-regularization","method_name":"Activation Regularization"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropconnect","method_name":"DropConnect"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"embedding-dropout","method_name":"Embedding Dropout"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"slanted-triangular-learning-rates","method_name":"Slanted Triangular Learning Rates"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"temporal-activation-regularization","method_name":"Temporal Activation Regularization"},{"method_slug":"ulmfit","method_name":"ULMFiT"},{"method_slug":"variational-dropout","method_name":"Variational Dropout"},{"method_slug":"weight-tying","method_name":"Weight Tying"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1903.09244","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}