{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/topicbert-for-energy-efficient-document","title":"TopicBERT for Energy Efficient Document Classification","arxiv_id":"2010.16407","date":"2020-10-15","proceeding":"Findings of the Association for Computational Linguistics 2020","authors":["Yatin Chaudhary","Pankaj Gupta","Khushbu Saxena","Vivek Kulkarni","Thomas Runkler","Hinrich Schütze"],"abstract":"Prior research notes that BERT's computational cost grows quadratically with sequence length thus leading to longer training times, higher GPU memory constraints and carbon emissions. While recent work seeks to address these scalability issues at pre-training, these issues are also prominent in fine-tuning especially for long sequence tasks like document classification. Our work thus focuses on optimizing the computational cost of fine-tuning for document classification. We achieve this by complementary learning of both topic and language models in a unified framework, named TopicBERT. This significantly reduces the number of self-attention operations - a main performance bottleneck. Consequently, our model achieves a 1.4x ($\\sim40\\%$) speedup with $\\sim40\\%$ reduction in $CO_2$ emission while retaining $99.9\\%$ performance over 5 datasets.","url_abs":"https://arxiv.org/abs/2010.16407v1","url_pdf":"https://arxiv.org/pdf/2010.16407v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"topicbert-for-energy-efficient-document","repo_url":"https://github.com/YatinChaudhary/TopicBERT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"document-classification","task_name":"Document Classification"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"classification","task_name":"General Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.16407","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}