{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lit-learned-intermediate-representation","title":"LIT: Learned Intermediate Representation Training for Model Compression","arxiv_id":null,"date":"2019-09-04","proceeding":null,"authors":["Animesh Koratana","Daniel Kang","Peter Bailis","Matei Zaharia"],"abstract":"Researchers have proposed a range of model compression techniques to reduce the computational\r\nand memory footprint of deep neural networks\r\n(DNNs). In this work, we introduce Learned\r\nIntermediate representation Training (LIT), a\r\nnovel model compression technique that outperforms a range of recent model compression techniques by leveraging the highly repetitive structure of modern DNNs (e.g., ResNet). LIT uses a\r\nteacher DNN to train a student DNN of reduced\r\ndepth by leveraging two key ideas: 1) LIT directly compares intermediate representations of\r\nthe teacher and student model and 2) LIT uses\r\nthe intermediate representation from the teacher\r\nmodel’s previous block as input to the current\r\nstudent block during training, improving stability of intermediate representations in the student\r\nnetwork. We show that LIT can substantially reduce network size without loss in accuracy on a\r\nrange of DNN architectures and datasets. For example, LIT can compress ResNet on CIFAR10 by\r\n3.4× outperforming network slimming and FitNets. Furthermore, LIT can compress, by depth,\r\nResNeXt 5.5× on CIFAR10 (image classification), VDCNN by 1.7× on Amazon Reviews\r\n(sentiment analysis), and StarGAN by 1.8× on\r\nCelebA (style transfer, i.e., GANs).","url_abs":"http://proceedings.mlr.press/v97/koratana19a/koratana19a.pdf","url_pdf":"http://proceedings.mlr.press/v97/koratana19a/koratana19a.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lit-learned-intermediate-representation","repo_url":"https://github.com/stanford-futuredata/lit-code","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"style-transfer","task_name":"Style Transfer"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}