{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distilling-model-knowledge","title":"Distilling Model Knowledge","arxiv_id":"1510.02437","date":"2015-10-08","proceeding":null,"authors":["George Papamakarios"],"abstract":"Top-performing machine learning systems, such as deep neural networks, large\nensembles and complex probabilistic graphical models, can be expensive to\nstore, slow to evaluate and hard to integrate into larger systems. Ideally, we\nwould like to replace such cumbersome models with simpler models that perform\nequally well.\n  In this thesis, we study knowledge distillation, the idea of extracting the\nknowledge contained in a complex model and injecting it into a more convenient\nmodel. We present a general framework for knowledge distillation, whereby a\nconvenient model of our choosing learns how to mimic a complex model, by\nobserving the latter's behaviour and being penalized whenever it fails to\nreproduce it.\n  We develop our framework within the context of three distinct machine\nlearning applications: (a) model compression, where we compress large\ndiscriminative models, such as ensembles of neural networks, into models of\nmuch smaller size; (b) compact predictive distributions for Bayesian inference,\nwhere we distil large bags of MCMC samples into compact predictive\ndistributions in closed form; (c) intractable generative models, where we\ndistil unnormalizable models such as RBMs into tractable models such as NADEs.\n  We contribute to the state of the art with novel techniques and ideas. In\nmodel compression, we describe and implement derivative matching, which allows\nfor better distillation when data is scarce. In compact predictive\ndistributions, we introduce online distillation, which allows for significant\nsavings in memory. Finally, in intractable generative models, we show how to\nuse distilled models to robustly estimate intractable quantities of the\noriginal model, such as its intractable partition function.","url_abs":"http://arxiv.org/abs/1510.02437v1","url_pdf":"http://arxiv.org/pdf/1510.02437v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distilling-model-knowledge","repo_url":"https://github.com/gpapamak/distilling_model_knowledge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}