{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-compression-via-distillation-and","title":"Model compression via distillation and quantization","arxiv_id":"1802.05668","date":"2018-02-15","proceeding":"ICLR 2018 1","authors":["Antonio Polino","Razvan Pascanu","Dan Alistarh"],"abstract":"Deep neural networks (DNNs) continue to make significant advances, solving\ntasks from image classification to translation or reinforcement learning. One\naspect of the field receiving considerable attention is efficiently executing\ndeep models in resource-constrained environments, such as mobile or embedded\ndevices. This paper focuses on this problem, and proposes two new compression\nmethods, which jointly leverage weight quantization and distillation of larger\nteacher networks into smaller student networks. The first method we propose is\ncalled quantized distillation and leverages distillation during the training\nprocess, by incorporating distillation loss, expressed with respect to the\nteacher, into the training of a student network whose weights are quantized to\na limited set of levels. The second method, differentiable quantization,\noptimizes the location of quantization points through stochastic gradient\ndescent, to better fit the behavior of the teacher model. We validate both\nmethods through experiments on convolutional and recurrent architectures. We\nshow that quantized shallow students can reach similar accuracy levels to\nfull-precision teacher models, while providing order of magnitude compression,\nand inference speedup that is linear in the depth reduction. In sum, our\nresults enable DNNs for resource-constrained environments to leverage\narchitecture and accuracy advances developed on more powerful devices.","url_abs":"http://arxiv.org/abs/1802.05668v1","url_pdf":"http://arxiv.org/pdf/1802.05668v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-compression-via-distillation-and","repo_url":"https://github.com/antspy/quantized_distillation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"model-compression-via-distillation-and","repo_url":"https://github.com/kuanzi/Bayesian_quant","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"model-compression-via-distillation-and","repo_url":"https://github.com/mzymzy/paper3-quantized_distillzation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"model-compression-via-distillation-and","repo_url":"https://github.com/pra-dan/Worth-Reading-ML-Papers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"model-compression-via-distillation-and","repo_url":"https://github.com/NervanaSystems/distiller","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"tasks":[{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1802.05668","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.05668"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kuanzi/Bayesian_quant","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pra-dan/Worth-Reading-ML-Papers","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/antspy/quantized_distillation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mzymzy/paper3-quantized_distillzation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NervanaSystems/distiller","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"antspy/quantized_distillation","repo_kind":"official","path":"cnn_models/resnet_kfilters.py","file_url":"https://github.com/antspy/quantized_distillation/blob/HEAD/cnn_models/resnet_kfilters.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"00e569acd6b45ef0","entry":"conv3x3","repo":"antspy/quantized_distillation","repo_kind":"official","path":"cnn_models/wide_resnet.py","file_url":"https://github.com/antspy/quantized_distillation/blob/HEAD/cnn_models/wide_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00e569acd6b45ef0"}},{"code_sha256_prefix":"977f190c6c245190","entry":"evaluateModel","repo":"antspy/quantized_distillation","repo_kind":"official","path":"cnn_models/help_fun.py","file_url":"https://github.com/antspy/quantized_distillation/blob/HEAD/cnn_models/help_fun.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"977f190c6c245190"}},{"code_sha256_prefix":"aea739de153a41a8","entry":"time_forward_pass","repo":"antspy/quantized_distillation","repo_kind":"official","path":"cnn_models/help_fun.py","file_url":"https://github.com/antspy/quantized_distillation/blob/HEAD/cnn_models/help_fun.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aea739de153a41a8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}