{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-modeling-is-compression","title":"Language Modeling Is Compression","arxiv_id":"2309.10668","date":"2023-09-19","proceeding":null,"authors":["Grégoire Delétang","Anian Ruoss","Paul-Ambroise Duquenne","Elliot Catt","Tim Genewein","Christopher Mattern","Jordi Grau-Moya","Li Kevin Wenliang","Matthew Aitchison","Laurent Orseau","Marcus Hutter","Joel Veness"],"abstract":"It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Finally, we show that the prediction-compression equivalence allows us to use any compressor (like gzip) to build a conditional generative model.","url_abs":"https://arxiv.org/abs/2309.10668v2","url_pdf":"https://arxiv.org/pdf/2309.10668v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-modeling-is-compression","repo_url":"https://github.com/google-deepmind/language_modeling_is_compression","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"chinchilla","method_name":"Chinchilla"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.10668","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.10668"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/language_modeling_is_compression","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"63e2ee5b6dd1c819","entry":"bits_to_bytes","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"utils.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63e2ee5b6dd1c819"}},{"code_sha256_prefix":"1100deb3c14e1b2b","entry":"bytes_to_bits","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"utils.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1100deb3c14e1b2b"}},{"code_sha256_prefix":"a97701702e1a2e67","entry":"compress","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"compressors/png.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/compressors/png.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a97701702e1a2e67"}},{"code_sha256_prefix":"b9d52dfa29c79c4b","entry":"decompress","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"compressors/png.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/compressors/png.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9d52dfa29c79c4b"}},{"code_sha256_prefix":"0f261d001735e36f","entry":"right_shift_bytes_by_one","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"utils.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f261d001735e36f"}},{"code_sha256_prefix":"f915c59353ea23e2","entry":"sinusoid_position_encoding","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"transformer.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f915c59353ea23e2"}},{"code_sha256_prefix":"ee725f97222e1399","entry":"embed_sequences","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"transformer.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee725f97222e1399"}},{"code_sha256_prefix":"a145acc7af542d06","entry":"layer_norm","repo":"google-deepmind/language_modeling_is_compression","repo_kind":"official","path":"transformer.py","file_url":"https://github.com/google-deepmind/language_modeling_is_compression/blob/HEAD/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a145acc7af542d06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}