{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/minimum-description-length-and-generalization-1","title":"Minimum Description Length and Generalization Guarantees for Representation Learning","arxiv_id":"2402.03254","date":"2024-02-05","proceeding":"NeurIPS 2023 11","authors":["Milad Sefidgaran","Abdellatif Zaidi","Piotr Krasnowski"],"abstract":"A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on unseen data. While the study of representation learning has spurred much interest, most existing such approaches are heuristic; and very little is known about theoretical generalization guarantees. In this paper, we establish a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm in terms of the \"Minimum Description Length\" (MDL) of the labels or the latent variables (representations). Rather than the mutual information between the encoder's input and the representation, which is often believed to reflect the algorithm's generalization capability in the related literature but in fact, falls short of doing so, our new bounds involve the \"multi-letter\" relative entropy between the distribution of the representations (or labels) of the training and test sets and a fixed prior. In particular, these new bounds reflect the structure of the encoder and are not vacuous for deterministic algorithms. Our compressibility approach, which is information-theoretic in nature, builds upon that of Blum-Langford for PAC-MDL bounds and introduces two essential ingredients: block-coding and lossy-compression. The latter allows our approach to subsume the so-called geometrical compressibility as a special case. To the best knowledge of the authors, the established generalization bounds are the first of their kind for Information Bottleneck (IB) type encoders and representation learning. Finally, we partly exploit the theoretical results by introducing a new data-dependent prior. Numerical simulations illustrate the advantages of well-chosen such priors over classical priors used in IB.","url_abs":"https://arxiv.org/abs/2402.03254v1","url_pdf":"https://arxiv.org/pdf/2402.03254v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"minimum-description-length-and-generalization-1","repo_url":"https://github.com/piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.03254","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.03254"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"558fdb5c17a3a411","entry":"cuda","repo":"piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning","repo_kind":"official","path":"utils.py","file_url":"https://github.com/piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"558fdb5c17a3a411"}},{"code_sha256_prefix":"e379ac8503ee2660","entry":"str2bool","repo":"piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning","repo_kind":"official","path":"utils.py","file_url":"https://github.com/piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e379ac8503ee2660"}},{"code_sha256_prefix":"05d0b6602a883bd3","entry":"return_data","repo":"piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/piotrkrasnowski/mdl_and_generalization_guarantees_for_representation_learning/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05d0b6602a883bd3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}