{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-deterministic-information-bottleneck","title":"The deterministic information bottleneck","arxiv_id":"1604.00268","date":"2016-04-01","proceeding":null,"authors":["DJ Strouse","David J. Schwab"],"abstract":"Lossy compression and clustering fundamentally involve a decision about what\nfeatures are relevant and which are not. The information bottleneck method (IB)\nby Tishby, Pereira, and Bialek formalized this notion as an\ninformation-theoretic optimization problem and proposed an optimal tradeoff\nbetween throwing away as many bits as possible, and selectively keeping those\nthat are most important. In the IB, compression is measure my mutual\ninformation. Here, we introduce an alternative formulation that replaces mutual\ninformation with entropy, which we call the deterministic information\nbottleneck (DIB), that we argue better captures this notion of compression. As\nsuggested by its name, the solution to the DIB problem turns out to be a\ndeterministic encoder, or hard clustering, as opposed to the stochastic\nencoder, or soft clustering, that is optimal under the IB. We compare the IB\nand DIB on synthetic data, showing that the IB and DIB perform similarly in\nterms of the IB cost function, but that the DIB significantly outperforms the\nIB in terms of the DIB cost function. We also empirically find that the DIB\noffers a considerable gain in computational efficiency over the IB, over a\nrange of convergence parameters. Our derivation of the DIB also suggests a\nmethod for continuously interpolating between the soft clustering of the IB and\nthe hard clustering of the DIB.","url_abs":"http://arxiv.org/abs/1604.00268v2","url_pdf":"http://arxiv.org/pdf/1604.00268v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-deterministic-information-bottleneck","repo_url":"https://github.com/djstrouse/information-bottleneck","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"the-deterministic-information-bottleneck","repo_url":"https://github.com/johncwok/IntegerIB.jl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1604.00268","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1604.00268"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/johncwok/IntegerIB.jl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/djstrouse/information-bottleneck","reach":null}],"summary":{"ran_fixture":2,"ran_honours":1},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"f825bb885d6e69b7","entry":"entropy","repo":"djstrouse/information-bottleneck","repo_kind":"listed","path":"IB.py","file_url":"https://github.com/djstrouse/information-bottleneck/blob/HEAD/IB.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f825bb885d6e69b7"}},{"code_sha256_prefix":"76cdbf862a129aba","entry":"entropy_single","repo":"djstrouse/information-bottleneck","repo_kind":"listed","path":"IB.py","file_url":"https://github.com/djstrouse/information-bottleneck/blob/HEAD/IB.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76cdbf862a129aba"}},{"code_sha256_prefix":"381aeef78431d7f4","entry":"entropy_term","repo":"djstrouse/information-bottleneck","repo_kind":"listed","path":"IB.py","file_url":"https://github.com/djstrouse/information-bottleneck/blob/HEAD/IB.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"381aeef78431d7f4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}