{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cauchy-schwarz-divergence-information","title":"Cauchy-Schwarz Divergence Information Bottleneck for Regression","arxiv_id":"2404.17951","date":"2024-04-27","proceeding":null,"authors":["Shujian Yu","Xi Yu","Sigurd Løkse","Robert Jenssen","Jose C. Principe"],"abstract":"The information bottleneck (IB) approach is popular to improve the generalization, robustness and explainability of deep neural networks. Essentially, it aims to find a minimum sufficient representation $\\mathbf{t}$ by striking a trade-off between a compression term $I(\\mathbf{x};\\mathbf{t})$ and a prediction term $I(y;\\mathbf{t})$, where $I(\\cdot;\\cdot)$ refers to the mutual information (MI). MI is for the IB for the most part expressed in terms of the Kullback-Leibler (KL) divergence, which in the regression case corresponds to prediction based on mean squared error (MSE) loss with Gaussian assumption and compression approximated by variational inference. In this paper, we study the IB principle for the regression problem and develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Schwarz (CS) divergence. By doing so, we move away from MSE-based regression and ease estimation by avoiding variational approximations or distributional assumptions. We investigate the improved generalization ability of our proposed CS-IB and demonstrate strong adversarial robustness guarantees. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches. We additionally observe that the solutions discovered by CS-IB always achieve the best trade-off between prediction accuracy and compression ratio in the information plane. The code is available at \\url{https://github.com/SJYuCNEL/Cauchy-Schwarz-Information-Bottleneck}.","url_abs":"https://arxiv.org/abs/2404.17951v1","url_pdf":"https://arxiv.org/pdf/2404.17951v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cauchy-schwarz-divergence-information","repo_url":"https://github.com/sjyucnel/cauchy-schwarz-information-bottleneck","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"information-plane","task_name":"Information Plane"},{"task_slug":"variational-inference","task_name":"Variational Inference"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.17951","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.17951"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sjyucnel/cauchy-schwarz-information-bottleneck","reach":null}],"summary":{"ran_honours":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"92ca7faf67a9928f","entry":"GaussianMatrix","repo":"sjyucnel/cauchy-schwarz-information-bottleneck","repo_kind":"official","path":"loss_cs.py","file_url":"https://github.com/sjyucnel/cauchy-schwarz-information-bottleneck/blob/HEAD/loss_cs.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92ca7faf67a9928f"}},{"code_sha256_prefix":"ca565da6831e8ade","entry":"distmat","repo":"sjyucnel/cauchy-schwarz-information-bottleneck","repo_kind":"official","path":"loss_cs.py","file_url":"https://github.com/sjyucnel/cauchy-schwarz-information-bottleneck/blob/HEAD/loss_cs.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca565da6831e8ade"}},{"code_sha256_prefix":"20b89645c8dfbf6d","entry":"sigma_estimation","repo":"sjyucnel/cauchy-schwarz-information-bottleneck","repo_kind":"official","path":"loss_cs.py","file_url":"https://github.com/sjyucnel/cauchy-schwarz-information-bottleneck/blob/HEAD/loss_cs.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20b89645c8dfbf6d"}},{"code_sha256_prefix":"5cf5bbeda93532d6","entry":"get_california_housing","repo":"sjyucnel/cauchy-schwarz-information-bottleneck","repo_kind":"official","path":"CS_IB_Housing.py","file_url":"https://github.com/sjyucnel/cauchy-schwarz-information-bottleneck/blob/HEAD/CS_IB_Housing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cf5bbeda93532d6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}