{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/opening-the-black-box-of-deep-neural-networks","title":"Opening the Black Box of Deep Neural Networks via Information","arxiv_id":"1703.00810","date":"2017-03-02","proceeding":null,"authors":["Ravid Shwartz-Ziv","Naftali Tishby"],"abstract":"Despite their great success, there is still no comprehensive theoretical\nunderstanding of learning with Deep Neural Networks (DNNs) or their inner\norganization. Previous work proposed to analyze DNNs in the \\textit{Information\nPlane}; i.e., the plane of the Mutual Information values that each layer\npreserves on the input and output variables. They suggested that the goal of\nthe network is to optimize the Information Bottleneck (IB) tradeoff between\ncompression and prediction, successively, for each layer.\n  In this work we follow up on this idea and demonstrate the effectiveness of\nthe Information-Plane visualization of DNNs. Our main results are: (i) most of\nthe training epochs in standard DL are spent on {\\emph compression} of the\ninput to efficient representation and not on fitting the training labels. (ii)\nThe representation compression phase begins when the training errors becomes\nsmall and the Stochastic Gradient Decent (SGD) epochs change from a fast drift\nto smaller training error into a stochastic relaxation, or random diffusion,\nconstrained by the training error value. (iii) The converged layers lie on or\nvery close to the Information Bottleneck (IB) theoretical bound, and the maps\nfrom the input to any hidden layer and from this hidden layer to the output\nsatisfy the IB self-consistent equations. This generalization through noise\nmechanism is unique to Deep Neural Networks and absent in one layer networks.\n(iv) The training time is dramatically reduced when adding more hidden layers.\nThus the main advantage of the hidden layers is computational. This can be\nexplained by the reduced relaxation time, as this it scales super-linearly\n(exponentially for simple diffusion) with the information compression from the\nprevious layer.","url_abs":"http://arxiv.org/abs/1703.00810v3","url_pdf":"http://arxiv.org/pdf/1703.00810v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/JeffersonLab/trackingML","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/StephanLorenzen/ExactIBAnalysisInQNNs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/alomrani/IDNNs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/dizcza/EmbedderSDR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/etherandrius/information-networks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/gtegner/mine-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/ikoloska/M.I.A.","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/makezur/information_bottleneck_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/ravidziv/IDNNs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/sepehr-rasouli/ITRL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/sepehr-rasouli/PCBS-ITRL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/sichen-x/Fashion-MNIST-NN-Modeling-Plotting","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"opening-the-black-box-of-deep-neural-networks","repo_url":"https://github.com/taolicheng/understanding-dnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"information-plane","task_name":"Information Plane"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1703.00810","atlas_url":"https://app.syntology.ai/?focus=1703.00810","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1703.00810"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gtegner/mine-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/etherandrius/information-networks","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sepehr-rasouli/ITRL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sepehr-rasouli/PCBS-ITRL","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ikoloska/M.I.A.","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ravidziv/IDNNs","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/taolicheng/understanding-dnn","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/makezur/information_bottleneck_pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alomrani/IDNNs","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/StephanLorenzen/ExactIBAnalysisInQNNs","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dizcza/EmbedderSDR","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JeffersonLab/trackingML","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sichen-x/Fashion-MNIST-NN-Modeling-Plotting","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":11,"ran_fixture":2,"ran_honours":3,"unverified":1},"by_repo_kind":{"listed":{"samples":14,"ran":13,"repositories":6}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"0625e50998d14878","entry":"binarize","repo":"etherandrius/information-networks","repo_kind":"listed","path":"information/NaftaliTishby.py","file_url":"https://github.com/etherandrius/information-networks/blob/HEAD/information/NaftaliTishby.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0625e50998d14878"}},{"code_sha256_prefix":"2e757a88026d14d3","entry":"build_dist","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"2e757a88026d14d3"}},{"code_sha256_prefix":"c64726ea73437f06","entry":"calc_information_for_layer","repo":"ravidziv/IDNNs","repo_kind":"listed","path":"idnns/information/information_process.py","file_url":"https://github.com/ravidziv/IDNNs/blob/HEAD/idnns/information/information_process.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c64726ea73437f06"}},{"code_sha256_prefix":"c7ac117fe021d1f1","entry":"calc_velocity","repo":"alomrani/IDNNs","repo_kind":"listed","path":"idnns/plots/plot_figures.py","file_url":"https://github.com/alomrani/IDNNs/blob/HEAD/idnns/plots/plot_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c7ac117fe021d1f1"}},{"code_sha256_prefix":"a1073d8f05df7b5a","entry":"ema","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a1073d8f05df7b5a"}},{"code_sha256_prefix":"33a893f5d09f90f7","entry":"ema_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"33a893f5d09f90f7"}},{"code_sha256_prefix":"5fcc1acb6f40d03b","entry":"entropy","repo":"sepehr-rasouli/ITRL","repo_kind":"listed","path":"mnist_mi.py","file_url":"https://github.com/sepehr-rasouli/ITRL/blob/HEAD/mnist_mi.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5fcc1acb6f40d03b"}},{"code_sha256_prefix":"0a8afec82ca4566b","entry":"entropy_of_probabilities","repo":"etherandrius/information-networks","repo_kind":"listed","path":"information/NaftaliTishby.py","file_url":"https://github.com/etherandrius/information-networks/blob/HEAD/information/NaftaliTishby.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a8afec82ca4566b"}},{"code_sha256_prefix":"0e5211fe3cbe7d5c","entry":"extract_array","repo":"alomrani/IDNNs","repo_kind":"listed","path":"idnns/plots/plot_figures.py","file_url":"https://github.com/alomrani/IDNNs/blob/HEAD/idnns/plots/plot_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0e5211fe3cbe7d5c"}},{"code_sha256_prefix":"583d051bd6b1090d","entry":"filename","repo":"etherandrius/information-networks","repo_kind":"listed","path":"main_as_if_random.py","file_url":"https://github.com/etherandrius/information-networks/blob/HEAD/main_as_if_random.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"583d051bd6b1090d"}},{"code_sha256_prefix":"43a34df93b4b67a4","entry":"get_probabilities","repo":"etherandrius/information-networks","repo_kind":"listed","path":"information/NaftaliTishby.py","file_url":"https://github.com/etherandrius/information-networks/blob/HEAD/information/NaftaliTishby.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43a34df93b4b67a4"}},{"code_sha256_prefix":"31621f47d5230d8a","entry":"get_probabilities","repo":"etherandrius/information-networks","repo_kind":"listed","path":"main_as_if_random.py","file_url":"https://github.com/etherandrius/information-networks/blob/HEAD/main_as_if_random.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31621f47d5230d8a"}},{"code_sha256_prefix":"ad4ae5354a08a3bc","entry":"get_probabilities_map","repo":"etherandrius/information-networks","repo_kind":"listed","path":"main_as_if_random.py","file_url":"https://github.com/etherandrius/information-networks/blob/HEAD/main_as_if_random.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad4ae5354a08a3bc"}},{"code_sha256_prefix":"d6626b73baa1f114","entry":"load_figures","repo":"alomrani/IDNNs","repo_kind":"listed","path":"idnns/plots/plot_figures.py","file_url":"https://github.com/alomrani/IDNNs/blob/HEAD/idnns/plots/plot_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d6626b73baa1f114"}},{"code_sha256_prefix":"9a35694d84f5e3b6","entry":"load_tishby_toy_dataset","repo":"makezur/information_bottleneck_pytorch","repo_kind":"listed","path":"pytorch_network.py","file_url":"https://github.com/makezur/information_bottleneck_pytorch/blob/HEAD/pytorch_network.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a35694d84f5e3b6"}},{"code_sha256_prefix":"f2a85ea2b597b0a1","entry":"train_network","repo":"makezur/information_bottleneck_pytorch","repo_kind":"listed","path":"pytorch_network.py","file_url":"https://github.com/makezur/information_bottleneck_pytorch/blob/HEAD/pytorch_network.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"none","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2a85ea2b597b0a1"}},{"code_sha256_prefix":"d5ce0afe906f0fd1","entry":"compute_MI","repo":"StephanLorenzen/ExactIBAnalysisInQNNs","repo_kind":"listed","path":"IB/experiment.py","file_url":"https://github.com/StephanLorenzen/ExactIBAnalysisInQNNs/blob/HEAD/IB/experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d5ce0afe906f0fd1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}