{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/partial-information-decomposition-reveals-the","title":"A Measure of the Complexity of Neural Representations based on Partial Information Decomposition","arxiv_id":"2209.10438","date":"2022-09-21","proceeding":null,"authors":["David A. Ehrlich","Andreas C. Schneider","Viola Priesemann","Michael Wibral","Abdullah Makkeh"],"abstract":"In neural networks, task-relevant information is represented jointly by groups of neurons. However, the specific way in which this mutual information about the classification label is distributed among the individual neurons is not well understood: While parts of it may only be obtainable from specific single neurons, other parts are carried redundantly or synergistically by multiple neurons. We show how Partial Information Decomposition (PID), a recent extension of information theory, can disentangle these different contributions. From this, we introduce the measure of \"Representational Complexity\", which quantifies the difficulty of accessing information spread across multiple neurons. We show how this complexity is directly computable for smaller layers. For larger layers, we propose subsampling and coarse-graining procedures and prove corresponding bounds on the latter. Empirically, for quantized deep neural networks solving the MNIST and CIFAR10 tasks, we observe that representational complexity decreases both through successive hidden layers and over training, and compare the results to related measures. Overall, we propose representational complexity as a principled and interpretable summary statistic for analyzing the structure and evolution of neural representations and complex systems in general.","url_abs":"https://arxiv.org/abs/2209.10438v2","url_pdf":"https://arxiv.org/pdf/2209.10438v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"partial-information-decomposition-reveals-the","repo_url":"https://github.com/priesemann-group/nninfo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.10438","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.10438"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/priesemann-group/nninfo","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"26ed40d9f57ad645","entry":"get_logger","repo":"priesemann-group/nninfo","repo_kind":"official","path":"nninfo/logger.py","file_url":"https://github.com/priesemann-group/nninfo/blob/HEAD/nninfo/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"26ed40d9f57ad645"}},{"code_sha256_prefix":"9212a8ced7784cdc","entry":"kl_loss","repo":"priesemann-group/nninfo","repo_kind":"official","path":"nninfo/trainer.py","file_url":"https://github.com/priesemann-group/nninfo/blob/HEAD/nninfo/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9212a8ced7784cdc"}},{"code_sha256_prefix":"19664c08d01a6748","entry":"quantizer_factory","repo":"priesemann-group/nninfo","repo_kind":"official","path":"nninfo/model/quantization.py","file_url":"https://github.com/priesemann-group/nninfo/blob/HEAD/nninfo/model/quantization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"19664c08d01a6748"}},{"code_sha256_prefix":"7216ae12750987b7","entry":"quantizer_list_factory","repo":"priesemann-group/nninfo","repo_kind":"official","path":"nninfo/model/quantization.py","file_url":"https://github.com/priesemann-group/nninfo/blob/HEAD/nninfo/model/quantization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7216ae12750987b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}