{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assessing-neural-network-representations-1","title":"Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy","arxiv_id":"2312.04823","date":"2023-12-04","proceeding":null,"authors":["Danqi Liao","Chen Liu","Benjamin W. Christensen","Alexander Tong","Guillaume Huguet","Guy Wolf","Maximilian Nickel","Ian Adelstein","Smita Krishnaswamy"],"abstract":"Entropy and mutual information in neural networks provide rich information on the learning process, but they have proven difficult to compute reliably in high dimensions. Indeed, in noisy and high-dimensional data, traditional estimates in ambient dimensions approach a fixed entropy and are prohibitively hard to compute. To address these issues, we leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. Specifically, we define diffusion spectral entropy (DSE) in neural representations of a dataset as well as diffusion spectral mutual information (DSMI) between different variables representing data. First, we show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data that outperform classic Shannon entropy, nonparametric estimation, and mutual information neural estimation (MINE). We then study the evolution of representations in classification networks with supervised learning, self-supervision, or overfitting. We observe that (1) DSE of neural representations increases during training; (2) DSMI with the class label increases during generalizable learning but stays stagnant during overfitting; (3) DSMI with the input signal shows differing trends: on MNIST it increases, while on CIFAR-10 and STL-10 it decreases. Finally, we show that DSE can be used to guide better network initialization and that DSMI can be used to predict downstream classification accuracy across 962 models on ImageNet. The official implementation is available at https://github.com/ChenLiu-1996/DiffusionSpectralEntropy.","url_abs":"https://arxiv.org/abs/2312.04823v1","url_pdf":"https://arxiv.org/pdf/2312.04823v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assessing-neural-network-representations-1","repo_url":"https://github.com/ChenLiu-1996/DiffusionSpectralEntropy","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.04823","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.04823"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ChenLiu-1996/DiffusionSpectralEntropy","reach":null}],"summary":{"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b5ba736044dc1d7c","entry":"compute_diffusion_matrix","repo":"ChenLiu-1996/DiffusionSpectralEntropy","repo_kind":"official","path":"dse_dsmi/diffusion.py","file_url":"https://github.com/ChenLiu-1996/DiffusionSpectralEntropy/blob/HEAD/dse_dsmi/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b5ba736044dc1d7c"}},{"code_sha256_prefix":"0ba31634be175ec1","entry":"adjacency_spectral_mutual_information","repo":"ChenLiu-1996/DiffusionSpectralEntropy","repo_kind":"official","path":"dse_dsmi/dsmi.py","file_url":"https://github.com/ChenLiu-1996/DiffusionSpectralEntropy/blob/HEAD/dse_dsmi/dsmi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0ba31634be175ec1"}},{"code_sha256_prefix":"bc60adc6f3746088","entry":"diffusion_spectral_mutual_information","repo":"ChenLiu-1996/DiffusionSpectralEntropy","repo_kind":"official","path":"dse_dsmi/dsmi.py","file_url":"https://github.com/ChenLiu-1996/DiffusionSpectralEntropy/blob/HEAD/dse_dsmi/dsmi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bc60adc6f3746088"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}