{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-utilising-a-range-of-neural","title":"Towards Utilising a Range of Neural Activations for Comprehending Representational Associations","arxiv_id":"2411.10019","date":"2024-11-15","proceeding":null,"authors":["Laura O'Mahony","Nikola S. Nikolov","David JP O'Sullivan"],"abstract":"Recent efforts to understand intermediate representations in deep neural networks have commonly attempted to label individual neurons and combinations of neurons that make up linear directions in the latent space by examining extremal neuron activations and the highest direction projections. In this paper, we show that this approach, although yielding a good approximation for many purposes, fails to capture valuable information about the behaviour of a representation. Neural network activations are generally dense, and so a more complex, but realistic scenario is that linear directions encode information at various levels of stimulation. We hypothesise that non-extremal level activations contain complex information worth investigating, such as statistical associations, and thus may be used to locate confounding human interpretable concepts. We explore the value of studying a range of neuron activations by taking the case of mid-level output neuron activations and demonstrate on a synthetic dataset how they can inform us about aspects of representations in the penultimate layer not evident through analysing maximal activations alone. We use our findings to develop a method to curate data from mid-range logit samples for retraining to mitigate spurious correlations, or confounding concepts in the penultimate layer, on real benchmark datasets. The success of our method exemplifies the utility of inspecting non-maximal activations to extract complex relationships learned by models.","url_abs":"https://arxiv.org/abs/2411.10019v1","url_pdf":"https://arxiv.org/pdf/2411.10019v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-utilising-a-range-of-neural","repo_url":"https://github.com/lauraaisling/MID-level-activations","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.10019","atlas_url":"https://app.syntology.ai/?focus=2411.10019","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.10019"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lauraaisling/MID-level-activations","reach":{"status":"ok"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"d0919ca0ea528a46","entry":"add_annotations","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/plot_util.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/plot_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0919ca0ea528a46"}},{"code_sha256_prefix":"01e8f7f7f94c0bb5","entry":"contrastive_loss","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/models.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01e8f7f7f94c0bb5"}},{"code_sha256_prefix":"38ab5d79861412de","entry":"get_biased_indices","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/data.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38ab5d79861412de"}},{"code_sha256_prefix":"6d1bc711d175c8b1","entry":"get_model_device","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/models.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d1bc711d175c8b1"}},{"code_sha256_prefix":"7d78019a3b5b588a","entry":"load_encoder","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/load.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/load.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d78019a3b5b588a"}},{"code_sha256_prefix":"af45d65347dd84c0","entry":"load_vae_decoder","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/load.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/load.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af45d65347dd84c0"}},{"code_sha256_prefix":"e4d9d078495bcf8d","entry":"plot_dsprite_image_doubles","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/plot_util.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/plot_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4d9d078495bcf8d"}},{"code_sha256_prefix":"4761ec100607bfb7","entry":"plot_dsprites_images","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/plot_util.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/plot_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4761ec100607bfb7"}},{"code_sha256_prefix":"2534fadaaa51e3f0","entry":"subsample_sampler","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/data.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2534fadaaa51e3f0"}},{"code_sha256_prefix":"4a30b3c76ee2cb2c","entry":"train_classifier","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/models.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a30b3c76ee2cb2c"}},{"code_sha256_prefix":"5e4d6609128560c9","entry":"train_test_split_idx","repo":"lauraaisling/MID-level-activations","repo_kind":"official","path":"src/dsprites/data.py","file_url":"https://github.com/lauraaisling/MID-level-activations/blob/HEAD/src/dsprites/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e4d6609128560c9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}