{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/brain-mapping-with-dense-features-grounding","title":"Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers","arxiv_id":"2410.05266","date":"2024-10-07","proceeding":null,"authors":["Andrew F. Luo","Jacob Yeung","Rushikesh Zawar","Shaurya Dewan","Margaret M. Henderson","Leila Wehbe","Michael J. Tarr"],"abstract":"We introduce BrainSAIL, a method for linking neural selectivity with spatially distributed semantic visual concepts in natural scenes. BrainSAIL leverages recent advances in large-scale artificial neural networks, using them to provide insights into the functional topology of the brain. To overcome the challenge presented by the co-occurrence of multiple categories in natural images, BrainSAIL exploits semantically consistent, dense spatial features from pre-trained vision models, building upon their demonstrated ability to robustly predict neural activity. This method derives clean, spatially dense embeddings without requiring any additional training, and employs a novel denoising process that leverages the semantic consistency of images under random augmentations. By unifying the space of whole-image embeddings and dense visual features and then applying voxel-wise encoding models to these features, we enable the identification of specific subregions of each image which drive selectivity patterns in different areas of the higher visual cortex. This provides a powerful tool for dissecting the neural mechanisms that underlie semantic visual processing for natural images. We validate BrainSAIL on cortical regions with known category selectivity, demonstrating its ability to accurately localize and disentangle selectivity to diverse visual concepts. Next, we demonstrate BrainSAIL's ability to characterize high-level visual selectivity to scene properties and low-level visual features such as depth, luminance, and saturation, providing insights into the encoding of complex visual information. Finally, we use BrainSAIL to directly compare the feature selectivity of different brain encoding models across different regions of interest in visual cortex. Our innovative method paves the way for significant advances in mapping and decomposing high-level visual representations in the human brain.","url_abs":"https://arxiv.org/abs/2410.05266v2","url_pdf":"https://arxiv.org/pdf/2410.05266v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"brain-mapping-with-dense-features-grounding","repo_url":"https://github.com/aluo-x/brainsail","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.05266","atlas_url":"https://app.syntology.ai/?focus=2410.05266","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05266"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aluo-x/brainsail","reach":null}],"summary":{"ran_fixture":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5da27cf28601a430","entry":"farthest_point_sample","repo":"aluo-x/brainsail","repo_kind":"official","path":"DINO_v2_reg_dense_distill.py","file_url":"https://github.com/aluo-x/brainsail/blob/HEAD/DINO_v2_reg_dense_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5da27cf28601a430"}},{"code_sha256_prefix":"c370c4275f834dd5","entry":"normalize_image_deterministic","repo":"aluo-x/brainsail","repo_kind":"official","path":"DINO_v2_reg_dense_distill.py","file_url":"https://github.com/aluo-x/brainsail/blob/HEAD/DINO_v2_reg_dense_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c370c4275f834dd5"}},{"code_sha256_prefix":"8d0fb02172659671","entry":"normalize_image_deterministic","repo":"aluo-x/brainsail","repo_kind":"official","path":"NACLIP_dense_distill.py","file_url":"https://github.com/aluo-x/brainsail/blob/HEAD/NACLIP_dense_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d0fb02172659671"}},{"code_sha256_prefix":"88907be98abcbc3a","entry":"shuffle_shift","repo":"aluo-x/brainsail","repo_kind":"official","path":"DINO_v2_reg_dense_distill.py","file_url":"https://github.com/aluo-x/brainsail/blob/HEAD/DINO_v2_reg_dense_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88907be98abcbc3a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}