{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoding-visual-neural-representations-by","title":"Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features","arxiv_id":"2210.06756","date":"2022-10-13","proceeding":null,"authors":["Changde Du","Kaicheng Fu","Jinpeng Li","Huiguang He"],"abstract":"Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that uses multimodal learning of brain-visual-linguistic features. We focus on modeling the relationships between brain, visual and linguistic features via multimodal deep generative models. Specifically, we leverage the mixture-of-product-of-experts formulation to infer a latent code that enables a coherent joint generation of all three modalities. To learn a more consistent joint representation and improve the data efficiency in the case of limited brain activity data, we exploit both intra- and inter-modality mutual information maximization regularization terms. In particular, our BraVL model can be trained under various semi-supervised scenarios to incorporate the visual and textual features obtained from the extra categories. Finally, we construct three trimodal matching datasets, and the extensive experiments lead to some interesting conclusions and cognitive insights: 1) decoding novel visual categories from human brain activity is practically possible with good accuracy; 2) decoding models using the combination of visual and linguistic features perform much better than those using either of them alone; 3) visual perception may be accompanied by linguistic influences to represent the semantics of visual stimuli. Code and data: https://github.com/ChangdeDu/BraVL.","url_abs":"https://arxiv.org/abs/2210.06756v2","url_pdf":"https://arxiv.org/pdf/2210.06756v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decoding-visual-neural-representations-by","repo_url":"https://github.com/changdedu/bravl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"decoding-visual-neural-representations-by","repo_url":"https://github.com/2023-MindSpore-4/Code4/tree/main/msBraVL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.06756","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.06756"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/2023-MindSpore-4/Code4/tree/main/msBraVL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/changdedu/bravl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d2c778735f0f162","entry":"get_ImageNet_val_dataset","repo":"changdedu/bravl","repo_kind":"official","path":"BraVL_fMRI/extract_fea_with_timm.py","file_url":"https://github.com/changdedu/bravl/blob/HEAD/BraVL_fMRI/extract_fea_with_timm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d2c778735f0f162"}},{"code_sha256_prefix":"ec82cb62e468fd41","entry":"get_default_ImageNet_val_loader_withpath","repo":"changdedu/bravl","repo_kind":"official","path":"BraVL_fMRI/extract_fea_with_timm.py","file_url":"https://github.com/changdedu/bravl/blob/HEAD/BraVL_fMRI/extract_fea_with_timm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec82cb62e468fd41"}},{"code_sha256_prefix":"57e9d8afedabaf34","entry":"get_default_val_trans","repo":"changdedu/bravl","repo_kind":"official","path":"BraVL_fMRI/extract_fea_with_timm.py","file_url":"https://github.com/changdedu/bravl/blob/HEAD/BraVL_fMRI/extract_fea_with_timm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57e9d8afedabaf34"}},{"code_sha256_prefix":"3d4bef859ed913bf","entry":"log_li","repo":"changdedu/bravl","repo_kind":"official","path":"BraVL_EEG/run_epochs_trimodal.py","file_url":"https://github.com/changdedu/bravl/blob/HEAD/BraVL_EEG/run_epochs_trimodal.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d4bef859ed913bf"}},{"code_sha256_prefix":"ceeb4f47398f42ea","entry":"log_mean_exp","repo":"changdedu/bravl","repo_kind":"official","path":"BraVL_EEG/run_epochs_trimodal.py","file_url":"https://github.com/changdedu/bravl/blob/HEAD/BraVL_EEG/run_epochs_trimodal.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ceeb4f47398f42ea"}},{"code_sha256_prefix":"1ee1fa414d87f996","entry":"m_cubo","repo":"changdedu/bravl","repo_kind":"official","path":"BraVL_EEG/run_epochs_trimodal.py","file_url":"https://github.com/changdedu/bravl/blob/HEAD/BraVL_EEG/run_epochs_trimodal.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ee1fa414d87f996"}},{"code_sha256_prefix":"9a1e19ddd27ed04b","entry":"stability_selection","repo":"changdedu/bravl","repo_kind":"official","path":"BraVL_fMRI/stability_selection.py","file_url":"https://github.com/changdedu/bravl/blob/HEAD/BraVL_fMRI/stability_selection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a1e19ddd27ed04b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}