{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mindgpt-interpreting-what-you-see-with-non","title":"MindGPT: Interpreting What You See with Non-invasive Brain Recordings","arxiv_id":"2309.15729","date":"2023-09-27","proceeding":null,"authors":["Jiaxuan Chen","Yu Qi","Yueming Wang","Gang Pan"],"abstract":"Decoding of seen visual contents with non-invasive brain recordings has important scientific and practical values. Efforts have been made to recover the seen images from brain signals. However, most existing approaches cannot faithfully reflect the visual contents due to insufficient image quality or semantic mismatches. Compared with reconstructing pixel-level visual images, speaking is a more efficient and effective way to explain visual information. Here we introduce a non-invasive neural decoder, termed as MindGPT, which interprets perceived visual stimuli into natural languages from fMRI signals. Specifically, our model builds upon a visually guided neural encoder with a cross-attention mechanism, which permits us to guide latent neural representations towards a desired language semantic direction in an end-to-end manner by the collaborative use of the large language model GPT. By doing so, we found that the neural representations of the MindGPT are explainable, which can be used to evaluate the contributions of visual properties to language semantics. Our experiments show that the generated word sequences truthfully represented the visual information (with essential details) conveyed in the seen stimuli. The results also suggested that with respect to language decoding tasks, the higher visual cortex (HVC) is more semantically informative than the lower visual cortex (LVC), and using only the HVC can recover most of the semantic information. The code of the MindGPT model will be publicly available at https://github.com/JxuanC/MindGPT.","url_abs":"https://arxiv.org/abs/2309.15729v1","url_pdf":"https://arxiv.org/pdf/2309.15729v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mindgpt-interpreting-what-you-see-with-non","repo_url":"https://github.com/jxuanc/mindgpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2309.15729","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.15729"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jxuanc/mindgpt","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"8f50794fd6ff2ac4","entry":"postprocess_preds","repo":"jxuanc/mindgpt","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jxuanc/mindgpt/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f50794fd6ff2ac4"}},{"code_sha256_prefix":"0750ce1a94dc82df","entry":"prep_strings","repo":"jxuanc/mindgpt","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jxuanc/mindgpt/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0750ce1a94dc82df"}},{"code_sha256_prefix":"e8d638fc7fcbd760","entry":"shift_tokens_right","repo":"jxuanc/mindgpt","repo_kind":"official","path":"modules/brain2text.py","file_url":"https://github.com/jxuanc/mindgpt/blob/HEAD/modules/brain2text.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8d638fc7fcbd760"}},{"code_sha256_prefix":"12035a2f77d8016c","entry":"get_1d_sincos_pos_embed_from_grid","repo":"jxuanc/mindgpt","repo_kind":"official","path":"modules/pos_embed.py","file_url":"https://github.com/jxuanc/mindgpt/blob/HEAD/modules/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12035a2f77d8016c"}},{"code_sha256_prefix":"3185afc3e87293ed","entry":"get_2d_sincos_pos_embed","repo":"jxuanc/mindgpt","repo_kind":"official","path":"modules/pos_embed.py","file_url":"https://github.com/jxuanc/mindgpt/blob/HEAD/modules/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3185afc3e87293ed"}},{"code_sha256_prefix":"f10004e059714d42","entry":"get_2d_sincos_pos_embed_from_grid","repo":"jxuanc/mindgpt","repo_kind":"official","path":"modules/pos_embed.py","file_url":"https://github.com/jxuanc/mindgpt/blob/HEAD/modules/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f10004e059714d42"}},{"code_sha256_prefix":"b969e70a00d92cb4","entry":"load_model","repo":"jxuanc/mindgpt","repo_kind":"official","path":"brain2text_infer.py","file_url":"https://github.com/jxuanc/mindgpt/blob/HEAD/brain2text_infer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b969e70a00d92cb4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}