{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-decoding-and-reconstruction-via-eeg","title":"Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion","arxiv_id":"2403.07721","date":"2024-03-12","proceeding":null,"authors":["Dongyang Li","Chen Wei","Shiying Li","Jiachen Zou","Haoyang Qin","Quanying Liu"],"abstract":"How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of visual decoding and reconstruction based on functional Magnetic Resonance Imaging (fMRI). However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for visual decoding based on electroencephalography (EEG). In this study, we present an end-to-end EEG-based visual reconstruction zero-shot framework, consisting of a tailored brain encoder, called the Adaptive Thinking Mapper (ATM), which projects neural signals from different sources into the shared subspace as the clip embedding, and a two-stage multi-pipe EEG-to-image generation strategy. In stage one, EEG is embedded to align the high-level clip embedding, and then the prior diffusion model refines EEG embedding into image priors. A blurry image also decoded from EEG for maintaining the low-level feature. In stage two, we input both the high-level clip embedding, the blurry image and caption from EEG latent to a pre-trained diffusion model. Furthermore, we analyzed the impacts of different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. The experimental results indicate that our EEG-based visual zero-shot framework achieves SOTA performance in classification, retrieval and reconstruction, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. Our code is available at https://github.com/ncclab-sustech/EEG_Image_decode.","url_abs":"https://arxiv.org/abs/2403.07721v7","url_pdf":"https://arxiv.org/pdf/2403.07721v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"visual-decoding-and-reconstruction-via-eeg","repo_url":"https://github.com/dongyangli-del/eeg_image_decode","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"visual-decoding-and-reconstruction-via-eeg","repo_url":"https://github.com/ncclab-sustech/eeg_image_decode","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"visual-decoding-and-reconstruction-via-eeg","repo_url":"https://github.com/nzwang/neural-mcrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"visual-decoding-and-reconstruction-via-eeg","repo_url":"https://github.com/woldier/siamseg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.07721","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.07721"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dongyangli-del/eeg_image_decode","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/woldier/siamseg","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nzwang/neural-mcrl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ncclab-sustech/eeg_image_decode","reach":null}],"summary":{"ran_draft_wrong":4,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"dde08c57d862ec73","entry":"evaluate_model","repo":"dongyangli-del/eeg_image_decode","repo_kind":"official","path":"Generation/ATMS_reconstruction.py","file_url":"https://github.com/dongyangli-del/eeg_image_decode/blob/HEAD/Generation/ATMS_reconstruction.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dde08c57d862ec73"}},{"code_sha256_prefix":"a8f9f460931bbeb6","entry":"evaluate_model","repo":"dongyangli-del/eeg_image_decode","repo_kind":"official","path":"Retrieval/ATMS_retrieval.py","file_url":"https://github.com/dongyangli-del/eeg_image_decode/blob/HEAD/Retrieval/ATMS_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8f9f460931bbeb6"}},{"code_sha256_prefix":"df2981298d15124f","entry":"extract_id_from_string","repo":"dongyangli-del/eeg_image_decode","repo_kind":"official","path":"Generation/ATMS_reconstruction.py","file_url":"https://github.com/dongyangli-del/eeg_image_decode/blob/HEAD/Generation/ATMS_reconstruction.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df2981298d15124f"}},{"code_sha256_prefix":"747fbfe5fbb06b48","entry":"train_model","repo":"dongyangli-del/eeg_image_decode","repo_kind":"official","path":"Generation/ATMS_reconstruction.py","file_url":"https://github.com/dongyangli-del/eeg_image_decode/blob/HEAD/Generation/ATMS_reconstruction.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"747fbfe5fbb06b48"}},{"code_sha256_prefix":"8fae4949de13669a","entry":"train_model","repo":"dongyangli-del/eeg_image_decode","repo_kind":"official","path":"Retrieval/ATMS_retrieval.py","file_url":"https://github.com/dongyangli-del/eeg_image_decode/blob/HEAD/Retrieval/ATMS_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fae4949de13669a"}},{"code_sha256_prefix":"1979eed793bbbbf9","entry":"evaluate_model","repo":"nzwang/neural-mcrl","repo_kind":"listed","path":"EEGToVisual/NeuralMCRL.py","file_url":"https://github.com/nzwang/neural-mcrl/blob/HEAD/EEGToVisual/NeuralMCRL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1979eed793bbbbf9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}