{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrast-attend-and-diffuse-to-decode-high","title":"Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities","arxiv_id":"2305.17214","date":"2023-05-26","proceeding":"NeurIPS 2023 11","authors":["Jingyuan Sun","Mingxiao Li","Zijiao Chen","Yunhao Zhang","Shaonan Wang","Marie-Francine Moens"],"abstract":"Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) presents an intriguing intersection between cognitive neuroscience and machine learning, promising advancements in understanding human visual perception and building non-invasive brain-machine interfaces. However, the task is challenging due to the noisy nature of fMRI signals and the intricate pattern of brain visual representations. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework. The first phase pre-trains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder. The optimized fMRI feature learner then conditions a latent diffusion model to reconstruct image stimuli from brain activities. Experimental results demonstrate our model's superiority in generating high-resolution and semantically accurate images, substantially exceeding previous state-of-the-art methods by 39.34% in the 50-way-top-1 semantic classification accuracy. Our research invites further exploration of the decoding task's potential and contributes to the development of non-invasive brain-machine interfaces.","url_abs":"https://arxiv.org/abs/2305.17214v4","url_pdf":"https://arxiv.org/pdf/2305.17214v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrast-attend-and-diffuse-to-decode-high","repo_url":"https://github.com/soinx0629/vis_dec_neurips","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"brain-decoding","task_name":"Brain Decoding"},{"task_slug":"brain-visual-reconstruction","task_name":"Brain Visual Reconstruction"},{"task_slug":"brain-visual-reconstruction-from-fmri","task_name":"Brain Visual Reconstruction from fMRI"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/brain-visual-reconstruction-from-fmri-on-god","task":"Brain Visual Reconstruction from fMRI","dataset":"GOD","model":"DC-LDM","rank_in_archive_order":1,"of":2,"metrics":{"50-way-top1-classfication accuract":"17.999"},"uses_additional_data":false},{"leaderboard":"/sota/brain-visual-reconstruction-from-fmri-on-god","task":"Brain Visual Reconstruction from fMRI","dataset":"GOD","model":"CAD (this paper)","rank_in_archive_order":2,"of":2,"metrics":{"50-way-top1-classfication accuract":"25.080"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.17214","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}