{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reconstructing-the-mind-s-eye-fmri-to-image","title":"Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors","arxiv_id":"2305.18274","date":"2023-05-29","proceeding":"NeurIPS 2023 11","authors":["Paul S. Scotti","Atmadeep Banerjee","Jimmie Goode","Stepan Shabalin","Alex Nguyen","Ethan Cohen","Aidan J. Dempster","Nathalie Verlinde","Elad Yundler","David Weisberg","Kenneth A. Norman","Tanishq Mathew Abraham"],"abstract":"We present MindEye, a novel fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity. Our model comprises two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). MindEye can map fMRI brain activity to any high dimensional multimodal latent space, like CLIP image space, enabling image reconstruction using generative models that accept embeddings from this latent space. We comprehensively compare our approach with other existing methods, using both qualitative side-by-side comparisons and quantitative evaluations, and show that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that MindEye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters. Furthermore, we show that MindEye can better preserve low-level image features in the reconstructions by using img2img, with outputs from a separate autoencoder. All code is available on GitHub.","url_abs":"https://arxiv.org/abs/2305.18274v2","url_pdf":"https://arxiv.org/pdf/2305.18274v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reconstructing-the-mind-s-eye-fmri-to-image","repo_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.18274","atlas_url":"https://app.syntology.ai/?focus=2305.18274","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.18274"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/medarc-ai/fmri-reconstruction-nsd","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9016165a0276448e","entry":"MLP","repo":"medarc-ai/fmri-reconstruction-nsd","repo_kind":"official","path":"src/convnext.py","file_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/HEAD/src/convnext.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9016165a0276448e"}},{"code_sha256_prefix":"10c69d73c8ea8690","entry":"Image_to_torch","repo":"medarc-ai/fmri-reconstruction-nsd","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10c69d73c8ea8690"}},{"code_sha256_prefix":"0a792899d6ffa09d","entry":"create_BOLD5000_dataset","repo":"medarc-ai/fmri-reconstruction-nsd","repo_kind":"official","path":"src/train_autoencoder_bold.py","file_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/HEAD/src/train_autoencoder_bold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a792899d6ffa09d"}},{"code_sha256_prefix":"e6d197791f0b0002","entry":"get_stimuli_list","repo":"medarc-ai/fmri-reconstruction-nsd","repo_kind":"official","path":"src/train_autoencoder_bold.py","file_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/HEAD/src/train_autoencoder_bold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6d197791f0b0002"}},{"code_sha256_prefix":"e7f5cd89fd5e1a5a","entry":"list_get_all_index","repo":"medarc-ai/fmri-reconstruction-nsd","repo_kind":"official","path":"src/train_autoencoder_bold.py","file_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/HEAD/src/train_autoencoder_bold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7f5cd89fd5e1a5a"}},{"code_sha256_prefix":"d36f277eefb3c8c9","entry":"np_to_Image","repo":"medarc-ai/fmri-reconstruction-nsd","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d36f277eefb3c8c9"}},{"code_sha256_prefix":"d36059a0fce5fa4e","entry":"torch_to_Image","repo":"medarc-ai/fmri-reconstruction-nsd","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d36059a0fce5fa4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}