{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neuroclips-towards-high-fidelity-and-smooth","title":"NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction","arxiv_id":"2410.19452","date":"2024-10-25","proceeding":null,"authors":["Zixuan Gong","Guangyin Bao","Qi Zhang","Zhongwei Wan","Duoqian Miao","Shoujin Wang","Lei Zhu","Changwei Wang","Rongtao Xu","Liang Hu","Ke Liu","Yu Zhang"],"abstract":"Reconstruction of static visual stimuli from non-invasion brain activity fMRI achieves great success, owning to advanced deep learning models such as CLIP and Stable Diffusion. However, the research on fMRI-to-video reconstruction remains limited since decoding the spatiotemporal perception of continuous visual experiences is formidably challenging. We contend that the key to addressing these challenges lies in accurately decoding both high-level semantics and low-level perception flows, as perceived by the brain in response to video stimuli. To the end, we propose NeuroClips, an innovative framework to decode high-fidelity and smooth video from fMRI. NeuroClips utilizes a semantics reconstructor to reconstruct video keyframes, guiding semantic accuracy and consistency, and employs a perception reconstructor to capture low-level perceptual details, ensuring video smoothness. During inference, it adopts a pre-trained T2V diffusion model injected with both keyframes and low-level perception flows for video reconstruction. Evaluated on a publicly available fMRI-video dataset, NeuroClips achieves smooth high-fidelity video reconstruction of up to 6s at 8FPS, gaining significant improvements over state-of-the-art models in various metrics, e.g., a 128% improvement in SSIM and an 81% improvement in spatiotemporal metrics. Our project is available at https://github.com/gongzix/NeuroClips.","url_abs":"https://arxiv.org/abs/2410.19452v3","url_pdf":"https://arxiv.org/pdf/2410.19452v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neuroclips-towards-high-fidelity-and-smooth","repo_url":"https://github.com/gongzix/neuroclips","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"ssim","task_name":"SSIM"},{"task_slug":"video-reconstruction","task_name":"Video Reconstruction"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.19452","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.19452"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/gongzix/NeuroClips","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gongzix/neuroclips","reach":{"status":"ok"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"10c69d73c8ea8690","entry":"Image_to_torch","repo":"gongzix/NeuroClips","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/gongzix/NeuroClips/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10c69d73c8ea8690"}},{"code_sha256_prefix":"d255526f231c7538","entry":"get_up_block","repo":"gongzix/NeuroClips","repo_kind":"official","path":"src/video_decoder.py","file_url":"https://github.com/gongzix/NeuroClips/blob/HEAD/src/video_decoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d255526f231c7538"}},{"code_sha256_prefix":"d36f277eefb3c8c9","entry":"np_to_Image","repo":"gongzix/NeuroClips","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/gongzix/NeuroClips/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d36f277eefb3c8c9"}},{"code_sha256_prefix":"d36059a0fce5fa4e","entry":"torch_to_Image","repo":"gongzix/NeuroClips","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/gongzix/NeuroClips/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d36059a0fce5fa4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}