{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neuroformer-multimodal-and-multitask","title":"Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data","arxiv_id":"2311.00136","date":"2023-10-31","proceeding":null,"authors":["Antonis Antoniades","Yiyi Yu","Joseph Canzano","William Wang","Spencer LaVere Smith"],"abstract":"State-of-the-art systems neuroscience experiments yield large-scale multimodal data, and these data sets require new tools for analysis. Inspired by the success of large pretrained models in vision and language domains, we reframe the analysis of large-scale, cellular-resolution neuronal spiking data into an autoregressive spatiotemporal generation problem. Neuroformer is a multimodal, multitask generative pretrained transformer (GPT) model that is specifically designed to handle the intricacies of data in systems neuroscience. It scales linearly with feature size, can process an arbitrary number of modalities, and is adaptable to downstream tasks, such as predicting behavior. We first trained Neuroformer on simulated datasets, and found that it both accurately predicted simulated neuronal circuit activity, and also intrinsically inferred the underlying neural circuit connectivity, including direction. When pretrained to decode neural responses, the model predicted the behavior of a mouse with only few-shot fine-tuning, suggesting that the model begins learning how to do so directly from the neural representations themselves, without any explicit supervision. We used an ablation study to show that joint training on neuronal responses and behavior boosted performance, highlighting the model's ability to associate behavioral and neural representations in an unsupervised manner. These findings show that Neuroformer can analyze neural datasets and their emergent properties, informing the development of models and hypotheses associated with the brain.","url_abs":"https://arxiv.org/abs/2311.00136v4","url_pdf":"https://arxiv.org/pdf/2311.00136v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neuroformer-multimodal-and-multitask","repo_url":"https://github.com/a-antoniades/neuroformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"neuroformer-multimodal-and-multitask","repo_url":"https://github.com/woanderer/Neuroformer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.00136","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.00136"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/woanderer/Neuroformer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/a-antoniades/neuroformer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":15,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":19,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d1186178a7836808","entry":"calc_corr_psth","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/analysis.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1186178a7836808"}},{"code_sha256_prefix":"4c1a31f765ef032e","entry":"combo3_V1AL_callback","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/datasets.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c1a31f765ef032e"}},{"code_sha256_prefix":"d496ef53516a57ca","entry":"contrastive_loss","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/modules.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d496ef53516a57ca"}},{"code_sha256_prefix":"ff4f6fdd29b64764","entry":"convolve_atts_3D","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/attention_vis.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/attention_vis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff4f6fdd29b64764"}},{"code_sha256_prefix":"c3503f3549cd9acf","entry":"get_attr","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/model_neuroformer.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/model_neuroformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3503f3549cd9acf"}},{"code_sha256_prefix":"33805df68cb6892e","entry":"get_emb","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/model_neuroformer.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/model_neuroformer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33805df68cb6892e"}},{"code_sha256_prefix":"f2941cf251031ecc","entry":"get_rates","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/analysis.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2941cf251031ecc"}},{"code_sha256_prefix":"2398af86975c9957","entry":"get_rates","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/analysis.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2398af86975c9957"}},{"code_sha256_prefix":"30d823e13e046f0b","entry":"model_ensemble","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/simulation.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d823e13e046f0b"}},{"code_sha256_prefix":"861a7367a2ed59cd","entry":"rollout_attentions","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/attention_vis.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/attention_vis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"861a7367a2ed59cd"}},{"code_sha256_prefix":"2e9f1a5673215030","entry":"separate_eos_tokens","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/modules.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e9f1a5673215030"}},{"code_sha256_prefix":"015233a9d8a82fc1","entry":"split_data_by_interval","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/datasets.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"015233a9d8a82fc1"}},{"code_sha256_prefix":"a9af76e30c2a8e46","entry":"trial_df","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/data_utils.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9af76e30c2a8e46"}},{"code_sha256_prefix":"2afb4903a5098278","entry":"trial_df_combo3","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/data_utils.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2afb4903a5098278"}},{"code_sha256_prefix":"8ad3185a7b0c4d25","entry":"trial_df_real","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/data_utils.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ad3185a7b0c4d25"}},{"code_sha256_prefix":"dc18b01aaf085b2c","entry":"visnav_callback","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/datasets.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc18b01aaf085b2c"}},{"code_sha256_prefix":"62b7dc6a0fcb3055","entry":"nearest","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/gather_atts.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/gather_atts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62b7dc6a0fcb3055"}},{"code_sha256_prefix":"99fdd80b9ca7f208","entry":"reshape_features","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/attention_vis.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/attention_vis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99fdd80b9ca7f208"}},{"code_sha256_prefix":"120a5300accb00db","entry":"top_k_top_p_filtering","repo":"a-antoniades/neuroformer","repo_kind":"official","path":"neuroformer/simulation.py","file_url":"https://github.com/a-antoniades/neuroformer/blob/HEAD/neuroformer/simulation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"120a5300accb00db"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}