{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-24693","title":"Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens","arxiv_id":"2509.24693","date":"2025-09-29","proceeding":"NeurIPS","authors":["Zijian Dong","Ruilin Li","Joanna Su Xian Chong","Niousha Dehestani","Yinghui Teng","Yi Lin","Zhizhou Li","Yichi Zhang","Yapei Xie","Leon Qi Rong Ooi","B. T. Thomas Yeo","Juan Helen Zhou"],"abstract":"We present Brain Harmony (BrainHarmonix), the first multimodal brain foundation model that unifies structural morphology and functional dynamics into compact 1D token representations. The model was pretrained on two of the largest neuroimaging datasets to date, encompassing 64,594 T1-weighted structural MRI 3D volumes (~ 14 million images) and 70,933 functional MRI (fMRI) time series. BrainHarmonix is grounded in two foundational neuroscience principles: structure complements function - structural and functional modalities offer distinct yet synergistic insights into brain organization; function follows structure - brain functional dynamics are shaped by cortical morphology. The modular pretraining process involves single-modality training with geometric pre-alignment followed by modality fusion through shared brain hub tokens. Notably, our dynamics encoder uniquely handles fMRI time series with heterogeneous repetition times (TRs), addressing a major limitation in existing models. BrainHarmonix is also the first to deeply compress high-dimensional neuroimaging signals into unified, continuous 1D tokens, forming a compact latent space of the human brain. BrainHarmonix achieves strong generalization across diverse downstream tasks, including neurodevelopmental and neurodegenerative disorder classification and cognition prediction - consistently outperforming previous approaches. Our models - pretrained on 8 H100 GPUs - aim to catalyze a new era of AI-driven neuroscience powered by large-scale multimodal neuroimaging.","url_abs":"https://arxiv.org/abs/2509.24693","url_pdf":"https://arxiv.org/pdf/2509.24693","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.24693","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.24693"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/hzlab/Brain-Harmony","reach":null}],"summary":{"ran":2,"ran_fixture":1,"ran_honours":1,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"862cf06383f8bde7","entry":"DropPath","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"862cf06383f8bde7"}},{"code_sha256_prefix":"00ab6e2a5e0dd98d","entry":"MLP","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00ab6e2a5e0dd98d"}},{"code_sha256_prefix":"55120f2026b56aa2","entry":"drop_path","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"55120f2026b56aa2"}},{"code_sha256_prefix":"83c2cef1c62331fd","entry":"get_1d_sincos_pos_embed","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83c2cef1c62331fd"}},{"code_sha256_prefix":"d3cc43506db5cff8","entry":"Attention","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3cc43506db5cff8"}},{"code_sha256_prefix":"6fdfbb3c57d4f9fe","entry":"Block","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fdfbb3c57d4f9fe"}},{"code_sha256_prefix":"df56dd1e07f7f87e","entry":"FlashAttention2","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df56dd1e07f7f87e"}},{"code_sha256_prefix":"2119ddf45caf867f","entry":"OneTokRegViT","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2119ddf45caf867f"}},{"code_sha256_prefix":"5e9bf5b2eaf4c4dd","entry":"is_flash_attn_greater_or_equal_2_10","repo":"hzlab/Brain-Harmony","repo_kind":"found_in_text","path":"modules/harmonizer/stage1_pretrain/models.py","file_url":"https://github.com/hzlab/Brain-Harmony/blob/HEAD/modules/harmonizer/stage1_pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e9bf5b2eaf4c4dd"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"q-bio.NC","source":"arxiv_api"},"syntology_extracted_results":null}