{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sleepfm-multi-modal-representation-learning","title":"SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals","arxiv_id":"2405.17766","date":"2024-05-28","proceeding":null,"authors":["Rahul Thapa","Bryan He","Magnus Ruud Kjaer","Hyatt Moore","Gauri Ganjoo","Emmanuel Mignot","James Zou"],"abstract":"Sleep is a complex physiological process evaluated through various modalities recording electrical brain, cardiac, and respiratory activities. We curate a large polysomnography dataset from over 14,000 participants comprising over 100,000 hours of multi-modal sleep recordings. Leveraging this extensive dataset, we developed SleepFM, the first multi-modal foundation model for sleep analysis. We show that a novel leave-one-out approach for contrastive learning significantly improves downstream task performance compared to representations from standard pairwise contrastive learning. A logistic regression model trained on SleepFM's learned embeddings outperforms an end-to-end trained convolutional neural network (CNN) on sleep stage classification (macro AUROC 0.88 vs 0.72 and macro AUPRC 0.72 vs 0.48) and sleep disordered breathing detection (AUROC 0.85 vs 0.69 and AUPRC 0.77 vs 0.61). Notably, the learned embeddings achieve 48% top-1 average accuracy in retrieving the corresponding recording clips of other modalities from 90,000 candidates. This work demonstrates the value of holistic multi-modal sleep modeling to fully capture the richness of sleep recordings. SleepFM is open source and available at https://github.com/rthapa84/sleepfm-codebase.","url_abs":"https://arxiv.org/abs/2405.17766v1","url_pdf":"https://arxiv.org/pdf/2405.17766v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sleepfm-multi-modal-representation-learning","repo_url":"https://github.com/rthapa84/sleepfm-codebase","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"logistic-regression","method_name":"Logistic Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.17766","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.17766"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rthapa84/sleepfm-codebase","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"be38301cc4c45a97","entry":"cache_pkl","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/model/dataset.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/model/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be38301cc4c45a97"}},{"code_sha256_prefix":"649ce0bf7b6ccf7e","entry":"count_parameters","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/2_pretrain.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/2_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"649ce0bf7b6ccf7e"}},{"code_sha256_prefix":"42245dbd78d632cc","entry":"extract_labels","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/0_extract_pretraining_data.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/0_extract_pretraining_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42245dbd78d632cc"}},{"code_sha256_prefix":"6135b0afef6f2ae2","entry":"getEDFFilenames","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/utils.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6135b0afef6f2ae2"}},{"code_sha256_prefix":"01dfb4f45e1655db","entry":"getEDFFiles","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/utils.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01dfb4f45e1655db"}},{"code_sha256_prefix":"198787f0addf9d90","entry":"load_data","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/utils.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"198787f0addf9d90"}},{"code_sha256_prefix":"e9351282d196962e","entry":"resample_signal","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/0_extract_pretraining_data.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/0_extract_pretraining_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9351282d196962e"}},{"code_sha256_prefix":"aad95e0e0f173703","entry":"cache_csv","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/model/dataset.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/model/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aad95e0e0f173703"}},{"code_sha256_prefix":"7e08abd687492a60","entry":"import_signal_names","repo":"rthapa84/sleepfm-codebase","repo_kind":"official","path":"sleepfm/0_extract_pretraining_data.py","file_url":"https://github.com/rthapa84/sleepfm-codebase/blob/HEAD/sleepfm/0_extract_pretraining_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e08abd687492a60"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}