{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/latent-masking-for-multimodal-self-supervised","title":"CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking","arxiv_id":"2307.16847","date":"2023-07-31","proceeding":null,"authors":["Shohreh Deldari","Dimitris Spathis","Mohammad Malekzadeh","Fahim Kawsar","Flora Salim","Akhil Mathur"],"abstract":"Limited availability of labeled data for machine learning on multimodal time-series extensively hampers progress in the field. Self-supervised learning (SSL) is a promising approach to learning data representations without relying on labels. However, existing SSL methods require expensive computations of negative pairs and are typically designed for single modalities, which limits their versatility. We introduce CroSSL (Cross-modal SSL), which puts forward two novel concepts: masking intermediate embeddings produced by modality-specific encoders, and their aggregation into a global embedding through a cross-modal aggregator that can be fed to down-stream classifiers. CroSSL allows for handling missing modalities and end-to-end cross-modal learning without requiring prior data preprocessing for handling missing inputs or negative-pair sampling for contrastive learning. We evaluate our method on a wide range of data, including motion sensors such as accelerometers or gyroscopes and biosignals (heart rate, electroencephalograms, electromyograms, electrooculograms, and electrodermal) to investigate the impact of masking ratios and masking strategies for various data types and the robustness of the learned representations to missing data. Overall, CroSSL outperforms previous SSL and supervised benchmarks using minimal labeled data, and also sheds light on how latent masking can improve cross-modal learning. Our code is open-sourced at https://github.com/dr-bell/CroSSL.","url_abs":"https://arxiv.org/abs/2307.16847v3","url_pdf":"https://arxiv.org/pdf/2307.16847v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"latent-masking-for-multimodal-self-supervised","repo_url":"https://github.com/dr-bell/crossl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"BSD-3-Clause-Clear"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"novel-concepts","task_name":"Novel Concepts"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"time-series-1","task_name":"Time Series"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.16847","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.16847"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dr-bell/crossl","reach":{"status":"ok","spdx":"BSD-3-Clause-Clear"}}],"summary":{"ran":5,"unverified":7},"by_repo_kind":{"official":{"samples":12,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"03bb2b314c3002e2","entry":"get_classifier_model","repo":"dr-bell/crossl","repo_kind":"official","path":"src/evaluation/MaskedClassifier.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/evaluation/MaskedClassifier.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"03bb2b314c3002e2"}},{"code_sha256_prefix":"5349fcff15f1999a","entry":"get_ds_info","repo":"dr-bell/crossl","repo_kind":"official","path":"src/ds_utils/data_helper.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/ds_utils/data_helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"5349fcff15f1999a"}},{"code_sha256_prefix":"3314c948377a5ce8","entry":"get_ts_modality_encoder","repo":"dr-bell/crossl","repo_kind":"official","path":"src/baselines/Cocoa.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/baselines/Cocoa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"3314c948377a5ce8"}},{"code_sha256_prefix":"d37f506991051599","entry":"load_har_ds_sup","repo":"dr-bell/crossl","repo_kind":"official","path":"src/baselines/sup_data_helper.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/baselines/sup_data_helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"d37f506991051599"}},{"code_sha256_prefix":"cc82c21932161eae","entry":"setup_system","repo":"dr-bell/crossl","repo_kind":"official","path":"src/baselines/DeepConvLSTM.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/baselines/DeepConvLSTM.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"cc82c21932161eae"}},{"code_sha256_prefix":"c5e7300952da78e7","entry":"get_audio_encoder","repo":"dr-bell/crossl","repo_kind":"official","path":"src/baselines/Cocoa.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/baselines/Cocoa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"c5e7300952da78e7"}},{"code_sha256_prefix":"52ddac84b5418461","entry":"get_classifier","repo":"dr-bell/crossl","repo_kind":"official","path":"src/evaluation/classifier.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/evaluation/classifier.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"52ddac84b5418461"}},{"code_sha256_prefix":"7551ec40f23ce2d4","entry":"get_video_encoder","repo":"dr-bell/crossl","repo_kind":"official","path":"src/baselines/Cocoa.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/baselines/Cocoa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"7551ec40f23ce2d4"}},{"code_sha256_prefix":"78b93501c7e686dc","entry":"load_sleep4_ds","repo":"dr-bell/crossl","repo_kind":"official","path":"src/ds_utils/data_helper.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/ds_utils/data_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"78b93501c7e686dc"}},{"code_sha256_prefix":"677a3bdc3b6b9e38","entry":"load_sleep4_ds_sup","repo":"dr-bell/crossl","repo_kind":"official","path":"src/baselines/sup_data_helper.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/baselines/sup_data_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"677a3bdc3b6b9e38"}},{"code_sha256_prefix":"9129d81d4de302ee","entry":"one_hot_encoder","repo":"dr-bell/crossl","repo_kind":"official","path":"src/baselines/sup_data_helper.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/baselines/sup_data_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"9129d81d4de302ee"}},{"code_sha256_prefix":"fafece7b03bd514d","entry":"setup_optimizer","repo":"dr-bell/crossl","repo_kind":"official","path":"src/trainer.py","file_url":"https://github.com/dr-bell/crossl/blob/HEAD/src/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause-Clear","inline_ok":false,"mcp_get_code":{"code_sha256":"fafece7b03bd514d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}