{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/frequency-aware-masked-autoencoders-for","title":"Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals","arxiv_id":"2309.05927","date":"2023-09-12","proceeding":null,"authors":["Ran Liu","Ellen L. Zippi","Hadi Pouransari","Chris Sandino","Jingping Nie","Hanlin Goh","Erdrin Azemi","Ali Moin"],"abstract":"Leveraging multimodal information from biosignals is vital for building a comprehensive representation of people's physical and mental states. However, multimodal biosignals often exhibit substantial distributional shifts between pretraining and inference datasets, stemming from changes in task specification or variations in modality compositions. To achieve effective pretraining in the presence of potential distributional shifts, we propose a frequency-aware masked autoencoder ($\\texttt{bio}$FAME) that learns to parameterize the representation of biosignals in the frequency space. $\\texttt{bio}$FAME incorporates a frequency-aware transformer, which leverages a fixed-size Fourier-based operator for global token mixing, independent of the length and sampling rate of inputs. To maintain the frequency components within each input channel, we further employ a frequency-maintain pretraining strategy that performs masked autoencoding in the latent space. The resulting architecture effectively utilizes multimodal information during pretraining, and can be seamlessly adapted to diverse tasks and modalities at test time, regardless of input size and order. We evaluated our approach on a diverse set of transfer experiments on unimodal time series, achieving an average of $\\uparrow$5.5% improvement in classification accuracy over the previous state-of-the-art. Furthermore, we demonstrated that our architecture is robust in modality mismatch scenarios, including unpredicted modality dropout or substitution, proving its practical utility in real-world applications. Code is available at https://github.com/apple/ml-famae .","url_abs":"https://arxiv.org/abs/2309.05927v2","url_pdf":"https://arxiv.org/pdf/2309.05927v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"frequency-aware-masked-autoencoders-for","repo_url":"https://github.com/apple/ml-famae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.05927","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.05927"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/apple/ml-famae","reach":null}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"42c8d64590336334","entry":"Attention","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"42c8d64590336334"}},{"code_sha256_prefix":"d24d34d7c3763ef1","entry":"FA_Transformer","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d24d34d7c3763ef1"}},{"code_sha256_prefix":"662d497a467bafd6","entry":"FeedForward","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"662d497a467bafd6"}},{"code_sha256_prefix":"275699f58a7e28ae","entry":"FreqFilter_multihead","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"275699f58a7e28ae"}},{"code_sha256_prefix":"12d3b78ad6507edb","entry":"PostNorm","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"12d3b78ad6507edb"}},{"code_sha256_prefix":"d1367547d97a6355","entry":"RevIN","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d1367547d97a6355"}},{"code_sha256_prefix":"830c324c0667edb8","entry":"Transformer","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"830c324c0667edb8"}},{"code_sha256_prefix":"b2ceb88a0b87afa6","entry":"bioFAME","repo":"apple/ml-famae","repo_kind":"official","path":"bioFAME/models/algorithms.py","file_url":"https://github.com/apple/ml-famae/blob/HEAD/bioFAME/models/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b2ceb88a0b87afa6"}},{"code_sha256_prefix":"b0bc80b1655a6802","entry":"get_algorithm_class","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"b0bc80b1655a6802"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}