{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/factorizephys-matrix-factorization-for","title":"FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing","arxiv_id":"2411.01542","date":"2024-11-03","proceeding":null,"authors":["Jitesh Joshi","Sos S. Agaian","Youngjun Cho"],"abstract":"Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.","url_abs":"https://arxiv.org/abs/2411.01542v1","url_pdf":"https://arxiv.org/pdf/2411.01542v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"factorizephys-matrix-factorization-for","repo_url":"https://github.com/physiologicailab/factorizephys","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.01542","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.01542"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/PhysiologicAILab/FactorizePhys","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/physiologicailab/factorizephys","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":6,"unverified":12},"by_repo_kind":{"official":{"samples":18,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"6e953ed1bef334e0","entry":"ConvBNReLU","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6e953ed1bef334e0"}},{"code_sha256_prefix":"7e6deb3d1596452c","entry":"ConvBlock3D","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7e6deb3d1596452c"}},{"code_sha256_prefix":"f6370889d40ce6ba","entry":"NMF","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f6370889d40ce6ba"}},{"code_sha256_prefix":"62f5d049c169e543","entry":"VQ","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"62f5d049c169e543"}},{"code_sha256_prefix":"d9172b8b2ece4bed","entry":"_MatrixDecompositionBase","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d9172b8b2ece4bed"}},{"code_sha256_prefix":"c6f24c2c9722d5d0","entry":"rPPG_FeatureExtractor","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c6f24c2c9722d5d0"}},{"code_sha256_prefix":"9cfd668e2f1adc6e","entry":"BVP_Head","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9cfd668e2f1adc6e"}},{"code_sha256_prefix":"cf521a70b7a58e4b","entry":"FactorizePhys","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cf521a70b7a58e4b"}},{"code_sha256_prefix":"1914d6b1ae446c06","entry":"FeaturesFactorizationModule","repo":"physiologicailab/factorizephys","repo_kind":"official","path":"neural_methods/model/FactorizePhys/FactorizePhys.py","file_url":"https://github.com/physiologicailab/factorizephys/blob/HEAD/neural_methods/model/FactorizePhys/FactorizePhys.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1914d6b1ae446c06"}},{"code_sha256_prefix":"d12ec52c95714d87","entry":"calculate_metric_per_video","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"evaluation/post_process.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/evaluation/post_process.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d12ec52c95714d87"}},{"code_sha256_prefix":"36f109f3279e5640","entry":"detrend","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"unsupervised_methods/utils.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/unsupervised_methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"36f109f3279e5640"}},{"code_sha256_prefix":"97b47578be179ffe","entry":"get_config","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"config.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"97b47578be179ffe"}},{"code_sha256_prefix":"6cb9bbf8e6dc7799","entry":"kl_loss","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"neural_methods/loss/PhysFormerLossComputer.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/neural_methods/loss/PhysFormerLossComputer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6cb9bbf8e6dc7799"}},{"code_sha256_prefix":"e02c853ace895512","entry":"normal_sampling","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"neural_methods/loss/PhysFormerLossComputer.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/neural_methods/loss/PhysFormerLossComputer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e02c853ace895512"}},{"code_sha256_prefix":"6cc56f582b060599","entry":"power2db","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"evaluation/post_process.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/evaluation/post_process.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6cc56f582b060599"}},{"code_sha256_prefix":"c47d6a12eff65869","entry":"process_video","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"unsupervised_methods/utils.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/unsupervised_methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c47d6a12eff65869"}},{"code_sha256_prefix":"848b026a94f60cf6","entry":"read_hr_label","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"evaluation/metrics.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"848b026a94f60cf6"}},{"code_sha256_prefix":"db1d2850c08d04e4","entry":"read_label","repo":"PhysiologicAILab/FactorizePhys","repo_kind":"official","path":"evaluation/metrics.py","file_url":"https://github.com/PhysiologicAILab/FactorizePhys/blob/HEAD/evaluation/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"db1d2850c08d04e4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}