{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoupled-multimodal-distilling-for-emotion","title":"Decoupled Multimodal Distilling for Emotion Recognition","arxiv_id":"2303.13802","date":"2023-03-24","proceeding":"CVPR 2023 1","authors":["Yong Li","Yuanzhi Wang","Zhen Cui"],"abstract":"Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decoupled into two parts, i.e., modality-irrelevant/-exclusive spaces, in a self-regression manner. DMD utilizes a graph distillation unit (GD-Unit) for each decoupled part so that each GD can be performed in a more specialized and effective manner. A GD-Unit consists of a dynamic graph where each vertice represents a modality and each edge indicates a dynamic knowledge distillation. Such GD paradigm provides a flexible knowledge transfer manner where the distillation weights can be automatically learned, thus enabling diverse crossmodal knowledge transfer patterns. Experimental results show DMD consistently obtains superior performance than state-of-the-art MER methods. Visualization results show the graph edges in DMD exhibit meaningful distributional patterns w.r.t. the modality-irrelevant/-exclusive feature spaces. Codes are released at \\url{https://github.com/mdswyz/DMD}.","url_abs":"https://arxiv.org/abs/2303.13802v1","url_pdf":"https://arxiv.org/pdf/2303.13802v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decoupled-multimodal-distilling-for-emotion","repo_url":"https://github.com/mdswyz/dmd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"multimodal-emotion-recognition","task_name":"Multimodal Emotion Recognition"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.13802","atlas_url":"https://app.syntology.ai/?focus=2303.13802","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.13802"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mdswyz/dmd","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mdswyz/DMD","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"2f3bc129645ecd7a","entry":"LayerNorm","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"2f3bc129645ecd7a"}},{"code_sha256_prefix":"6cde542a176e7ff1","entry":"SinusoidalPositionalEmbedding","repo":"mdswyz/DMD","repo_kind":"official","path":"trains/singleTask/model/dmd.py","file_url":"https://github.com/mdswyz/DMD/blob/HEAD/trains/singleTask/model/dmd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cde542a176e7ff1"}},{"code_sha256_prefix":"d13fbe5e7e90947f","entry":"buffered_future_mask","repo":"mdswyz/DMD","repo_kind":"official","path":"trains/singleTask/model/dmd.py","file_url":"https://github.com/mdswyz/DMD/blob/HEAD/trains/singleTask/model/dmd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d13fbe5e7e90947f"}},{"code_sha256_prefix":"c4736460ceb9ee69","entry":"DMD","repo":"mdswyz/DMD","repo_kind":"official","path":"trains/singleTask/model/dmd.py","file_url":"https://github.com/mdswyz/DMD/blob/HEAD/trains/singleTask/model/dmd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4736460ceb9ee69"}},{"code_sha256_prefix":"89990d89c23c5bfa","entry":"MultiheadAttention","repo":"mdswyz/DMD","repo_kind":"official","path":"trains/singleTask/model/dmd.py","file_url":"https://github.com/mdswyz/DMD/blob/HEAD/trains/singleTask/model/dmd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89990d89c23c5bfa"}},{"code_sha256_prefix":"8236465be0bb8229","entry":"TransformerEncoder","repo":"mdswyz/DMD","repo_kind":"official","path":"trains/singleTask/model/dmd.py","file_url":"https://github.com/mdswyz/DMD/blob/HEAD/trains/singleTask/model/dmd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8236465be0bb8229"}},{"code_sha256_prefix":"7b9d2681c13f15dc","entry":"TransformerEncoderLayer","repo":"mdswyz/DMD","repo_kind":"official","path":"trains/singleTask/model/dmd.py","file_url":"https://github.com/mdswyz/DMD/blob/HEAD/trains/singleTask/model/dmd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b9d2681c13f15dc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}