{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/token-level-contrastive-learning-with","title":"Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition","arxiv_id":"2312.14667","date":"2023-12-22","proceeding":null,"authors":["Qianrui Zhou","Hua Xu","Hao Li","Hanlei Zhang","Xiaohan Zhang","Yifan Wang","Kai Gao"],"abstract":"Multimodal intent recognition aims to leverage diverse modalities such as expressions, body movements and tone of speech to comprehend user's intent, constituting a critical task for understanding human language and behavior in real-world multimodal scenarios. Nevertheless, the majority of existing methods ignore potential correlations among different modalities and own limitations in effectively learning semantic features from nonverbal modalities. In this paper, we introduce a token-level contrastive learning method with modality-aware prompting (TCL-MAP) to address the above challenges. To establish an optimal multimodal semantic environment for text modality, we develop a modality-aware prompting module (MAP), which effectively aligns and fuses features from text, video and audio modalities with similarity-based modality alignment and cross-modality attention mechanism. Based on the modality-aware prompt and ground truth labels, the proposed token-level contrastive learning framework (TCL) constructs augmented samples and employs NT-Xent loss on the label token. Specifically, TCL capitalizes on the optimal textual semantic insights derived from intent labels to guide the learning processes of other modalities in return. Extensive experiments show that our method achieves remarkable improvements compared to state-of-the-art methods. Additionally, ablation analyses demonstrate the superiority of the modality-aware prompt over the handcrafted prompt, which holds substantial significance for multimodal prompt learning. The codes are released at https://github.com/thuiar/TCL-MAP.","url_abs":"https://arxiv.org/abs/2312.14667v2","url_pdf":"https://arxiv.org/pdf/2312.14667v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"token-level-contrastive-learning-with","repo_url":"https://github.com/thuiar/TCL-MAP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"intent-recognition","task_name":"Intent Recognition"},{"task_slug":"multimodal-intent-recognition","task_name":"Multimodal Intent Recognition"},{"task_slug":"prompt-learning","task_name":"Prompt Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"nt-xent","method_name":"NT-Xent"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/multimodal-intent-recognition-on-mintrec","task":"Multimodal Intent Recognition","dataset":"MIntRec","model":"TCL-MAP","rank_in_archive_order":2,"of":6,"metrics":{"Accuracy (20 classes)":"73.62"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.14667","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.14667"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thuiar/TCL-MAP","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"ran_fixture":1,"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10865bbb99140edd","entry":"Linear","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"methods/TCL_MAP/SubNets/transformers_encoder/transformer.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/methods/TCL_MAP/SubNets/transformers_encoder/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10865bbb99140edd"}},{"code_sha256_prefix":"d13fbe5e7e90947f","entry":"buffered_future_mask","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"methods/TCL_MAP/SubNets/transformers_encoder/transformer.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/methods/TCL_MAP/SubNets/transformers_encoder/transformer.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d13fbe5e7e90947f"}},{"code_sha256_prefix":"8f266b9d616de9b9","entry":"fill_with_neg_inf","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"methods/TCL_MAP/SubNets/transformers_encoder/transformer.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/methods/TCL_MAP/SubNets/transformers_encoder/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f266b9d616de9b9"}},{"code_sha256_prefix":"21958d540bd87a8c","entry":"make_positions","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"methods/TCL_MAP/SubNets/transformers_encoder/position_embedding.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/methods/TCL_MAP/SubNets/transformers_encoder/position_embedding.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21958d540bd87a8c"}},{"code_sha256_prefix":"c25b59874b2d48aa","entry":"restore_model","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"utils/functions.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/utils/functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c25b59874b2d48aa"}},{"code_sha256_prefix":"1ef6337ae6763042","entry":"add_config_param","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"configs/base.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/configs/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ef6337ae6763042"}},{"code_sha256_prefix":"8c6bc8fd52264b46","entry":"load_npy","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"utils/functions.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/utils/functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c6bc8fd52264b46"}},{"code_sha256_prefix":"d152ab9b27609574","entry":"set_output_path","repo":"thuiar/TCL-MAP","repo_kind":"official","path":"utils/functions.py","file_url":"https://github.com/thuiar/TCL-MAP/blob/HEAD/utils/functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d152ab9b27609574"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}