{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bridging-the-gap-for-test-time-multimodal","title":"Bridging the Gap for Test-Time Multimodal Sentiment Analysis","arxiv_id":"2412.07121","date":"2024-12-10","proceeding":null,"authors":["Zirun Guo","Tao Jin","Wenlong Xu","Wang Lin","Yangyang Wu"],"abstract":"Multimodal sentiment analysis (MSA) is an emerging research topic that aims to understand and recognize human sentiment or emotions through multiple modalities. However, in real-world dynamic scenarios, the distribution of target data is always changing and different from the source data used to train the model, which leads to performance degradation. Common adaptation methods usually need source data, which could pose privacy issues or storage overheads. Therefore, test-time adaptation (TTA) methods are introduced to improve the performance of the model at inference time. Existing TTA methods are always based on probabilistic models and unimodal learning, and thus can not be applied to MSA which is often considered as a multimodal regression task. In this paper, we propose two strategies: Contrastive Adaptation and Stable Pseudo-label generation (CASP) for test-time adaptation for multimodal sentiment analysis. The two strategies deal with the distribution shifts for MSA by enforcing consistency and minimizing empirical risk, respectively. Extensive experiments show that CASP brings significant and consistent improvements to the performance of the model across various distribution shift settings and with different backbones, demonstrating its effectiveness and versatility. Our codes are available at https://github.com/zrguo/CASP.","url_abs":"https://arxiv.org/abs/2412.07121v1","url_pdf":"https://arxiv.org/pdf/2412.07121v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bridging-the-gap-for-test-time-multimodal","repo_url":"https://github.com/zrguo/casp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multimodal-sentiment-analysis","task_name":"Multimodal Sentiment Analysis"},{"task_slug":"pseudo-label","task_name":"Pseudo Label"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"test-time-adaptation","task_name":"Test-time Adaptation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.07121","atlas_url":"https://app.syntology.ai/?focus=2412.07121","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.07121"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zrguo/casp","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_violates":1,"ran_honours":2,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10865bbb99140edd","entry":"Linear","repo":"zrguo/casp","repo_kind":"official","path":"modules/transformer.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/modules/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10865bbb99140edd"}},{"code_sha256_prefix":"184a5eddedaf6a42","entry":"buffered_future_mask","repo":"zrguo/casp","repo_kind":"official","path":"modules/transformer.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/modules/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"184a5eddedaf6a42"}},{"code_sha256_prefix":"8f266b9d616de9b9","entry":"fill_with_neg_inf","repo":"zrguo/casp","repo_kind":"official","path":"modules/transformer.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/modules/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f266b9d616de9b9"}},{"code_sha256_prefix":"21958d540bd87a8c","entry":"make_positions","repo":"zrguo/casp","repo_kind":"official","path":"modules/position_embedding.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/modules/position_embedding.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21958d540bd87a8c"}},{"code_sha256_prefix":"6d4da8365bb0f5d1","entry":"weighted_accuracy","repo":"zrguo/casp","repo_kind":"official","path":"utils/util.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/utils/util.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d4da8365bb0f5d1"}},{"code_sha256_prefix":"531e1ad90f7396a7","entry":"eval_senti","repo":"zrguo/casp","repo_kind":"official","path":"utils/util.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"531e1ad90f7396a7"}},{"code_sha256_prefix":"890c0885552ee7e6","entry":"getdataloader","repo":"zrguo/casp","repo_kind":"official","path":"utils/dataloader.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/utils/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"890c0885552ee7e6"}},{"code_sha256_prefix":"d68cb5b22ea8ad14","entry":"initiate","repo":"zrguo/casp","repo_kind":"official","path":"utils/pseudo_train.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/utils/pseudo_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d68cb5b22ea8ad14"}},{"code_sha256_prefix":"65494cdd1508772e","entry":"multiclass_acc","repo":"zrguo/casp","repo_kind":"official","path":"utils/util.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65494cdd1508772e"}},{"code_sha256_prefix":"8476f524c8f63eb5","entry":"train_model","repo":"zrguo/casp","repo_kind":"official","path":"utils/base_model_train.py","file_url":"https://github.com/zrguo/casp/blob/HEAD/utils/base_model_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8476f524c8f63eb5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}