{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-robust-multimodal-open-set-test-time","title":"Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization","arxiv_id":"2501.13924","date":"2025-01-23","proceeding":null,"authors":["Hao Dong","Eleni Chatzi","Olga Fink"],"abstract":"Test-time adaptation (TTA) has demonstrated significant potential in addressing distribution shifts between training and testing data. Open-set test-time adaptation (OSTTA) aims to adapt a source pre-trained model online to an unlabeled target domain that contains unknown classes. This task becomes more challenging when multiple modalities are involved. Existing methods have primarily focused on unimodal OSTTA, often filtering out low-confidence samples without addressing the complexities of multimodal data. In this work, we present Adaptive Entropy-aware Optimization (AEO), a novel framework specifically designed to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA) for the first time. Our analysis shows that the entropy difference between known and unknown samples in the target domain strongly correlates with MM-OSTTA performance. To leverage this, we propose two key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). These components enhance the ability of model to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples. To thoroughly evaluate our proposed methods in the MM-OSTTA setting, we establish a new benchmark derived from existing datasets. This benchmark includes two downstream tasks and incorporates five modalities. Extensive experiments across various domain shift situations demonstrate the efficacy and versatility of the AEO framework. Additionally, we highlight the strong performance of AEO in long-term and continual MM-OSTTA settings, both of which are challenging and highly relevant to real-world applications. Our source code is available at https://github.com/donghao51/AEO.","url_abs":"https://arxiv.org/abs/2501.13924v1","url_pdf":"https://arxiv.org/pdf/2501.13924v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-robust-multimodal-open-set-test-time","repo_url":"https://github.com/donghao51/aeo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"test-time-adaptation","task_name":"Test-time Adaptation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2501.13924","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.13924"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/donghao51/AEO","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/donghao51/aeo","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"2a70efefd5fd3fdf","entry":"normalized_prediction_entropy","repo":"donghao51/aeo","repo_kind":"official","path":"EPIC-rgb-flow-audio/test_video_audio_EPIC_OSTTA_hac.py","file_url":"https://github.com/donghao51/aeo/blob/HEAD/EPIC-rgb-flow-audio/test_video_audio_EPIC_OSTTA_hac.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a70efefd5fd3fdf"}},{"code_sha256_prefix":"27c636c26c1c2f18","entry":"auc_and_fpr_recall","repo":"donghao51/aeo","repo_kind":"official","path":"EPIC-rgb-flow-audio/test_video_audio_EPIC_OSTTA_hac.py","file_url":"https://github.com/donghao51/aeo/blob/HEAD/EPIC-rgb-flow-audio/test_video_audio_EPIC_OSTTA_hac.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27c636c26c1c2f18"}},{"code_sha256_prefix":"4ad7927af90aa220","entry":"compute_all_metrics","repo":"donghao51/aeo","repo_kind":"official","path":"EPIC-rgb-flow-audio/test_video_audio_EPIC_OSTTA_hac.py","file_url":"https://github.com/donghao51/aeo/blob/HEAD/EPIC-rgb-flow-audio/test_video_audio_EPIC_OSTTA_hac.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ad7927af90aa220"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}