{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/emoclip-a-vision-language-method-for-zero","title":"EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition","arxiv_id":"2310.16640","date":"2023-10-25","proceeding":null,"authors":["Niki Maria Foteinopoulou","Ioannis Patras"],"abstract":"Facial Expression Recognition (FER) is a crucial task in affective computing, but its conventional focus on the seven basic emotions limits its applicability to the complex and expanding emotional spectrum. To address the issue of new and unseen emotions present in dynamic in-the-wild FER, we propose a novel vision-language model that utilises sample-level text descriptions (i.e. captions of the context, expressions or emotional cues) as natural language supervision, aiming to enhance the learning of rich latent representations, for zero-shot classification. To test this, we evaluate using zero-shot classification of the model trained on sample-level descriptions on four popular dynamic FER datasets. Our findings show that this approach yields significant improvements when compared to baseline methods. Specifically, for zero-shot video FER, we outperform CLIP by over 10\\% in terms of Weighted Average Recall and 5\\% in terms of Unweighted Average Recall on several datasets. Furthermore, we evaluate the representations obtained from the network trained using sample-level descriptions on the downstream task of mental health symptom estimation, achieving performance comparable or superior to state-of-the-art methods and strong agreement with human experts. Namely, we achieve a Pearson's Correlation Coefficient of up to 0.85 on schizophrenia symptom severity estimation, which is comparable to human experts' agreement. The code is publicly available at: https://github.com/NickyFot/EmoCLIP.","url_abs":"https://arxiv.org/abs/2310.16640v2","url_pdf":"https://arxiv.org/pdf/2310.16640v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"emoclip-a-vision-language-method-for-zero","repo_url":"https://github.com/nickyfot/emoclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"facial-expression-recognition-1","task_name":"Facial Expression Recognition"},{"task_slug":"facial-expression-recognition","task_name":"Facial Expression Recognition (FER)"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"zero-shot-facial-expression-recognition","task_name":"Zero-Shot Facial Expression Recognition"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-facial-expression-recognition-on","task":"Zero-Shot Facial Expression Recognition","dataset":"MAFW","model":"EmoCLIP","rank_in_archive_order":1,"of":1,"metrics":{"UAR":"25.86","WAR":"33.49"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.16640","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.16640"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nickyfot/emoclip","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1bf0d337eeb02b7c","entry":"CCC","repo":"nickyfot/emoclip","repo_kind":"official","path":"utils.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bf0d337eeb02b7c"}},{"code_sha256_prefix":"69709fdf9daef9aa","entry":"PCC","repo":"nickyfot/emoclip","repo_kind":"official","path":"utils.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69709fdf9daef9aa"}},{"code_sha256_prefix":"6872ff3312de15d1","entry":"get_config","repo":"nickyfot/emoclip","repo_kind":"official","path":"config.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6872ff3312de15d1"}},{"code_sha256_prefix":"a029d2b4dcbbafcd","entry":"load_frames","repo":"nickyfot/emoclip","repo_kind":"official","path":"DataLoaders/utils.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/DataLoaders/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a029d2b4dcbbafcd"}},{"code_sha256_prefix":"65dccfb689803260","entry":"feat_scatter","repo":"nickyfot/emoclip","repo_kind":"official","path":"utils.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65dccfb689803260"}},{"code_sha256_prefix":"3e9c6de2f1b19fb6","entry":"get_afew_loaders","repo":"nickyfot/emoclip","repo_kind":"official","path":"DataLoaders/get_loader.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/DataLoaders/get_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9c6de2f1b19fb6"}},{"code_sha256_prefix":"2209cf8a59150766","entry":"get_dfew_loaders","repo":"nickyfot/emoclip","repo_kind":"official","path":"DataLoaders/get_loader.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/DataLoaders/get_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2209cf8a59150766"}},{"code_sha256_prefix":"368b5461230215d6","entry":"get_loaders","repo":"nickyfot/emoclip","repo_kind":"official","path":"DataLoaders/get_loader.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/DataLoaders/get_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"368b5461230215d6"}},{"code_sha256_prefix":"409d90a67f7048de","entry":"load_annotation","repo":"nickyfot/emoclip","repo_kind":"official","path":"DataLoaders/utils.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/DataLoaders/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"409d90a67f7048de"}},{"code_sha256_prefix":"b7d567db51876aea","entry":"load_video","repo":"nickyfot/emoclip","repo_kind":"official","path":"DataLoaders/utils.py","file_url":"https://github.com/nickyfot/emoclip/blob/HEAD/DataLoaders/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7d567db51876aea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}