{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/from-static-to-dynamic-adapting-landmark-1","title":"From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos","arxiv_id":"2312.05447","date":"2023-12-09","proceeding":null,"authors":["Yin Chen","Jia Li","Shiguang Shan","Meng Wang","Richang Hong"],"abstract":"Dynamic facial expression recognition (DFER) in the wild is still hindered by data limitations, e.g., insufficient quantity and diversity of pose, occlusion and illumination, as well as the inherent ambiguity of facial expressions. In contrast, static facial expression recognition (SFER) currently shows much higher performance and can benefit from more abundant high-quality training data. Moreover, the appearance features and dynamic dependencies of DFER remain largely unexplored. To tackle these challenges, we introduce a novel Static-to-Dynamic model (S2D) that leverages existing SFER knowledge and dynamic information implicitly encoded in extracted facial landmark-aware features, thereby significantly improving DFER performance. Firstly, we build and train an image model for SFER, which incorporates a standard Vision Transformer (ViT) and Multi-View Complementary Prompters (MCPs) only. Then, we obtain our video model (i.e., S2D), for DFER, by inserting Temporal-Modeling Adapters (TMAs) into the image model. MCPs enhance facial expression features with landmark-aware features inferred by an off-the-shelf facial landmark detector. And the TMAs capture and model the relationships of dynamic changes in facial expressions, effectively extending the pre-trained image model for videos. Notably, MCPs and TMAs only increase a fraction of trainable parameters (less than +10\\%) to the original image model. Moreover, we present a novel Emotion-Anchors (i.e., reference samples for each emotion category) based Self-Distillation Loss to reduce the detrimental influence of ambiguous emotion labels, further enhancing our S2D. Experiments conducted on popular SFER and DFER datasets show that we achieve the state of the art.","url_abs":"https://arxiv.org/abs/2312.05447v2","url_pdf":"https://arxiv.org/pdf/2312.05447v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"from-static-to-dynamic-adapting-landmark-1","repo_url":"https://github.com/FER-LMC/S2D","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"from-static-to-dynamic-adapting-landmark-1","repo_url":"https://github.com/msa-lmc/s2d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"dynamic-facial-expression-recognition","task_name":"Dynamic Facial Expression Recognition"},{"task_slug":"facial-expression-recognition-1","task_name":"Facial Expression Recognition"},{"task_slug":"facial-expression-recognition","task_name":"Facial Expression Recognition (FER)"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/facial-expression-recognition-on-affectnet","task":"Facial Expression Recognition (FER)","dataset":"AffectNet","model":"S2D","rank_in_archive_order":10,"of":50,"metrics":{"Accuracy (7 emotion)":"67.62","Accuracy (8 emotion)":"63.06"},"uses_additional_data":false},{"leaderboard":"/sota/facial-expression-recognition-on-raf-db","task":"Facial Expression Recognition (FER)","dataset":"RAF-DB","model":"S2D","rank_in_archive_order":5,"of":35,"metrics":{"Overall Accuracy":"92.57"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.05447","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.05447"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/msa-lmc/s2d","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FER-LMC/S2D","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"unverified":7},"by_repo_kind":{"official":{"samples":14,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bc39b8ed5fefa7eb","entry":"crop_clip","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/functional.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc39b8ed5fefa7eb"}},{"code_sha256_prefix":"2b53e457d5632b4e","entry":"get_resize_sizes","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/functional.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b53e457d5632b4e"}},{"code_sha256_prefix":"981936c73d3cdd12","entry":"resize_clip","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/functional.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"981936c73d3cdd12"}},{"code_sha256_prefix":"e866e1196986f728","entry":"shear_x","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e866e1196986f728"}},{"code_sha256_prefix":"7b2503c99c9447fb","entry":"shear_y","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b2503c99c9447fb"}},{"code_sha256_prefix":"6b9b585439223637","entry":"train_class_batch","repo":"FER-LMC/S2D","repo_kind":"official","path":"engine_for_finetuning.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/engine_for_finetuning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b9b585439223637"}},{"code_sha256_prefix":"323759f906fbb394","entry":"translate_x_rel","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"323759f906fbb394"}},{"code_sha256_prefix":"f00a047305ea9335","entry":"build_dataset","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/datasets.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f00a047305ea9335"}},{"code_sha256_prefix":"55b44eaa04001d35","entry":"build_pretraining_dataset","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/datasets.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55b44eaa04001d35"}},{"code_sha256_prefix":"e35ba287fd7fef0b","entry":"feature2token","repo":"FER-LMC/S2D","repo_kind":"official","path":"model_finetuning.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/model_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e35ba287fd7fef0b"}},{"code_sha256_prefix":"29416043c7035e7c","entry":"get_loss_scale_for_deepspeed","repo":"FER-LMC/S2D","repo_kind":"official","path":"engine_for_finetuning.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29416043c7035e7c"}},{"code_sha256_prefix":"7e49117a60f3b108","entry":"spatial_sampling","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/kinetics.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/kinetics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e49117a60f3b108"}},{"code_sha256_prefix":"14b3b13b2e7ee844","entry":"tensor_normalize","repo":"FER-LMC/S2D","repo_kind":"official","path":"datasets/kinetics.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/datasets/kinetics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14b3b13b2e7ee844"}},{"code_sha256_prefix":"dafcc70e40e4a867","entry":"token2feature","repo":"FER-LMC/S2D","repo_kind":"official","path":"model_finetuning.py","file_url":"https://github.com/FER-LMC/S2D/blob/HEAD/model_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dafcc70e40e4a867"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}