{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-multi-dimensional-edge-feature-based","title":"Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition","arxiv_id":"2205.01782","date":"2022-05-02","proceeding":null,"authors":["Cheng Luo","Siyang Song","Weicheng Xie","Linlin Shen","Hatice Gunes"],"abstract":"The activations of Facial Action Units (AUs) mutually influence one another. While the relationship between a pair of AUs can be complex and unique, existing approaches fail to specifically and explicitly represent such cues for each pair of AUs in each facial display. This paper proposes an AU relationship modelling approach that deep learns a unique graph to explicitly describe the relationship between each pair of AUs of the target facial display. Our approach first encodes each AU's activation status and its association with other AUs into a node feature. Then, it learns a pair of multi-dimensional edge features to describe multiple task-specific relationship cues between each pair of AUs. During both node and edge feature learning, our approach also considers the influence of the unique facial display on AUs' relationship by taking the full face representation as an input. Experimental results on BP4D and DISFA datasets show that both node and edge feature learning modules provide large performance improvements for CNN and transformer-based backbones, with our best systems achieving the state-of-the-art AU recognition results. Our approach not only has a strong capability in modelling relationship cues for AU recognition but also can be easily incorporated into various backbones. Our PyTorch code is made available.","url_abs":"https://arxiv.org/abs/2205.01782v2","url_pdf":"https://arxiv.org/pdf/2205.01782v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-multi-dimensional-edge-feature-based","repo_url":"https://github.com/cvi-szu/me-graphau","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-multi-dimensional-edge-feature-based","repo_url":"https://github.com/tomas-gajarsky/facetorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"facial-action-unit-detection","task_name":"Facial Action Unit Detection"},{"task_slug":null,"task_name":"Relation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/facial-action-unit-detection-on-bp4d","task":"Facial Action Unit Detection","dataset":"BP4D","model":"Multi-dimensional Edge Feature-based AU Relation Graph (Swin-B)","rank_in_archive_order":4,"of":10,"metrics":{"Average AUC":"83.1","Average F1":"65.5"},"uses_additional_data":false},{"leaderboard":"/sota/facial-action-unit-detection-on-bp4d","task":"Facial Action Unit Detection","dataset":"BP4D","model":"Multi-dimensional Edge Feature-based AU Relation Graph (ResNet 50)","rank_in_archive_order":5,"of":10,"metrics":{"Average AUC":"82.6","Average F1":"64.7"},"uses_additional_data":false},{"leaderboard":"/sota/facial-action-unit-detection-on-bp4d","task":"Facial Action Unit Detection","dataset":"BP4D","model":"Swin-B","rank_in_archive_order":7,"of":10,"metrics":{"Average F1":"62.6"},"uses_additional_data":false},{"leaderboard":"/sota/facial-action-unit-detection-on-bp4d","task":"Facial Action Unit Detection","dataset":"BP4D","model":"ResNet 50","rank_in_archive_order":9,"of":10,"metrics":{"Average F1":"59.1"},"uses_additional_data":false},{"leaderboard":"/sota/facial-action-unit-detection-on-disfa","task":"Facial Action Unit Detection","dataset":"DISFA","model":"Multi-dimensional Edge Feature-based AU Relation Graph (ResNet 50)","rank_in_archive_order":5,"of":8,"metrics":{"Average AUC":"92.9","Average F1":"63.1"},"uses_additional_data":false},{"leaderboard":"/sota/facial-action-unit-detection-on-disfa","task":"Facial Action Unit Detection","dataset":"DISFA","model":"Multi-dimensional Edge Feature-based AU Relation Graph (Swin-B)","rank_in_archive_order":6,"of":8,"metrics":{"Average AUC":"92.1","Average F1":"62.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.01782","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.01782"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tomas-gajarsky/facetorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cvi-szu/me-graphau","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/CVI-SZU/ME-GraphAU","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2feb320e2c23be1a","entry":"AUHead","repo":"tomas-gajarsky/facetorch","repo_kind":"listed","path":"model_defs/au_model.py","file_url":"https://github.com/tomas-gajarsky/facetorch/blob/HEAD/model_defs/au_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2feb320e2c23be1a"}},{"code_sha256_prefix":"183ee9ac63fa9c42","entry":"CrossAttn","repo":"CVI-SZU/ME-GraphAU","repo_kind":"official","path":"model/graph_edge_model.py","file_url":"https://github.com/CVI-SZU/ME-GraphAU/blob/HEAD/model/graph_edge_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"183ee9ac63fa9c42"}},{"code_sha256_prefix":"bb7a676b77dee7d2","entry":"GEM","repo":"CVI-SZU/ME-GraphAU","repo_kind":"official","path":"model/graph_edge_model.py","file_url":"https://github.com/CVI-SZU/ME-GraphAU/blob/HEAD/model/graph_edge_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb7a676b77dee7d2"}},{"code_sha256_prefix":"871d4585235c0cf7","entry":"GNN","repo":"tomas-gajarsky/facetorch","repo_kind":"listed","path":"model_defs/au_model.py","file_url":"https://github.com/tomas-gajarsky/facetorch/blob/HEAD/model_defs/au_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"871d4585235c0cf7"}},{"code_sha256_prefix":"12681ff944eff063","entry":"LinearBlock","repo":"tomas-gajarsky/facetorch","repo_kind":"listed","path":"model_defs/au_model.py","file_url":"https://github.com/tomas-gajarsky/facetorch/blob/HEAD/model_defs/au_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12681ff944eff063"}},{"code_sha256_prefix":"ceba7e37e845cfc5","entry":"OpenGraphAU","repo":"tomas-gajarsky/facetorch","repo_kind":"listed","path":"model_defs/au_model.py","file_url":"https://github.com/tomas-gajarsky/facetorch/blob/HEAD/model_defs/au_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ceba7e37e845cfc5"}},{"code_sha256_prefix":"3fd8be2d40a50da3","entry":"_normalize_digraph","repo":"tomas-gajarsky/facetorch","repo_kind":"listed","path":"model_defs/au_model.py","file_url":"https://github.com/tomas-gajarsky/facetorch/blob/HEAD/model_defs/au_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3fd8be2d40a50da3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}