{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/signattention-on-the-interpretability-of","title":"SignAttention: On the Interpretability of Transformer Models for Sign Language Translation","arxiv_id":"2410.14506","date":"2024-10-18","proceeding":null,"authors":["Pedro Alejandro Dal Bianco","Oscar Agustín Stanchi","Facundo Manuel Quiroga","Franco Ronchetti","Enzo Ferrante"],"abstract":"This paper presents the first comprehensive interpretability analysis of a Transformer-based Sign Language Translation (SLT) model, focusing on the translation from video-based Greek Sign Language to glosses and text. Leveraging the Greek Sign Language Dataset, we examine the attention mechanisms within the model to understand how it processes and aligns visual input with sequential glosses. Our analysis reveals that the model pays attention to clusters of frames rather than individual ones, with a diagonal alignment pattern emerging between poses and glosses, which becomes less distinct as the number of glosses increases. We also explore the relative contributions of cross-attention and self-attention at each decoding step, finding that the model initially relies on video frames but shifts its focus to previously predicted tokens as the translation progresses. This work contributes to a deeper understanding of SLT models, paving the way for the development of more transparent and reliable translation systems essential for real-world applications.","url_abs":"https://arxiv.org/abs/2410.14506v1","url_pdf":"https://arxiv.org/pdf/2410.14506v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"signattention-on-the-interpretability-of","repo_url":"https://github.com/pedroodb/sign_attention","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"sign-language-translation","task_name":"Sign Language Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.14506","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.14506"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pedroodb/sign_attention","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"d51b6cf0f07742fa","entry":"InterpDecoderLayer","repo":"pedroodb/sign_attention","repo_kind":"official","path":"src/interp/InterpTransformer.py","file_url":"https://github.com/pedroodb/sign_attention/blob/HEAD/src/interp/InterpTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d51b6cf0f07742fa"}},{"code_sha256_prefix":"cb47955a7b820fc7","entry":"InterpEncoderLayer","repo":"pedroodb/sign_attention","repo_kind":"official","path":"src/interp/InterpTransformer.py","file_url":"https://github.com/pedroodb/sign_attention/blob/HEAD/src/interp/InterpTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb47955a7b820fc7"}},{"code_sha256_prefix":"eeddd5b048859dff","entry":"InterpTransformer","repo":"pedroodb/sign_attention","repo_kind":"official","path":"src/interp/InterpTransformer.py","file_url":"https://github.com/pedroodb/sign_attention/blob/HEAD/src/interp/InterpTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eeddd5b048859dff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}