{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-lingual-similarity-of-multilingual","title":"Cross-lingual Similarity of Multilingual Representations Revisited","arxiv_id":"2212.01924","date":"2022-12-04","proceeding":null,"authors":["Maksym Del","Mark Fishel"],"abstract":"Related works used indexes like CKA and variants of CCA to measure the similarity of cross-lingual representations in multilingual language models. In this paper, we argue that assumptions of CKA/CCA align poorly with one of the motivating goals of cross-lingual learning analysis, i.e., explaining zero-shot cross-lingual transfer. We highlight what valuable aspects of cross-lingual similarity these indexes fail to capture and provide a motivating case study \\textit{demonstrating the problem empirically}. Then, we introduce \\textit{Average Neuron-Wise Correlation (ANC)} as a straightforward alternative that is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context. Finally, we use ANC to construct evidence that the previously introduced ``first align, then predict'' pattern takes place not only in masked language models (MLMs) but also in multilingual models with \\textit{causal language modeling} objectives (CLMs). Moreover, we show that the pattern extends to the \\textit{scaled versions} of the MLMs and CLMs (up to 85x original mBERT).\\footnote{Our code is publicly available at \\url{https://github.com/TartuNLP/xsim}}","url_abs":"https://arxiv.org/abs/2212.01924v1","url_pdf":"https://arxiv.org/pdf/2212.01924v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-lingual-similarity-of-multilingual","repo_url":"https://github.com/TartuNLP/xsim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"Causal Language Modeling"},{"task_slug":"cross-lingual-transfer","task_name":"Cross-Lingual Transfer"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"zero-shot-cross-lingual-transfer","task_name":"Zero-Shot Cross-Lingual Transfer"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.01924","atlas_url":"https://app.syntology.ai/?focus=2212.01924","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}