{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/correlation-coefficients-and-semantic-textual","title":"Correlation Coefficients and Semantic Textual Similarity","arxiv_id":"1905.07790","date":"2019-05-19","proceeding":"NAACL 2019 6","authors":["Vitalii Zhelezniak","Aleksandar Savkov","April Shen","Nils Y. Hammerla"],"abstract":"A large body of research into semantic textual similarity has focused on constructing state-of-the-art embeddings using sophisticated modelling, careful choice of learning signals and many clever tricks. By contrast, little attention has been devoted to similarity measures between these embeddings, with cosine similarity being used unquestionably in the majority of cases. In this work, we illustrate that for all common word vectors, cosine similarity is essentially equivalent to the Pearson correlation coefficient, which provides some justification for its use. We thoroughly characterise cases where Pearson correlation (and thus cosine similarity) is unfit as similarity measure. Importantly, we show that Pearson correlation is appropriate for some word vectors but not others. When it is not appropriate, we illustrate how common non-parametric rank correlation coefficients can be used instead to significantly improve performance. We support our analysis with a series of evaluations on word-level and sentence-level semantic textual similarity benchmarks. On the latter, we show that even the simplest averaged word vectors compared by rank correlation easily rival the strongest deep representations compared by cosine similarity.","url_abs":"https://arxiv.org/abs/1905.07790v1","url_pdf":"https://arxiv.org/pdf/1905.07790v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"correlation-coefficients-and-semantic-textual","repo_url":"https://github.com/Babylonpartners/corrsim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1905.07790","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.07790"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Babylonpartners/corrsim","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran_draft_wrong":1,"ran":1,"unverified":14},"by_repo_kind":{"official":{"samples":17,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a2a8999f1b9ffa71","entry":"cosine","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"senteval/utils.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/senteval/utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2a8999f1b9ffa71"}},{"code_sha256_prefix":"af7388308de7ccb4","entry":"create_dictionary","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/evaluation/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af7388308de7ccb4"}},{"code_sha256_prefix":"292a7349c4da48f7","entry":"create_dictionary","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"senteval/utils.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/senteval/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"292a7349c4da48f7"}},{"code_sha256_prefix":"99387298d487188b","entry":"apsynp","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"evaluation/wordsim_eval.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/evaluation/wordsim_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99387298d487188b"}},{"code_sha256_prefix":"b861377d2eb517b2","entry":"avg_cosine","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/baseline.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/baseline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b861377d2eb517b2"}},{"code_sha256_prefix":"e51eada32278286d","entry":"batcher","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"evaluation/corrsim_eval.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/evaluation/corrsim_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e51eada32278286d"}},{"code_sha256_prefix":"fa07e3df6f41686e","entry":"centering_matrix","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/cka.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/cka.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa07e3df6f41686e"}},{"code_sha256_prefix":"f2117d4a4682efc4","entry":"cosine","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"evaluation/wordsim_eval.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/evaluation/wordsim_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2117d4a4682efc4"}},{"code_sha256_prefix":"aac4af37868d65c4","entry":"cosine","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/baseline.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/baseline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aac4af37868d65c4"}},{"code_sha256_prefix":"8977dd7933a6415a","entry":"gaussian_kernel","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/cka.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/cka.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8977dd7933a6415a"}},{"code_sha256_prefix":"1bbd85a30996b26a","entry":"get_word_vec_path_by_name","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bbd85a30996b26a"}},{"code_sha256_prefix":"11ea4bc76f49da9a","entry":"get_wordvec","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11ea4bc76f49da9a"}},{"code_sha256_prefix":"9d895a661d2ba65a","entry":"kendall","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/correlation.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/correlation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d895a661d2ba65a"}},{"code_sha256_prefix":"f25e776a0755a0b9","entry":"linear_kernel","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/cka.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/cka.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f25e776a0755a0b9"}},{"code_sha256_prefix":"c37fc00aafe8b8aa","entry":"pearson","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/correlation.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/correlation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c37fc00aafe8b8aa"}},{"code_sha256_prefix":"c87a1937e68467a7","entry":"spearman","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"similarity/correlation.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/similarity/correlation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c87a1937e68467a7"}},{"code_sha256_prefix":"c7303175aea94e30","entry":"statistic","repo":"Babylonpartners/corrsim","repo_kind":"official","path":"evaluation/wordsim_eval.py","file_url":"https://github.com/Babylonpartners/corrsim/blob/HEAD/evaluation/wordsim_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7303175aea94e30"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}