{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/semrel2024-a-collection-of-semantic-textual","title":"SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages","arxiv_id":"2402.08638","date":"2024-02-13","proceeding":null,"authors":["Nedjma Ousidhoum","Shamsuddeen Hassan Muhammad","Mohamed Abdalla","Idris Abdulmumin","Ibrahim Said Ahmad","Sanchit Ahuja","Alham Fikri Aji","Vladimir Araujo","Abinew Ali Ayele","Pavan Baswani","Meriem Beloucif","Chris Biemann","Sofia Bourhim","Christine de Kock","Genet Shanko Dekebo","Oumaima Hourrane","Gopichand Kanumolu","Lokesh Madasu","Samuel Rutunda","Manish Shrivastava","Thamar Solorio","Nirmal Surange","Hailegnaw Getaneh Tilaye","Krishnapriya Vishnubhotla","Genta Winata","Seid Muhie Yimam","Saif M. Mohammad"],"abstract":"Exploring and quantifying semantic relatedness is central to representing language and holds significant implications across various NLP tasks. While earlier NLP research primarily focused on semantic similarity, often within the English language context, we instead investigate the broader phenomenon of semantic relatedness. In this paper, we present \\textit{SemRel}, a new semantic relatedness dataset collection annotated by native speakers across 13 languages: \\textit{Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Spanish,} and \\textit{Telugu}. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by a relatively limited availability of NLP resources. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. The scores are obtained using a comparative annotation framework. We describe the data collection and annotation processes, challenges when building the datasets, baseline experiments, and their impact and utility in NLP.","url_abs":"https://arxiv.org/abs/2402.08638v5","url_pdf":"https://arxiv.org/pdf/2402.08638v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"semrel2024-a-collection-of-semantic-textual","repo_url":"https://github.com/exploration-lab/iitk-semeval-2024-task-1","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"semrel2024-a-collection-of-semantic-textual","repo_url":"https://github.com/semantic-textual-relatedness/semantic_relatedness_semeval2024","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.08638","atlas_url":"https://app.syntology.ai/?focus=2402.08638","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.08638"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/semantic-textual-relatedness/semantic_relatedness_semeval2024","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/exploration-lab/iitk-semeval-2024-task-1","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"34bc01fab22f0f48","entry":"evaluate_embeddings","repo":"semantic-textual-relatedness/semantic_relatedness_semeval2024","repo_kind":"listed","path":"semrel_baselines/src/evaluate_bert.py","file_url":"https://github.com/semantic-textual-relatedness/semantic_relatedness_semeval2024/blob/HEAD/semrel_baselines/src/evaluate_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34bc01fab22f0f48"}},{"code_sha256_prefix":"4c3b1e4b6c6fa90e","entry":"mean_pooling","repo":"semantic-textual-relatedness/semantic_relatedness_semeval2024","repo_kind":"listed","path":"semrel_baselines/src/evaluate_bert.py","file_url":"https://github.com/semantic-textual-relatedness/semantic_relatedness_semeval2024/blob/HEAD/semrel_baselines/src/evaluate_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c3b1e4b6c6fa90e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}