Papers › UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without...

UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation

20 Feb 2024arXiv:2402.12730archive 2025-07-28

Shubhashis Roy Dipta, Sai Vallurupalli

The aim of SemEval-2024 Task 1, "Semantic Textual Relatedness for African and Asian Languages" is to develop models for identifying semantic textual relatedness (STR) between two sentences using multiple languages (14 African and Asian languages) and settings (supervised, unsupervised, and cross-lingual). Large language models (LLMs) have shown impressive performance on several natural language understanding tasks such as multilingual machine translation (MMT), semantic similarity (STS), and encoding sentence embeddings. Using a combination of LLMs that perform well on these tasks, we developed two STR models, TranSem and FineSem, for the supervised and cross-lingual settings. We explore the effectiveness of several training methods and the usefulness of machine translation. We find that direct fine-tuning on the task is comparable to using sentence embeddings and translating to English leads to better performance for some languages. In the supervised setting, our model performance is better than the official baseline for 3 languages with the remaining 4 performing on par. In the cross-lingual setting, our model performance is better than the baseline for 3 languages (leading to 1ˢᵗ place for Africaans and 2ⁿᵈ place for Indonesian), is on par for 2 languages and performs poorly on the remaining 7 languages. Our code is publicly available at https://github.com/dipta007/SemEval24-Task8.

Paper PDF Code

Code

dipta007/semeval24-task8 officialmentioned in paperpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Machine Translation Natural Language Understanding STS Semantic Similarity Semantic Textual Similarity Sentence Sentence Embedding Sentence Embeddings Sentence-Embedding Translation

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Adafactor Attention Attention Dropout BPE Dense Connections Dropout Gated Linear Unit Inverse Square Root Schedule Layer Normalization Linear Layer Multi-Head Attention Residual Connection SentencePiece Softmax T5

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections