Language Model Evaluation

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 31 papers with code (69 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation
  28 Dec 2023 2 repositories listed
  In this work, we introduce a novel evaluation paradigm for Large Language Models (LLMs) that compels them to transition from a traditional question-answering role, akin to a student, to a solution-scoring role, akin to…
• SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
  25 Aug 2023 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability.
• BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
  30 Jun 2022 2 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Training and evaluating language models increasingly requires the construction of meta-datasets --diverse collections of curated data with clear provenance.
• FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation
  30 May 2025 1 repository listed
  FABLE provides the first diagnostic benchmark to systematically evaluate data-flow reasoning and offers insights for developing models with stronger procedural understanding.
• Role-Playing Evaluation for Large Language Models
  19 May 2025 1 repository listed
  Large Language Models (LLMs) demonstrate a notable capacity for adopting personas and engaging in role-playing.
• M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysis
  17 Feb 2025 1 repository listed
  Aspect-based sentiment analysis (ABSA) is a crucial task in information extraction and sentiment analysis, aiming to identify aspects with associated sentiment elements in text.
• Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain
  10 Jan 2025 1 repository listed
  Generative AI holds significant potential for ecological and environmental applications such as monitoring, data analysis, education, and policy support.
• Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
  6 Jan 2025 1 repository listed
  The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation.
• Template Matters: Understanding the Role of Instruction Templates in Multimodal Language Model Evaluation and Training
  11 Dec 2024 1 repository listed
  Models tuned on our augmented dataset achieve the best overall performance when compared with the same scale MLMs tuned on at most 75 times the scale of our augmented dataset, highlighting the importance of instruction…
• C²LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation
  6 Dec 2024 1 repository listed
  Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data.
• DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA
  6 Dec 2024 1 repository listed Syntology ran 1 of 16 samples · 15 unverified · 16 pointer-only (licence)
  Recent advances in self-supervised models for natural language, vision, and protein sequences have inspired the development of large genomic DNA language models (DNALMs).
• Large Language Model Evaluation via Matrix Nuclear-Norm
  14 Oct 2024 1 repository listed
  While traditional metrics like Matrix Entropy offer valuable insights, they are computationally intensive for large-scale models due to their O(n³) time complexity with Singular Value Decomposition (SVD).
• Enterprise Benchmarks for Large Language Model Evaluation
  11 Oct 2024 1 repository listed
  The advancement of large language models (LLMs) has led to a greater challenge of having a rigorous and systematic evaluation of complex tasks performed, especially in enterprise applications.
• Mitigating the Bias of Large Language Model Evaluation
  25 Sep 2024 1 repository listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality.
• Salmon: A Suite for Acoustic Language Model Evaluation
  11 Sep 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Speech language models have recently demonstrated great potential as universal speech processing systems.
• Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation
  20 Jun 2024 1 repository listed
  On MMLU, using Inference-time Decontamination can lead to a decrease in the results of Phi3 and Mistral by 6.
• Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging
  20 May 2024 1 repository listed
  The rapid advancement of large language models has given rise to a plethora of applications across a myriad of real-world tasks, mainly centered on aligning with human intent.
• Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models
  17 Apr 2024 1 repository listed Syntology ran 3 of 4 samples · 1 unverified
  This paper studies the relationship between the surface form of a mathematical problem and its solvability by large language models.
• Evalverse: Unified and Accessible Library for Large Language Model Evaluation
  1 Apr 2024 1 repository listed
  This paper introduces Evalverse, a novel library that streamlines the evaluation of Large Language Models (LLMs) by unifying disparate evaluation tools into a single, user-friendly framework.
• Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform
  13 Mar 2024 1 repository listed
  Through this gateway, users have the opportunity to submit their questions, testing the models on a personalized and potentially broader range of capabilities.
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
  20 Feb 2024 1 repository listed Syntology ran 3 of 6 samples · 3 unverified · 6 pointer-only (licence)
  The focus of language model evaluation has transitioned towards reasoning and knowledge-intensive tasks, driven by advancements in pretraining large models.
• LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction
  19 Dec 2023 1 repository listed Syntology ran 2 of 6 samples · 4 unverified · 6 pointer-only (licence)
  LatestEval avoids data contamination by only using texts published within a recent time window, ensuring no overlap with the training corpora of pre-trained language models.
• Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
  15 Dec 2023 1 repository listed
  The success of large language models has shifted the evaluation paradigms in natural language processing (NLP).
• Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
  14 Nov 2023 1 repository listed
  In this work we systematically review the recent advancements in software engineering with language models, covering 70+ models, 40+ evaluation tasks, 180+ datasets, and 900 related works.
• Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation
  19 Sep 2023 1 repository listed
  Data contamination in model evaluation is getting increasingly prevalent as the massive training corpora of large language models often unintentionally include benchmark samples.
• AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
  8 Aug 2023 1 repository listed
  With ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question.
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
  20 Jul 2023 1 repository listed Syntology ran 5 of 5 samples · 0 unverified · 5 pointer-only (licence)
  Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction.
• C-STS: Conditional Semantic Textual Similarity
  24 May 2023 1 repository listed Syntology ran 2 of 4 samples · 2 unverified · 4 pointer-only (licence)
  Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information retrieval and natural language…
• PrOnto: Language Model Evaluations for 859 Languages
  22 May 2023 1 repository listed
  Evaluation datasets are critical resources for measuring the quality of pretrained language models.
• ZJUKLAB at SemEval-2021 Task 4: Negative Augmentation with Language Model for Reading Comprehension of Abstract Meaning
  25 Feb 2021 1 repository listed
  This paper presents our systems for the three Subtasks of SemEval Task4: Reading Comprehension of Abstract Meaning (ReCAM).

Syntology lines on 10 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.