Browse State-of-the-Art › Language Model Evaluation
Language Model Evaluation
31 papers with code · 0 benchmarks · 0 datasets archive 2025-07-28
The task of using LLMs as evaluators of large language and vision language models.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
No dataset record in the archive lists this task.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 69 · with a code link: 31 · where Syntology ran a sample: 12 (11 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (12 of 69 tagged: 11 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 69 of 69, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
28 Dec 2023 2 repositories listedIn this work, we introduce a novel evaluation paradigm for Large Language Models (LLMs) that compels them to transition from a traditional question-answering role, akin to a student, to a solution-scoring role, akin to…
-
25 Aug 2023 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability.
-
30 Jun 2022 2 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Training and evaluating language models increasingly requires the construction of meta-datasets --diverse collections of curated data with clear provenance.
-
30 May 2025 1 repository listedFABLE provides the first diagnostic benchmark to systematically evaluate data-flow reasoning and offers insights for developing models with stronger procedural understanding.
-
19 May 2025 1 repository listedLarge Language Models (LLMs) demonstrate a notable capacity for adopting personas and engaging in role-playing.
-
17 Feb 2025 1 repository listedAspect-based sentiment analysis (ABSA) is a crucial task in information extraction and sentiment analysis, aiming to identify aspects with associated sentiment elements in text.
-
10 Jan 2025 1 repository listedGenerative AI holds significant potential for ecological and environmental applications such as monitoring, data analysis, education, and policy support.
-
6 Jan 2025 1 repository listedThe rapid development of vision language models (VLMs) demands rigorous and reliable evaluation.
-
11 Dec 2024 1 repository listedModels tuned on our augmented dataset achieve the best overall performance when compared with the same scale MLMs tuned on at most 75 times the scale of our augmented dataset, highlighting the importance of instruction…
-
6 Dec 2024 1 repository listedRecent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data.
-
6 Dec 2024 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 5 where Syntology's instrument failed) · 5 unverified (of 16 harvested samples) · 16 pointer-only (licence)Recent advances in self-supervised models for natural language, vision, and protein sequences have inspired the development of large genomic DNA language models (DNALMs).
-
14 Oct 2024 1 repository listedWhile traditional metrics like Matrix Entropy offer valuable insights, they are computationally intensive for large-scale models due to their O(n³) time complexity with Singular Value Decomposition (SVD).
-
11 Oct 2024 1 repository listedThe advancement of large language models (LLMs) has led to a greater challenge of having a rigorous and systematic evaluation of complex tasks performed, especially in enterprise applications.
-
25 Sep 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality.
-
11 Sep 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Speech language models have recently demonstrated great potential as universal speech processing systems.
-
20 Jun 2024 1 repository listedOn MMLU, using Inference-time Decontamination can lead to a decrease in the results of Phi3 and Mistral by 6.
-
20 May 2024 1 repository listedThe rapid advancement of large language models has given rise to a plethora of applications across a myriad of real-world tasks, mainly centered on aligning with human intent.
-
17 Apr 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)This paper studies the relationship between the surface form of a mathematical problem and its solvability by large language models.
-
1 Apr 2024 1 repository listedThis paper introduces Evalverse, a novel library that streamlines the evaluation of Large Language Models (LLMs) by unifying disparate evaluation tools into a single, user-friendly framework.
-
13 Mar 2024 1 repository listedThrough this gateway, users have the opportunity to submit their questions, testing the models on a personalized and potentially broader range of capabilities.
-
20 Feb 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 6 pointer-only (licence)The focus of language model evaluation has transitioned towards reasoning and knowledge-intensive tasks, driven by advancements in pretraining large models.
-
19 Dec 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (of 6 harvested samples) · 6 pointer-only (licence)LatestEval avoids data contamination by only using texts published within a recent time window, ensuring no overlap with the training corpora of pre-trained language models.
-
15 Dec 2023 1 repository listedThe success of large language models has shifted the evaluation paradigms in natural language processing (NLP).
-
14 Nov 2023 1 repository listedIn this work we systematically review the recent advancements in software engineering with language models, covering 70+ models, 40+ evaluation tasks, 180+ datasets, and 900 related works.
-
19 Sep 2023 1 repository listedData contamination in model evaluation is getting increasingly prevalent as the massive training corpora of large language models often unintentionally include benchmark samples.
-
8 Aug 2023 1 repository listedWith ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question.
-
20 Jul 2023 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction.
-
24 May 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified; every one of the 2 samples that ran constructed an object rather than computing a result (of 4 harvested samples) · 4 pointer-only (licence)Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information retrieval and natural language…
-
22 May 2023 1 repository listedEvaluation datasets are critical resources for measuring the quality of pretrained language models.
-
25 Feb 2021 1 repository listedThis paper presents our systems for the three Subtasks of SemEval Task4: Reading Comprehension of Abstract Meaning (ReCAM).
-
3 Feb 2021 1 repository listed
-
Enterprise Large Language Model Evaluation Benchmark25 Jun 2025 0 repositories listed
-
Finance Language Model Evaluation (FLaME)18 Jun 2025 0 repositories listed
-
BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models2 Jun 2025 0 repositories listed
-
R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation4 May 2025 0 repositories listed
-
Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges30 Apr 2025 0 repositories listed
-
CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation29 Apr 2025 0 repositories listed
-
UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation19 Mar 2025 0 repositories listed
-
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation13 Mar 2025 0 repositories listed
-
Predicting Liquidity-Aware Bond Yields using Causal GANs and Deep Reinforcement Learning with LLM Evaluation24 Feb 2025 0 repositories listed
-
Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation31 Dec 2024 0 repositories listed
-
LMUnit: Fine-grained Evaluation with Natural Language Unit Tests17 Dec 2024 0 repositories listed
-
Benchmarking Harmonized Tariff Schedule Classification Models4 Dec 2024 0 repositories listed
-
ViDAS: Vision-based Danger Assessment and Scoring1 Oct 2024 0 repositories listed
-
Beyond Metrics: A Critical Analysis of the Variability in Large Language Model Evaluation Frameworks29 Jul 2024 0 repositories listed
-
Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation6 Jun 2024 0 repositories listed
-
DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation24 May 2024 0 repositories listed
-
iREPO: implicit Reward Pairwise Difference based Empirical Preference Optimization24 May 2024 0 repositories listed
-
Lessons from the Trenches on Reproducible Evaluation of Language Models23 May 2024 0 repositories listed
-
Generalization Measures for Zero-Shot Cross-Lingual Transfer24 Apr 2024 0 repositories listed
-
Rethinking Generative Large Language Model Evaluation for Semantic Comprehension12 Mar 2024 0 repositories listed
-
Advancing Chinese biomedical text mining with community challenges7 Mar 2024 0 repositories listed
-
KMMLU: Measuring Massive Multitask Language Understanding in Korean18 Feb 2024 0 repositories listed
-
CPSDBench: A Large Language Model Evaluation Benchmark and Baseline for Chinese Public Security Domain11 Feb 2024 0 repositories listed
-
Elo Uncovered: Robustness and Best Practices in Language Model Evaluation29 Nov 2023 0 repositories listed
-
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation23 Oct 2023 0 repositories listed
-
MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation21 Oct 2023 0 repositories listed Syntology 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 7 harvested samples) · 7 pointer-only (licence)
-
Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing19 Oct 2023 0 repositories listed
-
Pseudointelligence: A Unifying Framework for Language Model Evaluation18 Oct 2023 0 repositories listed
-
Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs22 Apr 2023 0 repositories listed
-
Controlling for Stereotypes in Multimodal Language Model Evaluation3 Feb 2023 0 repositories listed
-
A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation1 Jul 2022 0 repositories listed
-
BPoMP: The Benchmark of Poetic Minimal Pairs – Limericks, Rhyme, and Narrative Coherence1 Sep 2021 0 repositories listed
-
Language Model Evaluation in Open-ended Text Generation8 Aug 2021 0 repositories listed
-
Language Model Evaluation Beyond Perplexity31 May 2021 0 repositories listed
-
CLiMP: A Benchmark for Chinese Language Model Evaluation26 Jan 2021 0 repositories listed
-
Improving Explainable Recommendations with Synthetic Reviews18 Jul 2018 0 repositories listed
-
Contrastive Entropy: A new evaluation metric for unnormalized language models3 Jan 2016 0 repositories listed
Syntology lines on 12 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections