LMC (Language Model Council)

The Language Model Council (LMC) is a novel benchmarking framework proposed to address the challenge of ranking Large Language Models (LLMs) on highly subjective tasks¹. These tasks can include areas related to emotional intelligence, creative writing, or persuasiveness, which often lack majoritarian human agreement¹.

The LMC operates through a democratic process to¹: 1. Formulate a test set through equal participation, 2. Administer the test among council members, and 3. Evaluate responses as a collective jury¹.

The council is composed of the newest LLMs and they are deployed on an open-ended emotional intelligence task: responding to interpersonal dilemmas¹. The results from the LMC have shown to produce rankings that are more separable, robust, and less biased than those from any individual LLM judge¹. It is also more consistent with a human-established leaderboard compared to other benchmarks¹.

This benchmarking framework is designed to encourage the healthy development of the field, particularly through the lens of mathematical reasoning tasks². It provides a more comprehensive and fair comparison of different LLMs, especially for tasks that are highly subjective and often lack majoritarian human agreement¹.

(1) Language Model Council: Benchmarking Foundation Models on Highly .... https://arxiv.org/abs/2406.08598. (2) GitHub - GAIR-NLP/benbench: Benchmarking Benchmark Leakage in Large .... https://github.com/GAIR-NLP/benbench. (3) LLM Benchmarks: Understanding Language Model Performance. https://humanloop.com/blog/llm-benchmarks. (4) One Billion Word Benchmark for Measuring Progress in Statistical .... https://research.google.com/pubs/pub41880.html?source=post_page---------------------------.