{"url":"/dataset/lmc","name":"LMC","full_name":"Language Model Council","description_markdown":"The **Language Model Council (LMC)** is a novel benchmarking framework proposed to address the challenge of ranking Large Language Models (LLMs) on highly subjective tasks¹. These tasks can include areas related to emotional intelligence, creative writing, or persuasiveness, which often lack majoritarian human agreement¹.\r\n\r\nThe LMC operates through a democratic process to¹:\r\n1. Formulate a test set through equal participation,\r\n2. Administer the test among council members, and\r\n3. Evaluate responses as a collective jury¹.\r\n\r\nThe council is composed of the newest LLMs and they are deployed on an open-ended emotional intelligence task: responding to interpersonal dilemmas¹. The results from the LMC have shown to produce rankings that are more separable, robust, and less biased than those from any individual LLM judge¹. It is also more consistent with a human-established leaderboard compared to other benchmarks¹.\r\n\r\nThis benchmarking framework is designed to encourage the healthy development of the field, particularly through the lens of mathematical reasoning tasks². It provides a more comprehensive and fair comparison of different LLMs, especially for tasks that are highly subjective and often lack majoritarian human agreement¹.\r\n\r\n(1) Language Model Council: Benchmarking Foundation Models on Highly .... https://arxiv.org/abs/2406.08598.\r\n(2) GitHub - GAIR-NLP/benbench: Benchmarking Benchmark Leakage in Large .... https://github.com/GAIR-NLP/benbench.\r\n(3) LLM Benchmarks: Understanding Language Model Performance. https://humanloop.com/blog/llm-benchmarks.\r\n(4) One Billion Word Benchmark for Measuring Progress in Statistical .... https://research.google.com/pubs/pub41880.html?source=post_page---------------------------.","description_withheld":null,"homepage":"https://github.com/llm-council/llm-council","introduced_date":"2024-06-12","introduced_date_note":null,"introduced_by":{"paper":"/paper/language-model-council-benchmarking","title":"Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks","first_author":"Justin Zhao","url":null},"license":null,"modalities":[],"tasks":[],"languages":[],"variants":["LMC"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}