{"url":"/dataset/evalmuse-40k","name":"Evalmuse-40k","full_name":null,"description_markdown":"Recently, Text-to-Image (T2I) generation models have achieved significant advancements. Correspondingly, many automated metrics have emerged to evaluate the image-text alignment capabilities of generative models. However, the performance comparison among these automated metrics is limited by existing small datasets. Additionally, these datasets lack the capacity to assess the performance of automated metrics at a fine-grained level. In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks. In the construction process, we employ various strategies such as balanced prompt sampling and data re-annotation to ensure the diversity and reliability of our benchmark.","description_withheld":null,"homepage":"https://shh-han.github.io/EvalMuse-project/","introduced_date":"2024-12-24","introduced_date_note":null,"introduced_by":{"paper":"/paper/evalmuse-40k-a-reliable-and-fine-grained","title":"EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation","first_author":"Shuhao Han","url":null},"license":null,"modalities":[],"tasks":[{"name":"Image Generation","url":"/task/image-generation","datasets_with_task":"/datasets/task/image-generation"}],"languages":[],"variants":["Evalmuse-40k"],"data_loaders":[],"num_papers_in_archive":2,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}