HEIM (Holistic Evaluation of Text-to-Image Models)

HEIM stands for Holistic Evaluation of Text-To-Image Models. It is a comprehensive benchmark designed to assess the capabilities and risks of text-to-image generation models. Unlike previous evaluations that primarily focused on image-text alignment and image quality, HEIM considers 12 different aspects that are crucial for real-world model deployment:

1. Image-Text Alignment
2. Image Quality
3. Aesthetics
4. Originality
5. Reasoning
6. Knowledge
7. Bias
8. Toxicity
9. Fairness
10. Robustness
11. Multilinguality
12. Efficiency

By curating scenarios that encompass these aspects, HEIM evaluates state-of-the-art text-to-image models. Interestingly, no single model excels in all aspects; different models demonstrate strengths in different areas. For transparency, all prompts, generated images, and results are available on the HEIM website for exploration and study. Additionally, the GitHub repository provides a collection of models accessible via a unified API, along with metrics beyond accuracy, such as efficiency, bias, and toxicity.