CV-Bench (Cambrian Vision-Centric Benchmark)

The Cambrian Vision-Centric Benchmark (CV-Bench) is designed to address the limitations of existing vision-centric benchmarks by providing a comprehensive evaluation framework for multimodal large language models (MLLMs). With 2,638 manually-inspected examples, CV-Bench significantly surpasses other vision-centric MLLM benchmarks, offering 3.5 times more examples than RealWorldQA and 8.8 times more than MMVP.

Motivation and Content Summary:

CV-Bench repurposes standard vision benchmarks such as ADE20K, COCO, and Omni3D to assess models on classic vision tasks within a multimodal context. Leveraging the rich ground truth annotations from these benchmarks, natural language questions are formulated to probe the fundamental 2D and 3D understanding of models.

Potential Use Cases:

• Evaluating the spatial relationship and object counting capabilities of models (2D understanding).
• Assessing the depth order and relative distance understanding of models (3D understanding).
• Benchmarking the performance of multimodal models in both vision-specific and cross-modal tasks.

Dataset Characteristics:

• 2D Understanding Tasks:
• Spatial Relationship: Determine the relative position of an object with respect to the anchor object, considering left-right or top-bottom relationships.

• Object Count: Determine the number of instances present in the image.

• 3D Understanding Tasks:

• Depth Order: Determine which of the two distinct objects is closer to the camera.
• Relative Distance: Determine which of the two distinct objects is closer to the anchor object.

Curation Process:

Questions for each task are programmatically constructed and then manually inspected to ensure clarity and accuracy. Any unclear, ambiguous, or erroneous questions are removed to maintain the benchmark's reliability.