Comparative Question Completion

Comparative Question Completion is a dataset to evaluate what do large Language Models learn.

The dataset includes short questions in natural language that make comparisons between entity pairs, for example, “is a cockroach or beetle more dangerous?”

The questions are in three subject domains: animals, cities and NBA players.

In each sentence, one of the compared entities in the sentence has been 'masked' (replaced with a [MASK] symbol). For example, for the question above the masked sentence is: “is a [MASK] or beetle more dangerous?” The dataset presents the task of automatically recovering the masked entity name, and provides the original entity for evaluation purposes. In addition to the original masked entity text (e.g., 'cockroach'), it details the respective Wikidata entity ID, (e.g., 'Q18123008').