WANDS (Wayfair ANnotation Dataset)

The dataset contains:

• 42,994 candidate products with data comprising product class, title, description, attributes, category hierarchy, average rating, and number of reviews
• 480 search query strings with predicted product class
• 233,448 (query string, product) human relevance judgments with labels (exact match, partial match, irrelevant)

The purpose of the dataset is to evaluate retrieval models for product search in the e-commerce domain using expert judgment of whether a product is relevant to a given query. It can be used to benchmark different retrieval against each other. As of its publication in 2022, it was to the best of our knowledge the biggest such public dataset.

The accompanying publication describes in depth the annotation guidelines and process used to collect the dataset. It also includes a measure of the quality of the annotation and experimentally compares the dataset's ability to discriminate the effectiveness of different retrieval models vs other comparable evaluation datasets.