PieAPP dataset

The PieAPP dataset is a large-scale dataset used for training and testing perceptually-consistent image-error prediction algorithms. The dataset can be downloaded from: server containing a zip file with all data (2.2GB) or Google Drive (ideal for quick browsing).

The dataset contains undistorted high-quality reference images and several distorted versions of these reference images. Pairs of distorted images corresponding to a reference image are labeled with probability of preference labels. These labels indicate the fraction of human population that considers one image to be visually closer to the reference over another in the pair. To ensure reliable pairwise probability of preference labels, we query 40 human subjects via Amazon Mechanical Turk for each image pair. We then obtain the percentage of people who selected image A over B as the ground-truth label for this pair, which is the probability of preference of A over B (the supplementary document explains the choice of using 40 human subjects to capture accurate probabilities). This approach is more robust because it is easier to identify the visually closer image than to assign quality scores, and does not suffer from set-dependency or scalability issues like Swiss tournaments since we never label the images with per-image quality scores (see the associated paper and supplementary document for issues with such existing labeling schemes). A pairwise learning framework, discussed in the paper, can be used to train image error predictors on the PieAPP dataset.

Dataset statistics

We make this dataset available for non-commercial and educational purposes only. The dataset contains a total of 200 undistorted reference images, divided into train / validation / test split. These reference images are derived from the Waterloo Exploration Dataset. We release the subset of 200 reference images used in PieAPP from the Waterloo Exploration Dataset with permissions for non-commercial, educational, use from the authors. The users of the PieAPP dataset are requested to cite the Waterloo Exploration Dataset for the reference images, along with PieAPP dataset, as mentioned here.

The training + validation set contain a total of 160 reference images and test set contains 40 reference images. A total of 19,680 distorted images are generated for the train/val set and pairwise probability of preference labels for 77,280 image pairs are made available (derived from querying 40 human subjects for a pairwise comparison + max-likelihood estimation of some missing pairs).

For test set, 15 distorted images per reference (total 600 distorted images) are created and all possible pairwise comparisons (total 4200) are performed to label each image pair with a probability of preference derived from 40 human subjects' votes.

Overall, the PieAPP dataset provides a total of 20,280 distorted images derived from 200 reference images, and 81,480 pairwise probability-of-preference labels.

More details of dataset collection can be found in Sec.4 of the paper and supplementary document.