{"url":"/dataset/pieapp-dataset","name":"PieAPP dataset","full_name":"PieAPP dataset","description_markdown":"The PieAPP dataset is a large-scale dataset used for training and testing perceptually-consistent image-error prediction algorithms.\r\nThe dataset can be downloaded from:  [server containing a zip file with all data](https://web.ece.ucsb.edu/~ekta/projects/PieAPPv0.1/all_data_PieAPP_dataset_CVPR_2018.zip) (2.2GB) or [Google Drive](https://drive.google.com/drive/folders/10RmBhfZFHESCXhhWq0b3BkO5z8ryw85p?usp=sharing) (ideal for quick browsing). \r\n\r\nThe dataset contains undistorted high-quality reference images and several distorted versions of these reference images. Pairs of distorted images corresponding to a reference image are labeled with **probability of preference** labels.\r\n These labels indicate the fraction of human population that considers one image to be visually closer to the reference over another in the pair.\r\nTo ensure reliable pairwise probability of preference labels, we query 40 human subjects via Amazon Mechanical Turk for each image pair.\r\nWe then obtain the percentage of people who selected image A over B as the ground-truth label for this pair, which is the probability of preference of A over B (the [supplementary document](https://openaccess.thecvf.com/content_cvpr_2018/Supplemental/3483-supp.pdf) explains the choice of using 40 human subjects to capture accurate probabilities).\r\nThis approach is more robust because it is easier to identify the visually closer image than to assign quality scores, and does not suffer from set-dependency or scalability issues like Swiss tournaments since we never label the images with  per-image quality scores (see the associated  [paper](https://openaccess.thecvf.com/content_cvpr_2018/papers/Prashnani_PieAPP_Perceptual_Image-Error_CVPR_2018_paper.pdf) and [supplementary document](https://openaccess.thecvf.com/content_cvpr_2018/Supplemental/3483-supp.pdf) for issues with such existing labeling schemes). \r\nA pairwise learning framework, discussed in the paper, can be used to train image error predictors on the PieAPP dataset.\r\n\r\n## Dataset statistics\r\n\r\nWe make this dataset available for non-commercial and educational purposes only. \r\nThe dataset contains a total of 200 undistorted reference images, divided into train / validation / test split.\r\nThese reference images are derived from the [Waterloo Exploration Dataset](https://ece.uwaterloo.ca/~k29ma/exploration/). We release the subset of 200 reference images used in PieAPP from the Waterloo Exploration Dataset with permissions for non-commercial, educational, use from the authors.\r\nThe users of the PieAPP dataset are requested to cite the Waterloo Exploration Dataset for the reference images, along with PieAPP dataset, as mentioned [here](https://github.com/prashnani/PerceptualImageError/blob/master/dataset/dataset_README.md#terms-of-usage-and-how-to-cite-this-dataset).\r\n\r\nThe training + validation set contain a total of 160 reference images and test set contains 40 reference images.\r\nA total of 19,680 distorted images are generated for the train/val set and pairwise probability of preference labels for 77,280 image pairs are made available (derived from querying 40 human subjects for a pairwise comparison + max-likelihood estimation of some missing pairs).\r\n\r\nFor test set, 15 distorted images per reference (total 600 distorted images) are created and **all possible** pairwise comparisons (total 4200) are performed to label **each** image pair with a probability of preference derived from 40 human subjects' votes.\r\n\r\nOverall, the PieAPP dataset provides a total of 20,280 distorted images derived from 200 reference images, and 81,480 pairwise probability-of-preference labels.\r\n\r\nMore details of dataset collection can be found in Sec.4 of the [paper](https://openaccess.thecvf.com/content_cvpr_2018/papers/Prashnani_PieAPP_Perceptual_Image-Error_CVPR_2018_paper.pdf) and [supplementary document](https://openaccess.thecvf.com/content_cvpr_2018/Supplemental/3483-supp.pdf).","description_withheld":null,"homepage":"https://github.com/prashnani/PerceptualImageError/blob/master/dataset/dataset_README.md","introduced_date":"2018-06-06","introduced_date_note":null,"introduced_by":{"paper":"/paper/pieapp-perceptual-image-error-assessment","title":"PieAPP: Perceptual Image-Error Assessment through Pairwise Preference","first_author":"Ekta Prashnani","url":null},"license":{"name":"Modified BSD-2 License - for Non-Commercial Use Only","url":"https://github.com/prashnani/PerceptualImageError/blob/master/LICENSE.txt"},"modalities":[{"name":"Images","url":"/datasets/modality/images"}],"tasks":[],"languages":[],"variants":["PieAPP dataset"],"data_loaders":[],"num_papers_in_archive":6,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}