Pick-a-Filter

Pick-a-Filter is a semi-synthetic dataset constructed from Pick-a-Pic v1 to measure the capability of text-to-image models of adapting to heterogeneous preferences. We assign users from V1 randomly into two groups: those who prefer blue, cooler image tones (G1) and those who prefer red, warmer image tones (G2). After constructing this split, we apply the following logic to construct the dataset:

1. Apply “winning” and “losing” filters to appropriate images depending on label. For G1 the winning filter is blue, and for G2 the winning filter is red.
2. Randomly shortlist β % of samples to add filters. The remaining (1 − β) % of samples will remain unaltered (default images from Pick-a-Pic v1). The hyperparameter β is called the mixture ratio.
3. Randomly select 50% of above-shortlisted samples to apply a filter to only the winning image, and the remaining 50% to apply a filter to only losing image

We add these sources of randomness to make learning preferences on Pick-a-Filter less prone to hacking (e.g. the model could trivially learn to predict an image with a filter as the preferred image).

We provide the OpenCLIP H/14 embeddings of all versions of Pick-a-Filter used in PAL in this repository, with β varying from 0.0 (no filtering) to 1.0 (all images are filtered). For example, pick-a-filter with β = 0.4 is available in paf_0.4_mix_ratio.zip.