{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pick-a-pic-an-open-dataset-of-user","title":"Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation","arxiv_id":"2305.01569","date":"2023-05-02","proceeding":"NeurIPS 2023 11","authors":["Yuval Kirstain","Adam Polyak","Uriel Singer","Shahbuland Matiana","Joe Penna","Omer Levy"],"abstract":"The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.","url_abs":"https://arxiv.org/abs/2305.01569v2","url_pdf":"https://arxiv.org/pdf/2305.01569v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pick-a-pic-an-open-dataset-of-user","repo_url":"https://github.com/yuvalkirstain/pickscore","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"preference-mapping","task_name":"Preference Mapping"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.01569","atlas_url":"https://app.syntology.ai/?focus=2305.01569","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.01569"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuvalkirstain/pickscore","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"889813052606c2ee","entry":"instantiate_dummy_lr_scheduler","repo":"yuvalkirstain/pickscore","repo_kind":"official","path":"trainer/lr_schedulers/constant_with_warmup.py","file_url":"https://github.com/yuvalkirstain/pickscore/blob/HEAD/trainer/lr_schedulers/constant_with_warmup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"889813052606c2ee"}},{"code_sha256_prefix":"8083a7b336edee87","entry":"instantiate_with_cfg","repo":"yuvalkirstain/pickscore","repo_kind":"official","path":"trainer/configs/configs.py","file_url":"https://github.com/yuvalkirstain/pickscore/blob/HEAD/trainer/configs/configs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8083a7b336edee87"}},{"code_sha256_prefix":"ef1c92590a58880a","entry":"simple_collate","repo":"yuvalkirstain/pickscore","repo_kind":"official","path":"trainer/datasetss/clip_hf_dataset.py","file_url":"https://github.com/yuvalkirstain/pickscore/blob/HEAD/trainer/datasetss/clip_hf_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef1c92590a58880a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}