{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-prism-alignment-project-what","title":"The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models","arxiv_id":"2404.16019","date":"2024-04-24","proceeding":null,"authors":["Hannah Rose Kirk","Alexander Whitefield","Paul Röttger","Andrew Bean","Katerina Margatina","Juan Ciro","Rafael Mosquera","Max Bartolo","Adina Williams","He He","Bertie Vidgen","Scott A. Hale"],"abstract":"Human feedback is central to the alignment of Large Language Models (LLMs). However, open questions remain about methods (how), domains (where), people (who) and objectives (to what end) of feedback processes. To navigate these questions, we introduce PRISM, a dataset that maps the sociodemographics and stated preferences of 1,500 diverse participants from 75 countries, to their contextual preferences and fine-grained feedback in 8,011 live conversations with 21 LLMs. With PRISM, we contribute (i) wider geographic and demographic participation in feedback; (ii) census-representative samples for two countries (UK, US); and (iii) individualised ratings that link to detailed participant profiles, permitting personalisation and attribution of sample artefacts. We target subjective and multicultural perspectives on value-laden and controversial issues, where we expect interpersonal and cross-cultural disagreement. We use PRISM in three case studies to demonstrate the need for careful consideration of which humans provide what alignment data.","url_abs":"https://arxiv.org/abs/2404.16019v2","url_pdf":"https://arxiv.org/pdf/2404.16019v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-prism-alignment-project-what","repo_url":"https://github.com/hannahkirk/prism-alignment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"navigate","task_name":"Navigate"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.16019","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.16019"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hannahkirk/prism-alignment","reach":{"status":"ok"}}],"summary":{"ran":15,"unverified":3},"by_repo_kind":{"official":{"samples":18,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"8f893fc5c2101e0b","entry":"c_tf_idf","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_analysis/clustering.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_analysis/clustering.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f893fc5c2101e0b"}},{"code_sha256_prefix":"7b538c44dd3ca5bb","entry":"calculate_average_free_text_length","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/crowdwork_quality_control/run_worker_checks.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/crowdwork_quality_control/run_worker_checks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b538c44dd3ca5bb"}},{"code_sha256_prefix":"cd200f681e43c54f","entry":"calculate_open_spots","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/crowdwork_quality_control/run_study_checks.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/crowdwork_quality_control/run_study_checks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd200f681e43c54f"}},{"code_sha256_prefix":"74c3d4de78cdae10","entry":"calculate_overquota","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/crowdwork_quality_control/run_study_checks.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/crowdwork_quality_control/run_study_checks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74c3d4de78cdae10"}},{"code_sha256_prefix":"6ea36095a1b7452b","entry":"calculate_within_turn_rank","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_analysis/ranking.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_analysis/ranking.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ea36095a1b7452b"}},{"code_sha256_prefix":"1dae226470241792","entry":"convert_string_response_to_df","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_processing/lm_label_cleaner.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_processing/lm_label_cleaner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1dae226470241792"}},{"code_sha256_prefix":"0091aa0602580bde","entry":"count_default_sliders","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/crowdwork_quality_control/run_worker_checks.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/crowdwork_quality_control/run_worker_checks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0091aa0602580bde"}},{"code_sha256_prefix":"926a516d46122b0c","entry":"count_prefer_not_say","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/crowdwork_quality_control/run_worker_checks.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/crowdwork_quality_control/run_worker_checks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"926a516d46122b0c"}},{"code_sha256_prefix":"1cdb94fc2a59a796","entry":"get_additional_utterance_metrics","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_analysis/ranking.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_analysis/ranking.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cdb94fc2a59a796"}},{"code_sha256_prefix":"d885e3770a2c08b3","entry":"label_descriptors","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_processing/lm_label_cleaner.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_processing/lm_label_cleaner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d885e3770a2c08b3"}},{"code_sha256_prefix":"e0331e63f2954435","entry":"list_studies_in_project","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_collection/pull_crowdworkers.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_collection/pull_crowdworkers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0331e63f2954435"}},{"code_sha256_prefix":"e194f3f5f97031fd","entry":"normalize_score","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_analysis/ranking.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_analysis/ranking.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e194f3f5f97031fd"}},{"code_sha256_prefix":"67e9ce9ae671abef","entry":"preprocess","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_analysis/free_text.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_analysis/free_text.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67e9ce9ae671abef"}},{"code_sha256_prefix":"2879e02caea7b459","entry":"prolific_formula","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/crowdwork_quality_control/run_study_checks.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/crowdwork_quality_control/run_study_checks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2879e02caea7b459"}},{"code_sha256_prefix":"2187d7a5298d1250","entry":"simple_text_clean","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_processing/lm_label_cleaner.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_processing/lm_label_cleaner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2187d7a5298d1250"}},{"code_sha256_prefix":"c845facf4609322a","entry":"extract_top_words_per_cluster","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_analysis/clustering.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_analysis/clustering.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c845facf4609322a"}},{"code_sha256_prefix":"9e94383839a29a79","entry":"get_project","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_collection/pull_crowdworkers.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_collection/pull_crowdworkers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e94383839a29a79"}},{"code_sha256_prefix":"3524651c553fbb27","entry":"list_study_submissions","repo":"hannahkirk/prism-alignment","repo_kind":"official","path":"src/data_collection/pull_crowdworkers.py","file_url":"https://github.com/hannahkirk/prism-alignment/blob/HEAD/src/data_collection/pull_crowdworkers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3524651c553fbb27"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}