{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/questioning-the-survey-responses-of-large","title":"Questioning the Survey Responses of Large Language Models","arxiv_id":"2306.07951","date":"2023-06-13","proceeding":null,"authors":["Ricardo Dominguez-Olmedo","Moritz Hardt","Celestine Mendler-Dünner"],"abstract":"Surveys have recently gained popularity as a tool to study large language models. By comparing survey responses of models to those of human reference populations, researchers aim to infer the demographics, political opinions, or values best represented by current language models. In this work, we critically examine this methodology on the basis of the well-established American Community Survey by the U.S. Census Bureau. Evaluating 43 different language models using de-facto standard prompting methodologies, we establish two dominant patterns. First, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter \"A\". Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses, irrespective of model size or pre-training data. As a result, in contrast to conjectures from prior work, survey-derived alignment measures often permit a simple explanation: models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform for any survey under consideration.","url_abs":"https://arxiv.org/abs/2306.07951v4","url_pdf":"https://arxiv.org/pdf/2306.07951v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"questioning-the-survey-responses-of-large","repo_url":"https://github.com/socialfoundations/surveying-language-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"survey","task_name":"Survey"}],"methods":[{"method_slug":null,"method_name":"American"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.07951","atlas_url":"https://app.syntology.ai/?focus=2306.07951","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.07951"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/socialfoundations/surveying-language-models","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a582fdbf690b3ab","entry":"PUMS_to_model_generated","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/process_acs.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/process_acs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a582fdbf690b3ab"}},{"code_sha256_prefix":"c1a982017d824fbb","entry":"col2freq","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/load_responses.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/load_responses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1a982017d824fbb"}},{"code_sha256_prefix":"a1e7e5128d3218a0","entry":"get_choice_logprobs","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/fill_openai.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/fill_openai.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1e7e5128d3218a0"}},{"code_sha256_prefix":"e2560a0c5cd234dc","entry":"get_openai_logprobs","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/fill_openai.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/fill_openai.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2560a0c5cd234dc"}},{"code_sha256_prefix":"093daf2e93f4d661","entry":"load_tokenizer_model","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/utils.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"093daf2e93f4d661"}},{"code_sha256_prefix":"62fb9e3940a61e1c","entry":"move_tmp","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/utils.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62fb9e3940a61e1c"}},{"code_sha256_prefix":"ec0ed2a7a17e301d","entry":"openai_upper_bound","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/load_responses.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/load_responses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec0ed2a7a17e301d"}},{"code_sha256_prefix":"e53485caa3cf5ad4","entry":"process_naive","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/load_responses.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/load_responses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e53485caa3cf5ad4"}},{"code_sha256_prefix":"7cd6faa6ecc2e154","entry":"query_model_batch","repo":"socialfoundations/surveying-language-models","repo_kind":"official","path":"surveying_llms/fill.py","file_url":"https://github.com/socialfoundations/surveying-language-models/blob/HEAD/surveying_llms/fill.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cd6faa6ecc2e154"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}