{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-large-language-models-transform","title":"Can Large Language Models Transform Computational Social Science?","arxiv_id":"2305.03514","date":"2023-04-12","proceeding":null,"authors":["Caleb Ziems","William Held","Omar Shaikh","Jiaao Chen","Zhehao Zhang","Diyi Yang"],"abstract":"Large Language Models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the Computational Social Science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that the performance of today's LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.","url_abs":"https://arxiv.org/abs/2305.03514v3","url_pdf":"https://arxiv.org/pdf/2305.03514v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-large-language-models-transform","repo_url":"https://github.com/salt-nlp/llms_for_css","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"persuasiveness","task_name":"Persuasiveness"}],"methods":[{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.03514","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.03514"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salt-nlp/llms_for_css","reach":null}],"summary":{"ran_draft_wrong":3,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"2ebbae6a7dc1ac69","entry":"boolify","repo":"salt-nlp/llms_for_css","repo_kind":"official","path":"data_loader.py","file_url":"https://github.com/salt-nlp/llms_for_css/blob/HEAD/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ebbae6a7dc1ac69"}},{"code_sha256_prefix":"8f7e396eb1f31e86","entry":"deep_get","repo":"salt-nlp/llms_for_css","repo_kind":"official","path":"data_loader.py","file_url":"https://github.com/salt-nlp/llms_for_css/blob/HEAD/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f7e396eb1f31e86"}},{"code_sha256_prefix":"7708c4ff4bf2ce20","entry":"get_context_column","repo":"salt-nlp/llms_for_css","repo_kind":"official","path":"data_loader.py","file_url":"https://github.com/salt-nlp/llms_for_css/blob/HEAD/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7708c4ff4bf2ce20"}},{"code_sha256_prefix":"5dadda17530e1e09","entry":"tokenized_labelset","repo":"salt-nlp/llms_for_css","repo_kind":"official","path":"test_official_chat_css.py","file_url":"https://github.com/salt-nlp/llms_for_css/blob/HEAD/test_official_chat_css.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5dadda17530e1e09"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}