{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-a-helpful-assistant-the-best-role-for","title":"When \"A Helpful Assistant\" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models","arxiv_id":"2311.10054","date":"2023-11-16","proceeding":null,"authors":["Mingqian Zheng","Jiaxin Pei","Lajanugen Logeswaran","Moontae Lee","David Jurgens"],"abstract":"Prompting serves as the major way humans interact with Large Language Models (LLM). Commercial AI systems commonly define the role of the LLM in system prompts. For example, ChatGPT uses ``You are a helpful assistant'' as part of its default system prompt. Despite current practices of adding personas to system prompts, it remains unclear how different personas affect a model's performance on objective tasks. In this study, we present a systematic evaluation of personas in system prompts. We curate a list of 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise. Through extensive analysis of 4 popular families of LLMs and 2,410 factual questions, we demonstrate that adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added. Nevertheless, further analysis suggests that the gender, type, and domain of the persona can all influence the resulting prediction accuracies. We further experimented with a list of persona search strategies and found that, while aggregating results from the best persona for each question significantly improves prediction accuracy, automatically identifying the best persona is challenging, with predictions often performing no better than random selection. Overall, our findings suggest that while adding a persona may lead to performance gains in certain settings, the effect of each persona can be largely random. Code and data are available at https://github.com/Jiaxin-Pei/Prompting-with-Social-Roles.","url_abs":"https://arxiv.org/abs/2311.10054v3","url_pdf":"https://arxiv.org/pdf/2311.10054v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-a-helpful-assistant-the-best-role-for","repo_url":"https://github.com/jiaxin-pei/prompting-with-social-roles","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.10054","atlas_url":"https://app.syntology.ai/?focus=2311.10054","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.10054"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiaxin-pei/prompting-with-social-roles","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"30bbdd7131945ece","entry":"format_choices","repo":"jiaxin-pei/prompting-with-social-roles","repo_kind":"official","path":"scripts/vllm_inference_pipeline.py","file_url":"https://github.com/jiaxin-pei/prompting-with-social-roles/blob/HEAD/scripts/vllm_inference_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30bbdd7131945ece"}},{"code_sha256_prefix":"8c2983401b9fcfe3","entry":"get_data","repo":"jiaxin-pei/prompting-with-social-roles","repo_kind":"official","path":"scripts/classifier.py","file_url":"https://github.com/jiaxin-pei/prompting-with-social-roles/blob/HEAD/scripts/classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c2983401b9fcfe3"}},{"code_sha256_prefix":"3573e7c6b7e05d39","entry":"get_key_for_value","repo":"jiaxin-pei/prompting-with-social-roles","repo_kind":"official","path":"scripts/vllm_inference_pipeline.py","file_url":"https://github.com/jiaxin-pei/prompting-with-social-roles/blob/HEAD/scripts/vllm_inference_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3573e7c6b7e05d39"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}