{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-second-thought-let-s-not-think-step-by","title":"On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning","arxiv_id":"2212.08061","date":"2022-12-15","proceeding":null,"authors":["Omar Shaikh","Hongxin Zhang","William Held","Michael Bernstein","Diyi Yang"],"abstract":"Generating a Chain of Thought (CoT) has been shown to consistently improve large language model (LLM) performance on a wide range of NLP tasks. However, prior work has mainly focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situated contexts. Concretely, we perform a controlled evaluation of zero-shot CoT across two socially sensitive domains: harmful questions and stereotype benchmarks. We find that zero-shot CoT reasoning in sensitive domains significantly increases a model's likelihood to produce harmful or undesirable output, with trends holding across different prompt formats and model variants. Furthermore, we show that harmful CoTs increase with model size, but decrease with improved instruction following. Our work suggests that zero-shot CoT should be used with caution on socially important tasks, especially when marginalized groups or sensitive topics are involved.","url_abs":"https://arxiv.org/abs/2212.08061v2","url_pdf":"https://arxiv.org/pdf/2212.08061v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-second-thought-let-s-not-think-step-by","repo_url":"https://github.com/salt-nlp/chain-of-thought-bias","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.08061","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.08061"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salt-nlp/chain-of-thought-bias","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0b6870c8a431430b","entry":"get_completion","repo":"salt-nlp/chain-of-thought-bias","repo_kind":"official","path":"08_qa_bad.py","file_url":"https://github.com/salt-nlp/chain-of-thought-bias/blob/HEAD/08_qa_bad.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b6870c8a431430b"}},{"code_sha256_prefix":"b044186755a41420","entry":"matching_heuristic","repo":"salt-nlp/chain-of-thought-bias","repo_kind":"official","path":"prompts.py","file_url":"https://github.com/salt-nlp/chain-of-thought-bias/blob/HEAD/prompts.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b044186755a41420"}},{"code_sha256_prefix":"921532cfebdc2fd9","entry":"prompt_template_bigbench_c","repo":"salt-nlp/chain-of-thought-bias","repo_kind":"official","path":"prompts.py","file_url":"https://github.com/salt-nlp/chain-of-thought-bias/blob/HEAD/prompts.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"921532cfebdc2fd9"}},{"code_sha256_prefix":"dc01f9ec2ba417be","entry":"prompt_template_q_c","repo":"salt-nlp/chain-of-thought-bias","repo_kind":"official","path":"prompts.py","file_url":"https://github.com/salt-nlp/chain-of-thought-bias/blob/HEAD/prompts.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc01f9ec2ba417be"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}