{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/code-prompting-elicits-conditional-reasoning","title":"Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs","arxiv_id":"2401.10065","date":"2024-01-18","proceeding":null,"authors":["Haritz Puerto","Martin Tutek","Somak Aditya","Xiaodan Zhu","Iryna Gurevych"],"abstract":"Reasoning is a fundamental component of language understanding. Recent prompting techniques, such as chain of thought, have consistently improved LLMs' performance on various reasoning tasks. Nevertheless, there is still little understanding of what triggers reasoning abilities in LLMs in the inference stage. In this paper, we introduce code prompting, a chain of prompts that transforms a natural language problem into code and directly prompts the LLM using the generated code without resorting to external code execution. We hypothesize that code prompts can elicit certain reasoning capabilities of LLMs trained on text and code and utilize the proposed method to improve conditional reasoning, the ability to infer different conclusions depending on the fulfillment of certain conditions. We find that code prompting exhibits a high-performance boost for multiple LLMs (up to 22.52 percentage points on GPT 3.5, 7.75 on Mixtral, and 16.78 on Mistral) across multiple conditional reasoning datasets. We then conduct comprehensive experiments to understand how code prompts trigger reasoning abilities and which capabilities are elicited in the underlying models. Our analysis of GPT 3.5 reveals that the code formatting of the input problem is essential for performance improvement. Furthermore, code prompts improve sample efficiency of in-context learning and facilitate state tracking of variables or entities.","url_abs":"https://arxiv.org/abs/2401.10065v3","url_pdf":"https://arxiv.org/pdf/2401.10065v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"code-prompting-elicits-conditional-reasoning","repo_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.10065","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.10065"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"61322bcb67f876e7","entry":"compute_metrics","repo":"ukplab/arxiv2024-conditional-reasoning-llms","repo_kind":"official","path":"src/conditionalqa/evaluation.py","file_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms/blob/HEAD/src/conditionalqa/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"61322bcb67f876e7"}},{"code_sha256_prefix":"a53867df2046c76b","entry":"create_contextualized_rationales","repo":"ukplab/arxiv2024-conditional-reasoning-llms","repo_kind":"official","path":"src/conditionalqa/oracle_retriever.py","file_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms/blob/HEAD/src/conditionalqa/oracle_retriever.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a53867df2046c76b"}},{"code_sha256_prefix":"43bb95dff6d09d00","entry":"evaluate","repo":"ukplab/arxiv2024-conditional-reasoning-llms","repo_kind":"official","path":"src/boardgameqa/evaluation.py","file_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms/blob/HEAD/src/boardgameqa/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43bb95dff6d09d00"}},{"code_sha256_prefix":"948fa8f0528211bf","entry":"get_sections","repo":"ukplab/arxiv2024-conditional-reasoning-llms","repo_kind":"official","path":"src/conditionalqa/oracle_retriever.py","file_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms/blob/HEAD/src/conditionalqa/oracle_retriever.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"948fa8f0528211bf"}},{"code_sha256_prefix":"9853eec4ddb2b039","entry":"get_summarized_doc","repo":"ukplab/arxiv2024-conditional-reasoning-llms","repo_kind":"official","path":"src/conditionalqa/oracle_retriever.py","file_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms/blob/HEAD/src/conditionalqa/oracle_retriever.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9853eec4ddb2b039"}},{"code_sha256_prefix":"167380b25c7e22fd","entry":"load_answers","repo":"ukplab/arxiv2024-conditional-reasoning-llms","repo_kind":"official","path":"src/conditionalqa/evaluation.py","file_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms/blob/HEAD/src/conditionalqa/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"167380b25c7e22fd"}},{"code_sha256_prefix":"80ea8ce64f656cc3","entry":"evaluate","repo":"ukplab/arxiv2024-conditional-reasoning-llms","repo_kind":"official","path":"src/conditionalqa/evaluation.py","file_url":"https://github.com/ukplab/arxiv2024-conditional-reasoning-llms/blob/HEAD/src/conditionalqa/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80ea8ce64f656cc3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}