{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/code-simulation-challenges-for-large-language","title":"Code Simulation Challenges for Large Language Models","arxiv_id":"2401.09074","date":"2024-01-17","proceeding":null,"authors":["Emanuele La Malfa","Christoph Weinhuber","Orazio Torre","Fangru Lin","Samuele Marro","Anthony Cohn","Nigel Shadbolt","Michael Wooldridge"],"abstract":"Many reasoning, planning, and problem-solving tasks share an intrinsic algorithmic nature: correctly simulating each step is a sufficient condition to solve them correctly. This work studies to what extent Large Language Models (LLMs) can simulate coding and algorithmic tasks to provide insights into general capabilities in such algorithmic reasoning tasks. We introduce benchmarks for straight-line programs, code that contains critical paths, and approximate and redundant instructions. We further assess the simulation capabilities of LLMs with sorting algorithms and nested loops and show that a routine's computational complexity directly affects an LLM's ability to simulate its execution. While the most powerful LLMs exhibit relatively strong simulation capabilities, the process is fragile, seems to rely heavily on pattern recognition, and is affected by memorisation. We propose a novel off-the-shelf prompting method, Chain of Simulation (CoSm), which instructs LLMs to simulate code execution line by line/follow the computation pattern of compilers. CoSm efficiently helps LLMs reduce memorisation and shallow pattern recognition while improving simulation performance. We consider the success of CoSm in code simulation to be inspirational for other general routine simulation reasoning tasks.","url_abs":"https://arxiv.org/abs/2401.09074v4","url_pdf":"https://arxiv.org/pdf/2401.09074v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"code-simulation-challenges-for-large-language","repo_url":"https://github.com/emanuelelm/codesimulation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.09074","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.09074"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/emanuelelm/codesimulation","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"c6f0bf31fd2574f4","entry":"fibo","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"cosm_generic_problems.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/cosm_generic_problems.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c6f0bf31fd2574f4"}},{"code_sha256_prefix":"60afad91ae40ec9b","entry":"get_problems","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"cosm_generic_problems.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/cosm_generic_problems.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"60afad91ae40ec9b"}},{"code_sha256_prefix":"d6fe64f75cf28d10","entry":"metafunction","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"approximate_computation.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/approximate_computation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d6fe64f75cf28d10"}},{"code_sha256_prefix":"fa8a035d10127882","entry":"metafunction","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"critical_path.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/critical_path.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fa8a035d10127882"}},{"code_sha256_prefix":"f2202365e6963b1b","entry":"metafunction","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"nested_loops.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/nested_loops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f2202365e6963b1b"}},{"code_sha256_prefix":"864b4f87dcd43604","entry":"metafunction","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"parallel_computation.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/parallel_computation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"864b4f87dcd43604"}},{"code_sha256_prefix":"10cb54375e5fb533","entry":"metafunction","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"redundant_computation.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/redundant_computation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"10cb54375e5fb533"}},{"code_sha256_prefix":"6565f7e15347a223","entry":"metafunction","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"straight_line.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/straight_line.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6565f7e15347a223"}},{"code_sha256_prefix":"8a49c89cd9b087b8","entry":"queryllama","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"utils.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8a49c89cd9b087b8"}},{"code_sha256_prefix":"8d7082af814fd4a3","entry":"queryllamacode","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"utils.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8d7082af814fd4a3"}},{"code_sha256_prefix":"a2c8416aac16581b","entry":"sort","repo":"emanuelelm/codesimulation","repo_kind":"official","path":"cosm_generic_problems.py","file_url":"https://github.com/emanuelelm/codesimulation/blob/HEAD/cosm_generic_problems.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a2c8416aac16581b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}