{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/graph-enhanced-large-language-models-in","title":"Graph-enhanced Large Language Models in Asynchronous Plan Reasoning","arxiv_id":"2402.02805","date":"2024-02-05","proceeding":null,"authors":["Fangru Lin","Emanuele La Malfa","Valentin Hofmann","Elle Michelle Yang","Anthony Cohn","Janet B. Pierrehumbert"],"abstract":"Planning is a fundamental property of human intelligence. Reasoning about asynchronous plans is challenging since it requires sequential and parallel planning to optimize time costs. Can large language models (LLMs) succeed at this task? Here, we present the first large-scale study investigating this question. We find that a representative set of closed and open-source LLMs, including GPT-4 and LLaMA-2, behave poorly when not supplied with illustrations about the task-solving process in our benchmark AsyncHow. We propose a novel technique called Plan Like a Graph (PLaG) that combines graphs with natural language prompts and achieves state-of-the-art results. We show that although PLaG can boost model performance, LLMs still suffer from drastic degradation when task complexity increases, highlighting the limits of utilizing LLMs for simulating digital devices. We see our study as an exciting step towards using LLMs as efficient autonomous agents. Our code and data are available at https://github.com/fangru-lin/graph-llm-asynchow-plan.","url_abs":"https://arxiv.org/abs/2402.02805v2","url_pdf":"https://arxiv.org/pdf/2402.02805v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"graph-enhanced-large-language-models-in","repo_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.02805","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02805"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/fangru-lin/graph-llm-asynchow-plan","reach":null}],"summary":{"ran_fixture":2,"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8ae89fd11601a661","entry":"createGraph","repo":"fangru-lin/graph-llm-asynchow-plan","repo_kind":"official","path":"prototypical/longest_path.py","file_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan/blob/HEAD/prototypical/longest_path.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ae89fd11601a661"}},{"code_sha256_prefix":"beb6f978b0460558","entry":"generate_nshot_prompt_graph","repo":"fangru-lin/graph-llm-asynchow-plan","repo_kind":"official","path":"benchmark_llm/benchmark_llm.py","file_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan/blob/HEAD/benchmark_llm/benchmark_llm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"beb6f978b0460558"}},{"code_sha256_prefix":"3b90664fabed7d8a","entry":"generate_prompt","repo":"fangru-lin/graph-llm-asynchow-plan","repo_kind":"official","path":"prototypical/longest_path.py","file_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan/blob/HEAD/prototypical/longest_path.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b90664fabed7d8a"}},{"code_sha256_prefix":"48e26b4369fa09b7","entry":"prompt_model_for_all_templates","repo":"fangru-lin/graph-llm-asynchow-plan","repo_kind":"official","path":"benchmark_llm/benchmark_llm.py","file_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan/blob/HEAD/benchmark_llm/benchmark_llm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48e26b4369fa09b7"}},{"code_sha256_prefix":"072a250c0113af7d","entry":"prompt_model_for_combined_templates","repo":"fangru-lin/graph-llm-asynchow-plan","repo_kind":"official","path":"benchmark_llm/benchmark_llm.py","file_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan/blob/HEAD/benchmark_llm/benchmark_llm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"072a250c0113af7d"}},{"code_sha256_prefix":"73847c2701147093","entry":"read_prompts","repo":"fangru-lin/graph-llm-asynchow-plan","repo_kind":"official","path":"prototypical/longest_path.py","file_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan/blob/HEAD/prototypical/longest_path.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73847c2701147093"}},{"code_sha256_prefix":"ee0e9e373d8175be","entry":"longest_simple_paths","repo":"fangru-lin/graph-llm-asynchow-plan","repo_kind":"official","path":"prototypical/longest_path.py","file_url":"https://github.com/fangru-lin/graph-llm-asynchow-plan/blob/HEAD/prototypical/longest_path.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee0e9e373d8175be"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}