{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/microstructures-and-accuracy-of-graph-recall","title":"Microstructures and Accuracy of Graph Recall by Large Language Models","arxiv_id":"2402.11821","date":"2024-02-19","proceeding":null,"authors":["Yanbang Wang","Hejie Cui","Jon Kleinberg"],"abstract":"Graphs data is crucial for many applications, and much of it exists in the relations described in textual format. As a result, being able to accurately recall and encode a graph described in earlier text is a basic yet pivotal ability that LLMs need to demonstrate if they are to perform reasoning tasks that involve graph-structured information. Human performance at graph recall has been studied by cognitive scientists for decades, and has been found to often exhibit certain structural patterns of bias that align with human handling of social relationships. To date, however, we know little about how LLMs behave in analogous graph recall tasks: do their recalled graphs also exhibit certain biased patterns, and if so, how do they compare with humans and affect other graph reasoning tasks? In this work, we perform the first systematical study of graph recall by LLMs, investigating the accuracy and biased microstructures (local structural patterns) in their recall. We find that LLMs not only underperform often in graph recall, but also tend to favor more triangles and alternating 2-paths. Moreover, we find that more advanced LLMs have a striking dependence on the domain that a real-world graph comes from -- by yielding the best recall accuracy when the graph is narrated in a language style consistent with its original domain.","url_abs":"https://arxiv.org/abs/2402.11821v3","url_pdf":"https://arxiv.org/pdf/2402.11821v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"microstructures-and-accuracy-of-graph-recall","repo_url":"https://github.com/abel0828/llm-graph-recall","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.11821","atlas_url":"https://app.syntology.ai/?focus=2402.11821","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.11821"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abel0828/llm-graph-recall","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"5b8ea2946ba61652","entry":"concatenate_segments","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"utils.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b8ea2946ba61652"}},{"code_sha256_prefix":"11cba458079c4ef0","entry":"get_fname","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"network_recall.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/network_recall.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11cba458079c4ef0"}},{"code_sha256_prefix":"190871a24a3d1816","entry":"get_response","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"network_recall.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/network_recall.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"190871a24a3d1816"}},{"code_sha256_prefix":"0976b2f534ca72a4","entry":"load_string","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"utils.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0976b2f534ca72a4"}},{"code_sha256_prefix":"a17be44fe4aba8a7","entry":"set_up_logger","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"utils.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a17be44fe4aba8a7"}},{"code_sha256_prefix":"298f32909bb3e9c7","entry":"generate_all_pairs","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"network_recall.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/network_recall.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"298f32909bb3e9c7"}},{"code_sha256_prefix":"0f72adf8ddbb5ec1","entry":"num_tokens_from_messages","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"api.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f72adf8ddbb5ec1"}},{"code_sha256_prefix":"b66ff4988d3f6fe1","entry":"num_tokens_from_prompt","repo":"abel0828/llm-graph-recall","repo_kind":"official","path":"api.py","file_url":"https://github.com/abel0828/llm-graph-recall/blob/HEAD/api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b66ff4988d3f6fe1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}