{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/graphomni-a-comprehensive-and-extendable","title":"GraphOmni: A Comprehensive and Extendable Benchmark Framework for Large Language Models on Graph-theoretic Tasks","arxiv_id":"2504.12764","date":"2025-04-17","proceeding":null,"authors":["Hao Xu","Xiangru Jian","Xinjian Zhao","Wei Pang","Chao Zhang","Suyuchen Wang","Qixin Zhang","Zhengyuan Dong","Joao Monteiro","Bang Liu","Qiuzhuang Sun","Tianshu Yu"],"abstract":"This paper introduces GraphOmni, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs on graph-theoretic tasks articulated in natural language. GraphOmni encompasses diverse graph types, serialization formats, and prompting schemes, significantly exceeding prior efforts in both scope and depth. Through extensive systematic evaluation, we identify critical interactions among these dimensions, demonstrating their substantial impact on model performance. Our experiments reveal that state-of-the-art models like Claude-3.5 and o4-mini consistently outperform other models, yet even these leading models exhibit substantial room for improvement. Performance variability is evident depending on the specific combinations of factors we considered, underscoring the necessity of comprehensive evaluations across these interconnected dimensions. Additionally, we observe distinct impacts of serialization and prompting strategies between open-source and closed-source models, encouraging the development of tailored approaches. Motivated by the findings, we also propose a reinforcement learning-inspired framework that adaptively selects the optimal factors influencing LLM reasoning capabilities. This flexible and extendable benchmark not only deepens our understanding of LLM performance on structured tasks but also provides a robust foundation for advancing research in LLM-based graph reasoning. The code and datasets are available at https://github.com/GAI-Community/GraphOmni.","url_abs":"https://arxiv.org/abs/2504.12764v3","url_pdf":"https://arxiv.org/pdf/2504.12764v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"graphomni-a-comprehensive-and-extendable","repo_url":"https://github.com/gai-community/graphomni","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"graphomni-a-comprehensive-and-extendable","repo_url":"https://github.com/more1643127748/graphomni","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2504.12764","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.12764"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gai-community/graphomni","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/more1643127748/graphomni","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":27},"by_repo_kind":{"official":{"samples":27,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"83b83cb8a93f63f4","entry":"build_graph","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/verification.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/verification.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83b83cb8a93f63f4"}},{"code_sha256_prefix":"d7ca9d0e9d027761","entry":"build_graph_from_csv","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/verification.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/verification.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7ca9d0e9d027761"}},{"code_sha256_prefix":"e3b4cb8414c7b5ec","entry":"check_and_download_model","repo":"gai-community/graphomni","repo_kind":"official","path":"openllm/predict.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/openllm/predict.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3b4cb8414c7b5ec"}},{"code_sha256_prefix":"138606b1a03b7b4d","entry":"convert_english_numbers","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/triangle_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/triangle_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"138606b1a03b7b4d"}},{"code_sha256_prefix":"c4fa9aad367464fa","entry":"count_tokens","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/token.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/token.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4fa9aad367464fa"}},{"code_sha256_prefix":"70beee368bff933c","entry":"create_adj_matrix","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/token.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/token.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70beee368bff933c"}},{"code_sha256_prefix":"b16ac88a9e21c924","entry":"eval_connectivity","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/connectivity_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/connectivity_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b16ac88a9e21c924"}},{"code_sha256_prefix":"a7c5dc906f2a4be7","entry":"eval_cycle","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/cycle_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/cycle_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7c5dc906f2a4be7"}},{"code_sha256_prefix":"2d2637dd0f93dd94","entry":"eval_diameter_number","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/diameter_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/diameter_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d2637dd0f93dd94"}},{"code_sha256_prefix":"2f51bb646f57b609","entry":"evaltask","repo":"gai-community/graphomni","repo_kind":"official","path":"main_closemodel.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/main_closemodel.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f51bb646f57b609"}},{"code_sha256_prefix":"21523d87e7f7f8d1","entry":"evaluate_bfsorder","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/bfsorder_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/bfsorder_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21523d87e7f7f8d1"}},{"code_sha256_prefix":"a336e706c03b0e4e","entry":"evaluate_shortest_path","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/shortest_path_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/shortest_path_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a336e706c03b0e4e"}},{"code_sha256_prefix":"3efcfb2d6b5a05d1","entry":"extract_and_check_numbers_bfsorder","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/bfsorder_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/bfsorder_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3efcfb2d6b5a05d1"}},{"code_sha256_prefix":"b7515efb0fb3c7a9","entry":"extract_and_check_numbers_dimater","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/diameter_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/diameter_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7515efb0fb3c7a9"}},{"code_sha256_prefix":"d61be97e2d23280d","entry":"extract_and_check_numbers_path","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/shortest_path_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/shortest_path_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d61be97e2d23280d"}},{"code_sha256_prefix":"d8333297634fc3ad","entry":"extract_and_check_numbers_triangle","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/triangle_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/triangle_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8333297634fc3ad"}},{"code_sha256_prefix":"08ecf1166f290426","entry":"extract_data_from_jsonl","repo":"gai-community/graphomni","repo_kind":"official","path":"closedllm/batchapi_openai_6tasks_retrive.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/closedllm/batchapi_openai_6tasks_retrive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08ecf1166f290426"}},{"code_sha256_prefix":"f6524b105c2bfbee","entry":"extract_numbers_before_node","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/bfsorder_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/bfsorder_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6524b105c2bfbee"}},{"code_sha256_prefix":"a1917cc156eef5d6","entry":"extract_relevant_sentences","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/connectivity_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/connectivity_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1917cc156eef5d6"}},{"code_sha256_prefix":"8027b8a1f685b048","entry":"extract_relevant_sentences","repo":"more1643127748/graphomni","repo_kind":"official","path":"eval_fun/cycle_eval_fun.py","file_url":"https://github.com/more1643127748/graphomni/blob/HEAD/eval_fun/cycle_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8027b8a1f685b048"}},{"code_sha256_prefix":"5474a9d6201092ed","entry":"filter_sentences","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/connectivity_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/connectivity_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5474a9d6201092ed"}},{"code_sha256_prefix":"dd84d21546412fc1","entry":"find_batch_ids_in_logs","repo":"gai-community/graphomni","repo_kind":"official","path":"closedllm/batchapi_openai_6tasks_retrive.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/closedllm/batchapi_openai_6tasks_retrive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd84d21546412fc1"}},{"code_sha256_prefix":"f700f4d3a7aa2d5c","entry":"get_edges_from_str","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/token.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/token.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f700f4d3a7aa2d5c"}},{"code_sha256_prefix":"e80463f3f6994f9a","entry":"getquery","repo":"gai-community/graphomni","repo_kind":"official","path":"main_closemodel.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/main_closemodel.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e80463f3f6994f9a"}},{"code_sha256_prefix":"4fe8ce25424e3f7f","entry":"loadqery","repo":"gai-community/graphomni","repo_kind":"official","path":"closedllm/batchapi_openai_6tasks.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/closedllm/batchapi_openai_6tasks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fe8ce25424e3f7f"}},{"code_sha256_prefix":"75a1d893016bda9f","entry":"merge_short_paragraphs","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/diameter_eval_fun.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/diameter_eval_fun.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75a1d893016bda9f"}},{"code_sha256_prefix":"090ce4b1b724e715","entry":"validate_bfs","repo":"gai-community/graphomni","repo_kind":"official","path":"eval_fun/verification.py","file_url":"https://github.com/gai-community/graphomni/blob/HEAD/eval_fun/verification.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"090ce4b1b724e715"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}