{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-18084","title":"WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting","arxiv_id":"2607.18084","date":"2026-07-20","proceeding":null,"authors":["Zhaokai Wang","Tianlin Gui","Jiayuan Rao","Shangzhe Di","Yihong Tang","Dingli Liang"],"abstract":"Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available. We present WorldCupArena, a dynamic benchmark for language models and deep-research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across 104 matches and 13 systems, models with similar result accuracy differ more clearly on detailed predictions. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, prompts, predictions, and evaluation scripts are open sourced at https://github.com/wzk1015/WorldCupArena.","url_abs":"https://arxiv.org/abs/2607.18084","url_pdf":"https://arxiv.org/pdf/2607.18084","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.18084","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.18084"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wzk1015/WorldCupArena","reach":null}],"summary":{"ran_draft_wrong":4,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":11,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8dff7f601498c838","entry":"_round_probs","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dff7f601498c838"}},{"code_sha256_prefix":"bc5b8e5c6c691c18","entry":"_win_probs","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc5b8e5c6c691c18"}},{"code_sha256_prefix":"4b3e599a3fbe6abe","entry":"derive_win_probs_from_score_dist","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b3e599a3fbe6abe"}},{"code_sha256_prefix":"4a51948ec08ee41a","entry":"score_outcome","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a51948ec08ee41a"}},{"code_sha256_prefix":"e41e86e03b857c80","entry":"_event_keys","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e41e86e03b857c80"}},{"code_sha256_prefix":"e380beecb195ccff","entry":"_f1_with_ndcg","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e380beecb195ccff"}},{"code_sha256_prefix":"3ada3a8ed8100f0b","entry":"_jaccard_with_position","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ada3a8ed8100f0b"}},{"code_sha256_prefix":"846673c29752ab34","entry":"_norm","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"846673c29752ab34"}},{"code_sha256_prefix":"3a9c9cff5bcea576","entry":"_stats_smape","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a9c9cff5bcea576"}},{"code_sha256_prefix":"dc1836415223bd74","entry":"grade_match","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc1836415223bd74"}},{"code_sha256_prefix":"8e7158f5b28feeac","entry":"load_tasks","repo":"wzk1015/WorldCupArena","repo_kind":"found_in_text","path":"src/graders/grade_match.py","file_url":"https://github.com/wzk1015/WorldCupArena/blob/HEAD/src/graders/grade_match.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e7158f5b28feeac"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}