{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-17765","title":"FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches","arxiv_id":"2607.17765","date":"2026-07-20","proceeding":null,"authors":["Jiacheng Ding","Cong Guo","Jason Xu"],"abstract":"We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events. For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models -- Claude Opus 4.8, ChatGPT (GPT-5.5, high reasoning), Gemini 3.1 Pro, and Grok (Expert Mode) -- ran an identical search-act-reflect loop: gather evidence with a web tool, commit to a 1X2 (team-A win / draw / team-B win) distribution and a virtual 100-USD bet, and, after the match, reflect given only the final score. Because every match kicked off after the models' training cutoffs, the benchmark is contamination-free by construction. Crucially, we pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market -- collected as per-match 1X2 odds, giving an economically grounded baseline and letting us score not just what an agent predicts but what it does with money. The release contains 416 forecasts and 414 reflections with verbatim reasoning, ground truth (including penalty shootouts), odds, and a reproducible evaluation suite. A reference evaluation surfaces findings that raw accuracy hides: the four agents issue an identical top pick in 92% of matches and none beats the market's Brier score; indeed, a naive flat stake on the market favorite out-earns all four agents. Yet the agents diverge sharply as decision-makers: betting return-on-investment ranges from -18% to +10%, fading the market is unprofitable for all four, the share of forecasts that cite the market ranges from 12% to 100%, and self-reported error rates on wrong picks range from 36% to 86%. The benchmark thus measures calibration, decision quality, and self-knowledge -- axes on which frontier models differ even when their predictions do not. Data and code: https://github.com/graphuofm/FIFA2026LLM","url_abs":"https://arxiv.org/abs/2607.17765","url_pdf":"https://arxiv.org/pdf/2607.17765","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.17765","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.17765"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/graphuofm/FIFA2026LLM","reach":null}],"summary":{"ran_draft_wrong":13,"ran_violates":1,"ran_honours":1,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":21,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"819d6392a05f7cfe","entry":"_clean_claude","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"819d6392a05f7cfe"}},{"code_sha256_prefix":"d8ee8a1075becc1d","entry":"_clean_json","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d8ee8a1075becc1d"}},{"code_sha256_prefix":"c3f9fbd5e30a13b5","entry":"_try","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c3f9fbd5e30a13b5"}},{"code_sha256_prefix":"51c4615320bf1b2c","entry":"block_teams","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"51c4615320bf1b2c"}},{"code_sha256_prefix":"992c5708a40d86f0","entry":"extract_score_block","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"992c5708a40d86f0"}},{"code_sha256_prefix":"e826136c915b1267","entry":"is_template","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e826136c915b1267"}},{"code_sha256_prefix":"2f7a30ff2a419992","entry":"iter_top_objects","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2f7a30ff2a419992"}},{"code_sha256_prefix":"f5d4cc3e324cc9f5","entry":"norm_team","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f5d4cc3e324cc9f5"}},{"code_sha256_prefix":"b3e049516f8d3d87","entry":"parse_compact","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b3e049516f8d3d87"}},{"code_sha256_prefix":"a0feab9524e6d42d","entry":"parse_fifa","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a0feab9524e6d42d"}},{"code_sha256_prefix":"51a911d7ca45f902","entry":"prob_fields","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"51a911d7ca45f902"}},{"code_sha256_prefix":"1e13eac6bd3590fc","entry":"refl_fields","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1e13eac6bd3590fc"}},{"code_sha256_prefix":"726bc06f1fbeb61b","entry":"reflection_ends","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"726bc06f1fbeb61b"}},{"code_sha256_prefix":"6dbe8c9dd2853563","entry":"segment","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6dbe8c9dd2853563"}},{"code_sha256_prefix":"caa9db28c5e6f604","entry":"strip_md","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"caa9db28c5e6f604"}},{"code_sha256_prefix":"85e8df521bba1fd9","entry":"_recover_object","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"85e8df521bba1fd9"}},{"code_sha256_prefix":"a627f040f13a9194","entry":"build","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a627f040f13a9194"}},{"code_sha256_prefix":"8559d5772b506370","entry":"build_schedule","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8559d5772b506370"}},{"code_sha256_prefix":"c38bbbbb214d7571","entry":"extract_responses","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c38bbbbb214d7571"}},{"code_sha256_prefix":"1007b4fc0c057ce9","entry":"parse_header","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1007b4fc0c057ce9"}},{"code_sha256_prefix":"e8469856b83f0bf2","entry":"read","repo":"graphuofm/FIFA2026LLM","repo_kind":"found_in_text","path":"src/build_dataset.py","file_url":"https://github.com/graphuofm/FIFA2026LLM/blob/HEAD/src/build_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e8469856b83f0bf2"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}