{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2512-00371","title":"Evaluating LLMs in Open-Source Games","arxiv_id":"2512.00371","date":"2025-11-29","proceeding":"NeurIPS","authors":["Swadesh Sistla","Max Kleiman-Weiner"],"abstract":"Large Language Models' (LLMs) programming capabilities enable their participation in open-source games: a game-theoretic setting in which players submit computer programs in lieu of actions. These programs offer numerous advantages, including interpretability, inter-agent transparency, and formal verifiability; additionally, they enable program equilibria, solutions that leverage the transparency of code and are inaccessible within normal-form settings. We evaluate the capabilities of leading open- and closed-weight LLMs to predict and classify program strategies and evaluate features of the approximate program equilibria reached by LLM agents in dyadic and evolutionary settings. We identify the emergence of payoff-maximizing, cooperative, and deceptive strategies, characterize the adaptation of mechanisms within these programs over repeated open-source games, and analyze their comparative evolutionary fitness. We find that open-source games serve as a viable environment to study and steer the emergence of cooperative strategy in multi-agent dilemmas.","url_abs":"https://arxiv.org/abs/2512.00371","url_pdf":"https://arxiv.org/pdf/2512.00371","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2512.00371","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2512.00371"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/swadeshs/llm-osgt","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"found_in_text":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d867ea06c2c6b481","entry":"clean_syntax_errors","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"program_attribute.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/program_attribute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d867ea06c2c6b481"}},{"code_sha256_prefix":"b86289c0894be593","entry":"count_loc","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"ipd_visualization.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/ipd_visualization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b86289c0894be593"}},{"code_sha256_prefix":"e2426bb42d44b3e8","entry":"find_experiment_runs","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"analysis_suite.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/analysis_suite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2426bb42d44b3e8"}},{"code_sha256_prefix":"8fa887248bce7b1b","entry":"get_strategy_stats_from_files","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"ipd_visualization.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/ipd_visualization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fa887248bce7b1b"}},{"code_sha256_prefix":"94d4d7431829fe3b","entry":"load_file_content","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"analysis_suite.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/analysis_suite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94d4d7431829fe3b"}},{"code_sha256_prefix":"e6828a59838d7108","entry":"mask_strategy_names","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"program_attribute.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/program_attribute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6828a59838d7108"}},{"code_sha256_prefix":"f3980068c1136ea7","entry":"parse_classifier_info","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"ipd_visualization.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/ipd_visualization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3980068c1136ea7"}},{"code_sha256_prefix":"4efc2840362d248e","entry":"strip_comments_and_docstrings","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"program_attribute.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/program_attribute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4efc2840362d248e"}},{"code_sha256_prefix":"f99aeea4a356e803","entry":"verify_program_obfuscation","repo":"swadeshs/llm-osgt","repo_kind":"found_in_text","path":"ipd_experiment.py","file_url":"https://github.com/swadeshs/llm-osgt/blob/HEAD/ipd_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f99aeea4a356e803"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.GT","source":"arxiv_api"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}