{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-09907","title":"Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem","arxiv_id":"2510.09907","date":"2025-10-10","proceeding":"NeurIPS","authors":["Muhammad Maaz","Liam DeVoe","Zac Hatfield-Dodds","Nicholas Carlini"],"abstract":"Property-based testing (PBT) is a lightweight formal method, typically implemented as a randomized testing framework. Users specify the input domain for their test using combinators supplied by the PBT framework, and the expected properties or invariants as a unit-test function. The framework then searches for a counterexample, e.g. by generating inputs and calling the test function. In this work, we demonstrate an LLM-based agent which analyzes Python modules, infers function-specific and cross-function properties from code and documentation, synthesizes and executes PBTs, reflects on outputs of these tests to confirm true bugs, and finally outputs actionable bug reports for the developer. We perform an extensive evaluation of our agent across 100 popular Python packages. Of the bug reports generated by the agent, we found after manual review that 56\\% were valid bugs and 32\\% were valid bugs that we would report to maintainers. We then developed a ranking rubric to surface high-priority valid bugs to developers, and found that of the 21 top-scoring bugs, 86\\% were valid and 81\\% we would report. The bugs span diverse failure modes from serialization failures to numerical precision errors to flawed cache implementations. We reported 5 bugs, 4 with patches, including to NumPy and cloud computing SDKs, with 3 patches merged successfully. Our results suggest that LLMs with PBT provides a rigorous and scalable method for autonomously testing software. Our code and artifacts are available at: https://github.com/mmaaz-git/agentic-pbt.","url_abs":"https://arxiv.org/abs/2510.09907","url_pdf":"https://arxiv.org/pdf/2510.09907","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.09907","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.09907"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/mmaaz-git/agentic-pbt","reach":{"status":"ok"}}],"summary":{"unverified":14},"by_repo_kind":{"found_in_text":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"a0d5adad0abd50aa","entry":"bfs_discover_from_seeds","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"paper/evaluation/select_packages.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/paper/evaluation/select_packages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0d5adad0abd50aa"}},{"code_sha256_prefix":"9a309fc2cb2906c9","entry":"call_agent","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"utils/run_check_issues.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/utils/run_check_issues.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a309fc2cb2906c9"}},{"code_sha256_prefix":"f15259880ae6ce7e","entry":"count_files","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"paper/evaluation/clean_data.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/paper/evaluation/clean_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f15259880ae6ce7e"}},{"code_sha256_prefix":"1642249794c2d532","entry":"create_deduplication_prompt","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"utils/dedupe_bug_reports.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/utils/dedupe_bug_reports.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1642249794c2d532"}},{"code_sha256_prefix":"1ee2f903fcfa50fc","entry":"dedupe_by_github","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"example_packages/generate_10k_packages.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/example_packages/generate_10k_packages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ee2f903fcfa50fc"}},{"code_sha256_prefix":"4c0939f8ac665b3f","entry":"find_import_name","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"paper/evaluation/select_packages.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/paper/evaluation/select_packages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c0939f8ac665b3f"}},{"code_sha256_prefix":"122958b0dc4c8186","entry":"infer_pkg_from_report","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"utils/run_check_issues.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/utils/run_check_issues.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"122958b0dc4c8186"}},{"code_sha256_prefix":"c8b677bea160a0b1","entry":"load_dataset","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"paper/evaluation/pypi_packages.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/paper/evaluation/pypi_packages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8b677bea160a0b1"}},{"code_sha256_prefix":"810918bb36f11895","entry":"load_top_packages","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"example_packages/generate_10k_packages.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/example_packages/generate_10k_packages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"810918bb36f11895"}},{"code_sha256_prefix":"48f5c550b2294dba","entry":"read_bug_reports","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"utils/dedupe_bug_reports.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/utils/dedupe_bug_reports.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48f5c550b2294dba"}},{"code_sha256_prefix":"bc684c2ea8fa8a79","entry":"run","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"example_packages/generate_10k_packages.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/example_packages/generate_10k_packages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc684c2ea8fa8a79"}},{"code_sha256_prefix":"f8e06bca59169ce1","entry":"score_bug_report","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"scoring.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/scoring.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8e06bca59169ce1"}},{"code_sha256_prefix":"8ffd6688a04eadf7","entry":"should_skip_module","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"paper/evaluation/select_packages.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/paper/evaluation/select_packages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ffd6688a04eadf7"}},{"code_sha256_prefix":"c3bfc03647cd4f25","entry":"verify_no_orphans","repo":"mmaaz-git/agentic-pbt","repo_kind":"found_in_text","path":"paper/evaluation/clean_data.py","file_url":"https://github.com/mmaaz-git/agentic-pbt/blob/HEAD/paper/evaluation/clean_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3bfc03647cd4f25"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.SE","source":"arxiv_api"},"syntology_extracted_results":null}