{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unsupervised-evaluation-of-code-llms-with","title":"Unsupervised Evaluation of Code LLMs with Round-Trip Correctness","arxiv_id":"2402.08699","date":"2024-02-13","proceeding":null,"authors":["Miltiadis Allamanis","Sheena Panthaplackel","Pengcheng Yin"],"abstract":"To evaluate code large language models (LLMs), research has relied on a few small manually curated benchmarks, such as HumanEval and MBPP, which represent a narrow part of the real-world software domains. In this work, we introduce round-trip correctness (RTC) as an alternative evaluation method. RTC allows Code LLM evaluation on a broader spectrum of real-world software domains without the need for costly human curation. RTC rests on the idea that we can ask a model to make a prediction (e.g., describe some code using natural language), feed that prediction back (e.g., synthesize code from the predicted description), and check if this round-trip leads to code that is semantically equivalent to the original input. We show how to employ RTC to evaluate code synthesis and editing. We find that RTC strongly correlates with model performance on existing narrow-domain code synthesis benchmarks while allowing us to expand to a much broader set of domains and tasks which was not previously possible without costly human annotations.","url_abs":"https://arxiv.org/abs/2402.08699v2","url_pdf":"https://arxiv.org/pdf/2402.08699v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unsupervised-evaluation-of-code-llms-with","repo_url":"https://github.com/google-deepmind/icml2024-roundtrip-correctness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"unsupervised-evaluation-of-code-llms-with","repo_url":"https://github.com/codelion/optillm/blob/main/optillm/rto.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":null,"task_name":"mbpp"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.08699","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.08699"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/icml2024-roundtrip-correctness","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codelion/optillm/blob/main/optillm/rto.py","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"listed":{"samples":4,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f36d35da5765481","entry":"extract_code_from_prompt","repo":"codelion/optillm","repo_kind":"listed","path":"optillm/rto.py","file_url":"https://github.com/codelion/optillm/blob/HEAD/optillm/rto.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f36d35da5765481"}},{"code_sha256_prefix":"b37b4c70bebbb0f3","entry":"round_trip_optimization","repo":"codelion/optillm","repo_kind":"listed","path":"optillm/rto.py","file_url":"https://github.com/codelion/optillm/blob/HEAD/optillm/rto.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b37b4c70bebbb0f3"}},{"code_sha256_prefix":"f916ca93760b9f83","entry":"get_span_with_context","repo":"google-deepmind/icml2024-roundtrip-correctness","repo_kind":"listed","path":"text_utils.py","file_url":"https://github.com/google-deepmind/icml2024-roundtrip-correctness/blob/HEAD/text_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f916ca93760b9f83"}},{"code_sha256_prefix":"ad751d2ae89bc365","entry":"truncate_to_stop_token","repo":"google-deepmind/icml2024-roundtrip-correctness","repo_kind":"listed","path":"llm_interface.py","file_url":"https://github.com/google-deepmind/icml2024-roundtrip-correctness/blob/HEAD/llm_interface.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad751d2ae89bc365"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}