{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fault-aware-neural-code-rankers","title":"Fault-Aware Neural Code Rankers","arxiv_id":"2206.03865","date":"2022-06-04","proceeding":null,"authors":["Jeevana Priya Inala","Chenglong Wang","Mei Yang","Andres Codas","Mark Encarnación","Shuvendu K Lahiri","Madanlal Musuvathi","Jianfeng Gao"],"abstract":"Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks. In many instances, LLMs can generate a correct program for a task when given numerous trials. Consequently, a recent trend is to do large scale sampling of programs using a model and then filtering/ranking the programs based on the program execution on a small number of known unit tests to select one candidate solution. However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as file manipulations). Both of the above assumptions are impractical in real-world software development. In this paper, we propose CodeRanker, a neural ranker that can predict the correctness of a sampled program without executing it. Our CodeRanker is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError or a TypeError). We show that CodeRanker can significantly increase the pass@1 accuracy of various code generation models (including Codex, GPT-Neo, GPT-J) on APPS, HumanEval and MBPP datasets.","url_abs":"https://arxiv.org/abs/2206.03865v2","url_pdf":"https://arxiv.org/pdf/2206.03865v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fault-aware-neural-code-rankers","repo_url":"https://github.com/microsoft/coderanker","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":null,"task_name":"mbpp"}],"methods":[{"method_slug":"gpt-neo","method_name":"GPT-Neo"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.03865","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.03865"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/coderanker","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1700760e0cf3e3f9","entry":"FaultAwareModelOutput","repo":"microsoft/coderanker","repo_kind":"official","path":"src/run_seq_classification_and_line_prediction.py","file_url":"https://github.com/microsoft/coderanker/blob/HEAD/src/run_seq_classification_and_line_prediction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1700760e0cf3e3f9"}},{"code_sha256_prefix":"ff8254638aec856b","entry":"compute_vanilla_metrics","repo":"microsoft/coderanker","repo_kind":"official","path":"src/compute_metrics.py","file_url":"https://github.com/microsoft/coderanker/blob/HEAD/src/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff8254638aec856b"}},{"code_sha256_prefix":"acfd98dfb7817a6f","entry":"get_data","repo":"microsoft/coderanker","repo_kind":"official","path":"src/compute_metrics.py","file_url":"https://github.com/microsoft/coderanker/blob/HEAD/src/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acfd98dfb7817a6f"}},{"code_sha256_prefix":"62af43e66bbbc806","entry":"pass_at_k","repo":"microsoft/coderanker","repo_kind":"official","path":"src/compute_metrics.py","file_url":"https://github.com/microsoft/coderanker/blob/HEAD/src/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62af43e66bbbc806"}},{"code_sha256_prefix":"c983d662daf116af","entry":"RobertaForFaultAwareRanker","repo":"microsoft/coderanker","repo_kind":"official","path":"src/run_seq_classification_and_line_prediction.py","file_url":"https://github.com/microsoft/coderanker/blob/HEAD/src/run_seq_classification_and_line_prediction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c983d662daf116af"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}