{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enhancing-large-language-models-in-coding","title":"Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency","arxiv_id":"2309.17272","date":"2023-09-29","proceeding":null,"authors":["Baizhou Huang","Shuai Lu","Weizhu Chen","Xiaojun Wan","Nan Duan"],"abstract":"Large language models (LLMs) have exhibited remarkable ability in code generation. However, generating the correct solution in a single attempt still remains a challenge. Prior works utilize verification properties in software engineering to verify and re-rank solutions in a majority voting manner. But the assumption behind them that generated verification properties have better qualities than solutions may not always hold. In this paper, we treat them equally as different perspectives of LLMs' reasoning processes. We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both inter- and intra-consistency across outputs from multiple perspectives. Specifically, we prompt LLMs to generate diverse outputs from three perspectives, Solution, Specification and Test case, constructing a 3-partite graph. With two measure functions of consistency, we embed both inter- and intra-consistency information into the graph. The optimal choice of solutions is then determined based on analysis in the graph. MPSC significantly boosts performance of foundation models (ChatGPT in this paper) on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%), even surpassing GPT-4.","url_abs":"https://arxiv.org/abs/2309.17272v3","url_pdf":"https://arxiv.org/pdf/2309.17272v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"enhancing-large-language-models-in-coding","repo_url":"https://github.com/skpig/MPSC","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":null,"task_name":"mbpp"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":null,"method_name":"None"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.17272","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.17272"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/skpig/MPSC","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":14,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"662ac9159f885532","entry":"check_spec_with_test_cases","repo":"skpig/MPSC","repo_kind":"official","path":"src/_execution.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/_execution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"662ac9159f885532"}},{"code_sha256_prefix":"d51f1b0260e83cc7","entry":"check_with_specs_and_casual_inputs","repo":"skpig/MPSC","repo_kind":"official","path":"src/_execution.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/_execution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d51f1b0260e83cc7"}},{"code_sha256_prefix":"6e48435bd779492a","entry":"extend_ispassed","repo":"skpig/MPSC","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e48435bd779492a"}},{"code_sha256_prefix":"cc9fb7f50eed473a","entry":"extend_sorted","repo":"skpig/MPSC","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc9fb7f50eed473a"}},{"code_sha256_prefix":"9eb10383a49b0d9a","entry":"load_jsonl","repo":"skpig/MPSC","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9eb10383a49b0d9a"}},{"code_sha256_prefix":"570ad34bd1af44a8","entry":"load_pickle","repo":"skpig/MPSC","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"570ad34bd1af44a8"}},{"code_sha256_prefix":"54b4162aedf9341b","entry":"merge_completion_and_prompt","repo":"skpig/MPSC","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54b4162aedf9341b"}},{"code_sha256_prefix":"087bb497861f2052","entry":"pass_at_K_by_task","repo":"skpig/MPSC","repo_kind":"official","path":"src/_evaluation.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"087bb497861f2052"}},{"code_sha256_prefix":"2b1a510149e7802e","entry":"check_correctness_with_test_cases","repo":"skpig/MPSC","repo_kind":"official","path":"src/_execution.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/_execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b1a510149e7802e"}},{"code_sha256_prefix":"4eeec614744ccbf7","entry":"get_result_of_sorted_solutions","repo":"skpig/MPSC","repo_kind":"official","path":"src/_evaluation.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4eeec614744ccbf7"}},{"code_sha256_prefix":"13947e62b70c4cc0","entry":"pass_at_K","repo":"skpig/MPSC","repo_kind":"official","path":"src/_evaluation.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13947e62b70c4cc0"}},{"code_sha256_prefix":"167df5063944caeb","entry":"postconditions","repo":"skpig/MPSC","repo_kind":"official","path":"src/exemplars/3.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/exemplars/3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"167df5063944caeb"}},{"code_sha256_prefix":"c8d414f3e3e34df2","entry":"process_fn","repo":"skpig/MPSC","repo_kind":"official","path":"src/graph.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/src/graph.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8d414f3e3e34df2"}},{"code_sha256_prefix":"a65e35f50c46f055","entry":"run","repo":"skpig/MPSC","repo_kind":"official","path":"MPSC/graph.py","file_url":"https://github.com/skpig/MPSC/blob/HEAD/MPSC/graph.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a65e35f50c46f055"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}