{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prefer-prompt-ensemble-learning-via-feedback","title":"PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine","arxiv_id":"2308.12033","date":"2023-08-23","proceeding":null,"authors":["Chenrui Zhang","Lin Liu","Jinpeng Wang","Chuyuan Wang","Xiao Sun","Hongyu Wang","Mingchen Cai"],"abstract":"As an effective tool for eliciting the power of Large Language Models (LLMs), prompting has recently demonstrated unprecedented abilities across a variety of complex tasks. To further improve the performance, prompt ensemble has attracted substantial interest for tackling the hallucination and instability of LLMs. However, existing methods usually adopt a two-stage paradigm, which requires a pre-prepared set of prompts with substantial manual effort, and is unable to perform directed optimization for different weak learners. In this paper, we propose a simple, universal, and automatic method named PREFER (Pompt Ensemble learning via Feedback-Reflect-Refine) to address the stated limitations. Specifically, given the fact that weak learners are supposed to focus on hard examples during boosting, PREFER builds a feedback mechanism for reflecting on the inadequacies of existing weak learners. Based on this, the LLM is required to automatically synthesize new prompts for iterative refinement. Moreover, to enhance stability of the prompt effect evaluation, we propose a novel prompt bagging method involving forward and backward thinking, which is superior to majority voting and is beneficial for both feedback and weight calculation in boosting. Extensive experiments demonstrate that our PREFER achieves state-of-the-art performance in multiple types of tasks by a significant margin. We have made our code publicly available.","url_abs":"https://arxiv.org/abs/2308.12033v1","url_pdf":"https://arxiv.org/pdf/2308.12033v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prefer-prompt-ensemble-learning-via-feedback","repo_url":"https://github.com/zcrwind/prefer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"ensemble-learning","task_name":"Ensemble Learning"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.12033","atlas_url":"https://app.syntology.ai/?focus=2308.12033","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.12033"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zcrwind/prefer","reach":null}],"summary":{"ran":1,"ran_draft_wrong":3,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"377ef4c66c11a227","entry":"ResPromptBooster","repo":"zcrwind/prefer","repo_kind":"official","path":"prefer.py","file_url":"https://github.com/zcrwind/prefer/blob/HEAD/prefer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"377ef4c66c11a227"}},{"code_sha256_prefix":"5592e63376c0d5fa","entry":"get_label","repo":"zcrwind/prefer","repo_kind":"official","path":"scripts/generate_k_shot_data.py","file_url":"https://github.com/zcrwind/prefer/blob/HEAD/scripts/generate_k_shot_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5592e63376c0d5fa"}},{"code_sha256_prefix":"01f18670482745e3","entry":"get_num_classes","repo":"zcrwind/prefer","repo_kind":"official","path":"prefer.py","file_url":"https://github.com/zcrwind/prefer/blob/HEAD/prefer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01f18670482745e3"}},{"code_sha256_prefix":"635e168c990149fd","entry":"load_datasets","repo":"zcrwind/prefer","repo_kind":"official","path":"scripts/generate_k_shot_data.py","file_url":"https://github.com/zcrwind/prefer/blob/HEAD/scripts/generate_k_shot_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"635e168c990149fd"}},{"code_sha256_prefix":"efd1b7718b950470","entry":"split_header","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"efd1b7718b950470"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}