{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/connecting-large-language-models-with","title":"EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers","arxiv_id":"2309.08532","date":"2023-09-15","proceeding":null,"authors":["Qingyan Guo","Rui Wang","Junliang Guo","Bei Li","Kaitao Song","Xu Tan","Guoqing Liu","Jiang Bian","Yujiu Yang"],"abstract":"Large Language Models (LLMs) excel in various tasks, but they rely on carefully crafted prompts that often demand substantial human effort. To automate this process, in this paper, we propose a novel framework for discrete prompt optimization, called EvoPrompt, which borrows the idea of evolutionary algorithms (EAs) as they exhibit good performance and fast convergence. To enable EAs to work on discrete prompts, which are natural language expressions that need to be coherent and human-readable, we connect LLMs with EAs. This approach allows us to simultaneously leverage the powerful language processing capabilities of LLMs and the efficient optimization performance of EAs. Specifically, abstaining from any gradients or parameters, EvoPrompt starts from a population of prompts and iteratively generates new prompts with LLMs based on the evolutionary operators, improving the population based on the development set. We optimize prompts for both closed- and open-source LLMs including GPT-3.5 and Alpaca, on 31 datasets covering language understanding, generation tasks, as well as BIG-Bench Hard (BBH) tasks. EvoPrompt significantly outperforms human-engineered prompts and existing methods for automatic prompt generation (e.g., up to 25% on BBH). Furthermore, EvoPrompt demonstrates that connecting LLMs with EAs creates synergies, which could inspire further research on the combination of LLMs and conventional algorithms.","url_abs":"https://arxiv.org/abs/2309.08532v3","url_pdf":"https://arxiv.org/pdf/2309.08532v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"connecting-large-language-models-with","repo_url":"https://github.com/batorskq/prl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"connecting-large-language-models-with","repo_url":"https://github.com/finitearth/promptolution","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"evolutionary-algorithms","task_name":"Evolutionary Algorithms"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.08532","atlas_url":"https://app.syntology.ai/?focus=2309.08532","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.08532"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/beeevita/EvoPrompt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/batorskq/prl","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/finitearth/promptolution","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":6,"ran_honours":2,"ran_fixture":2,"ran":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b330c6d185528db2","entry":"batchify","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b330c6d185528db2"}},{"code_sha256_prefix":"f287013a24f6555f","entry":"evaluate_optimized_prompt","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f287013a24f6555f"}},{"code_sha256_prefix":"76fda09860d6a6ba","entry":"extract_numbers","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76fda09860d6a6ba"}},{"code_sha256_prefix":"d6d67571db47df2d","entry":"extract_seconds","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6d67571db47df2d"}},{"code_sha256_prefix":"c160c6c484af61e4","entry":"form_request","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c160c6c484af61e4"}},{"code_sha256_prefix":"cb966a71519c6a02","entry":"get_final_prompt","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb966a71519c6a02"}},{"code_sha256_prefix":"2d3025cc9c9fc3eb","entry":"k_init_pop","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d3025cc9c9fc3eb"}},{"code_sha256_prefix":"0ef65ffd26166327","entry":"llm_query","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ef65ffd26166327"}},{"code_sha256_prefix":"94a4deb83a99ae36","entry":"load_cls_data","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94a4deb83a99ae36"}},{"code_sha256_prefix":"5db86cda527c983b","entry":"paraphrase","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5db86cda527c983b"}},{"code_sha256_prefix":"0e0098507bc5fbe9","entry":"read_lines","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e0098507bc5fbe9"}},{"code_sha256_prefix":"57c7f6a20eb98788","entry":"Evoluter","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57c7f6a20eb98788"}},{"code_sha256_prefix":"c8d2f9365df18753","entry":"GAEvoluter","repo":"beeevita/EvoPrompt","repo_kind":"found_in_text","path":"evoluter.py","file_url":"https://github.com/beeevita/EvoPrompt/blob/HEAD/evoluter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8d2f9365df18753"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}