{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fantastically-ordered-prompts-and-where-to","title":"Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity","arxiv_id":"2104.08786","date":"2021-04-18","proceeding":"ACL 2022 5","authors":["Yao Lu","Max Bartolo","Alastair Moore","Sebastian Riedel","Pontus Stenetorp"],"abstract":"When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models. We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are \"fantastic\" and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good permutation for one model is not transferable to another. While one could use a development set to determine which permutations are performant, this would deviate from the true few-shot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statistics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks.","url_abs":"https://arxiv.org/abs/2104.08786v2","url_pdf":"https://arxiv.org/pdf/2104.08786v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fantastically-ordered-prompts-and-where-to","repo_url":"https://github.com/chicagohai/active-example-selection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fantastically-ordered-prompts-and-where-to","repo_url":"https://github.com/RUCAIBox/LLMBox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"sensitivity","task_name":"Sensitivity"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.08786","atlas_url":"https://app.syntology.ai/?focus=2104.08786","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08786"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RUCAIBox/LLMBox","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chicagohai/active-example-selection","reach":null}],"summary":{"ran_draft_wrong":3,"unverified":7},"by_repo_kind":{"listed":{"samples":10,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b3d1ae61efca9244","entry":"print_json","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3d1ae61efca9244"}},{"code_sha256_prefix":"aaa2f935eb008592","entry":"read_json_file","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aaa2f935eb008592"}},{"code_sha256_prefix":"2746f9dc0486a406","entry":"to_device","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2746f9dc0486a406"}},{"code_sha256_prefix":"fc7136906d9c9a7c","entry":"BaseProcessor","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc7136906d9c9a7c"}},{"code_sha256_prefix":"46f065bae1a6397f","entry":"BaseStrategy","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46f065bae1a6397f"}},{"code_sha256_prefix":"994279d788f316a5","entry":"GPT2Wrapper","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"994279d788f316a5"}},{"code_sha256_prefix":"d6240bea8a80b9a2","entry":"GenerationOutput","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6240bea8a80b9a2"}},{"code_sha256_prefix":"e8a1fe25e604a705","entry":"GlobalEntropyOrderingStrategy","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8a1fe25e604a705"}},{"code_sha256_prefix":"e75bb0724ba4a22a","entry":"probe","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e75bb0724ba4a22a"}},{"code_sha256_prefix":"35ce439826a06f86","entry":"stop","repo":"chicagohai/active-example-selection","repo_kind":"listed","path":"src/prompting/strategies/ordering_strategy.py","file_url":"https://github.com/chicagohai/active-example-selection/blob/HEAD/src/prompting/strategies/ordering_strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35ce439826a06f86"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}