{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prompt-position-really-matters-in-few-shot","title":"Do prompt positions really matter?","arxiv_id":"2305.14493","date":"2023-05-23","proceeding":null,"authors":["Junyu Mao","Stuart E. Middleton","Mahesan Niranjan"],"abstract":"Prompt-based models have gathered a lot of attention from researchers due to their remarkable advancements in the fields of zero-shot and few-shot learning. Developing an effective prompt template plays a critical role. However, prior studies have mainly focused on prompt vocabulary searching or embedding initialization within a predefined template with the prompt position fixed. In this empirical study, we conduct the most comprehensive analysis to date of prompt position for diverse Natural Language Processing (NLP) tasks. Our findings quantify the substantial impact prompt position has on model performance. We observe that the prompt positions used in prior studies are often sub-optimal, and this observation is consistent even in widely used instruction-tuned models. These findings suggest prompt position optimisation as a valuable research direction to augment prompt engineering methodologies and prompt position-aware instruction tuning as a potential way to build more robust models in the future.","url_abs":"https://arxiv.org/abs/2305.14493v4","url_pdf":"https://arxiv.org/pdf/2305.14493v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prompt-position-really-matters-in-few-shot","repo_url":"https://github.com/milliemaoo/prompt-position","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":null,"task_name":"Position"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.14493","atlas_url":"https://app.syntology.ai/?focus=2305.14493","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.14493"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/milliemaoo/prompt-position","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":5,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0928f497e20fb443","entry":"mean","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/metrics.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0928f497e20fb443"}},{"code_sha256_prefix":"57e732507bad40db","entry":"escaped_split","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/utils.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57e732507bad40db"}},{"code_sha256_prefix":"d253589f2c78dad0","entry":"get_result","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/models/gpt3.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/models/gpt3.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d253589f2c78dad0"}},{"code_sha256_prefix":"359c35fdcd7a0ee5","entry":"group","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/utils.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"359c35fdcd7a0ee5"}},{"code_sha256_prefix":"ea06eaae4fc1eaf0","entry":"hash_args","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/base.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/base.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea06eaae4fc1eaf0"}},{"code_sha256_prefix":"518b2201c66c2ad3","entry":"pop_stddev","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/metrics.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"518b2201c66c2ad3"}},{"code_sha256_prefix":"d40f0123e8a8f5c0","entry":"sample_stddev","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/metrics.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d40f0123e8a8f5c0"}},{"code_sha256_prefix":"cd999ad59caf84a7","entry":"simple_parse_args_string","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/utils.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd999ad59caf84a7"}},{"code_sha256_prefix":"a5cf72836c37574e","entry":"anthropic_completion","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/models/anthropic_llms.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/models/anthropic_llms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5cf72836c37574e"}},{"code_sha256_prefix":"9d95c14130837ee4","entry":"make_table","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/evaluator.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/evaluator.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d95c14130837ee4"}},{"code_sha256_prefix":"069ffbe9aadaa2ae","entry":"stop_sequences_criteria","repo":"milliemaoo/prompt-position","repo_kind":"official","path":"lm_eval/models/huggingface.py","file_url":"https://github.com/milliemaoo/prompt-position/blob/HEAD/lm_eval/models/huggingface.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"069ffbe9aadaa2ae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}