{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/autoprompt-eliciting-knowledge-from-language","title":"AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts","arxiv_id":"2010.15980","date":"2020-10-29","proceeding":"EMNLP 2020 11","authors":["Taylor Shin","Yasaman Razeghi","Robert L. Logan IV","Eric Wallace","Sameer Singh"],"abstract":"The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.","url_abs":"https://arxiv.org/abs/2010.15980v2","url_pdf":"https://arxiv.org/pdf/2010.15980v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"autoprompt-eliciting-knowledge-from-language","repo_url":"https://github.com/ucinlp/autoprompt","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"autoprompt-eliciting-knowledge-from-language","repo_url":"https://github.com/amanb2000/magic_words","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"autoprompt-eliciting-knowledge-from-language","repo_url":"https://github.com/csinva/interpretable-autoprompting","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"autoprompt-eliciting-knowledge-from-language","repo_url":"https://github.com/jellyfish042/rwkv-statetuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"autoprompt-eliciting-knowledge-from-language","repo_url":"https://github.com/ucinlp/null-prompts","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"relation-extraction","task_name":"Relation Extraction"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[{"method_slug":"lama","method_name":"LAMA"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.15980","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.15980"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amanb2000/magic_words","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ucinlp/null-prompts","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jellyfish042/rwkv-statetuning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/csinva/interpretable-autoprompting","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ucinlp/autoprompt","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"143ddd4a9eb16ee5","entry":"extract_integer","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"143ddd4a9eb16ee5"}},{"code_sha256_prefix":"20a9293a820f617d","entry":"get_embeddings","repo":"ucinlp/autoprompt","repo_kind":"official","path":"autoprompt/create_trigger.py","file_url":"https://github.com/ucinlp/autoprompt/blob/HEAD/autoprompt/create_trigger.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20a9293a820f617d"}},{"code_sha256_prefix":"309e71f4157403d6","entry":"hotflip_attack","repo":"ucinlp/autoprompt","repo_kind":"official","path":"autoprompt/create_trigger.py","file_url":"https://github.com/ucinlp/autoprompt/blob/HEAD/autoprompt/create_trigger.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"309e71f4157403d6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}