{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crowdsourcing-multiple-choice-science","title":"Crowdsourcing Multiple Choice Science Questions","arxiv_id":"1707.06209","date":"2017-07-19","proceeding":"WS 2017 9","authors":["Johannes Welbl","Nelson F. Liu","Matt Gardner"],"abstract":"We present a novel method for obtaining high-quality, domain-targeted\nmultiple choice questions from crowd workers. Generating these questions can be\ndifficult without trading away originality, relevance or diversity in the\nanswer options. Our method addresses these problems by leveraging a large\ncorpus of domain-specific text and a small set of existing questions. It\nproduces model suggestions for document selection and answer distractor choice\nwhich aid the human question generation process. With this method we have\nassembled SciQ, a dataset of 13.7K multiple choice science exam questions\n(Dataset available at http://allenai.org/data.html). We demonstrate that the\nmethod produces in-domain questions by providing an analysis of this new\ndataset and by showing that humans cannot distinguish the crowdsourced\nquestions from original questions. When using SciQ as additional training data\nto existing questions, we observe accuracy improvements on real science exams.","url_abs":"http://arxiv.org/abs/1707.06209v1","url_pdf":"http://arxiv.org/pdf/1707.06209v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-generation","task_name":"Question Generation"},{"task_slug":"question-generation","task_name":"Question-Generation"}],"methods":[],"datasets_introduced":[{"slug":"sciq","name":"SciQ","full_name":"SciQ"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1707.06209","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}