{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-dataset-of-peer-reviews-peerread-collection","title":"A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications","arxiv_id":"1804.09635","date":"2018-04-25","proceeding":"NAACL 2018 6","authors":["Dongyeop Kang","Waleed Ammar","Bhavana Dalvi","Madeleine van Zuylen","Sebastian Kohlmeier","Eduard Hovy","Roy Schwartz"],"abstract":"Peer reviewing is a central component in the scientific publishing process.\nWe present the first public dataset of scientific peer reviews available for\nresearch purposes (PeerRead v1) providing an opportunity to study this\nimportant artifact. The dataset consists of 14.7K paper drafts and the\ncorresponding accept/reject decisions in top-tier venues including ACL, NIPS\nand ICLR. The dataset also includes 10.7K textual peer reviews written by\nexperts for a subset of the papers. We describe the data collection process and\nreport interesting observed phenomena in the peer reviews. We also propose two\nnovel NLP tasks based on this dataset and provide simple baseline models. In\nthe first task, we show that simple models can predict whether a paper is\naccepted with up to 21% error reduction compared to the majority baseline. In\nthe second task, we predict the numerical scores of review aspects and show\nthat simple models can outperform the mean baseline for aspects with high\nvariance such as 'originality' and 'impact'.","url_abs":"http://arxiv.org/abs/1804.09635v1","url_pdf":"http://arxiv.org/pdf/1804.09635v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-dataset-of-peer-reviews-peerread-collection","repo_url":"https://github.com/allenai/PeerRead","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[{"slug":"peerread","name":"PeerRead","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.09635","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}