{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/design-and-analysis-of-the-nips-2016-review","title":"Design and Analysis of the NIPS 2016 Review Process","arxiv_id":"1708.09794","date":"2017-08-31","proceeding":null,"authors":["Nihar B. Shah","Behzad Tabibian","Krikamol Muandet","Isabelle Guyon","Ulrike Von Luxburg"],"abstract":"Neural Information Processing Systems (NIPS) is a top-tier annual conference\nin machine learning. The 2016 edition of the conference comprised more than\n2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents\na growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and\nover 100% in terms of attendees as compared to the previous year. The massive\nscale as well as rapid growth of the conference calls for a thorough quality\nassessment of the peer-review process and novel means of improvement. In this\npaper, we analyze several aspects of the data collected during the review\nprocess, including an experiment investigating the efficacy of collecting\nordinal rankings from reviewers. Our goal is to check the soundness of the\nreview process, and provide insights that may be useful in the design of the\nreview process of subsequent conferences.","url_abs":"http://arxiv.org/abs/1708.09794v2","url_pdf":"http://arxiv.org/pdf/1708.09794v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"design-and-analysis-of-the-nips-2016-review","repo_url":"https://github.com/btabibian/conference-analysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1708.09794","atlas_url":"https://app.syntology.ai/?focus=1708.09794","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}