{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crowdsourcing-lightweight-pyramids-for-manual","title":"Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation","arxiv_id":"1904.05929","date":"2019-04-11","proceeding":"NAACL 2019 6","authors":["Ori Shapira","David Gabay","Yang Gao","Hadar Ronen","Ramakanth Pasunuru","Mohit Bansal","Yael Amsterdamer","Ido Dagan"],"abstract":"Conducting a manual evaluation is considered an essential part of summary\nevaluation methodology. Traditionally, the Pyramid protocol, which exhaustively\ncompares system summaries to references, has been perceived as very reliable,\nproviding objective scores. Yet, due to the high cost of the Pyramid method and\nthe required expertise, researchers resorted to cheaper and less thorough\nmanual evaluation methods, such as Responsiveness and pairwise comparison,\nattainable via crowdsourcing. We revisit the Pyramid approach, proposing a\nlightweight sampling-based version that is crowdsourcable. We analyze the\nperformance of our method in comparison to original expert-based Pyramid\nevaluations, showing higher correlation relative to the common Responsiveness\nmethod. We release our crowdsourced Summary-Content-Units, along with all\ncrowdsourcing scripts, for future evaluations.","url_abs":"http://arxiv.org/abs/1904.05929v1","url_pdf":"http://arxiv.org/pdf/1904.05929v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crowdsourcing-lightweight-pyramids-for-manual","repo_url":"https://github.com/OriShapira/LitePyramids","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.05929","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}