{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/annotating-derivations-a-new-evaluation","title":"Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems","arxiv_id":"1609.07197","date":"2016-09-23","proceeding":"EACL 2017 4","authors":["Shyam Upadhyay","Ming-Wei Chang"],"abstract":"We propose a new evaluation for automatic solvers for algebra word problems,\nwhich can identify mistakes that existing evaluations overlook. Our proposal is\nto evaluate such solvers using derivations, which reflect how an equation\nsystem was constructed from the word problem. To accomplish this, we develop an\nalgorithm for checking the equivalence between two derivations, and show how\nderivation an- notations can be semi-automatically added to existing datasets.\nTo make our experiments more comprehensive, we include the derivation\nannotation for DRAW-1K, a new dataset containing 1000 general algebra word\nproblems. In our experiments, we found that the annotated derivations enable a\nmore accurate evaluation of automatic solvers than previously used metrics. We\nrelease derivation annotations for over 2300 algebra word problems for future\nevaluations.","url_abs":"http://arxiv.org/abs/1609.07197v2","url_pdf":"http://arxiv.org/pdf/1609.07197v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"annotating-derivations-a-new-evaluation","repo_url":"https://github.com/lab-v2/chatgpt_mwp_eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"CC0-1.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1609.07197","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}