{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/eraser-a-benchmark-to-evaluate-rationalized","title":"ERASER: A Benchmark to Evaluate Rationalized NLP Models","arxiv_id":"1911.03429","date":"2019-11-08","proceeding":"ACL 2020 6","authors":["Jay DeYoung","Sarthak Jain","Nazneen Fatema Rajani","Eric Lehman","Caiming Xiong","Richard Socher","Byron C. Wallace"],"abstract":"State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the `reasoning' behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the Evaluating Rationales And Simple English Reasoning (ERASER) benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of \"rationales\" (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/","url_abs":"https://arxiv.org/abs/1911.03429v2","url_pdf":"https://arxiv.org/pdf/1911.03429v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"eraser-a-benchmark-to-evaluate-rationalized","repo_url":"https://github.com/jayded/eraserbenchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"eraser-a-benchmark-to-evaluate-rationalized","repo_url":"https://github.com/mandeep-rathee/bagel-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1911.03429","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03429"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jayded/eraserbenchmark","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mandeep-rathee/bagel-benchmark","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2843042d2c0e6f50","entry":"annotations_from_jsonl","repo":"jayded/eraserbenchmark","repo_kind":"official","path":"rationale_benchmark/utils.py","file_url":"https://github.com/jayded/eraserbenchmark/blob/HEAD/rationale_benchmark/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2843042d2c0e6f50"}},{"code_sha256_prefix":"e1878c4a3bf022ae","entry":"load_datasets","repo":"jayded/eraserbenchmark","repo_kind":"official","path":"rationale_benchmark/utils.py","file_url":"https://github.com/jayded/eraserbenchmark/blob/HEAD/rationale_benchmark/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1878c4a3bf022ae"}},{"code_sha256_prefix":"49077456b34beefd","entry":"load_jsonl","repo":"jayded/eraserbenchmark","repo_kind":"official","path":"rationale_benchmark/utils.py","file_url":"https://github.com/jayded/eraserbenchmark/blob/HEAD/rationale_benchmark/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49077456b34beefd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}