{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/roscoe-a-suite-of-metrics-for-scoring-step-by","title":"ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning","arxiv_id":"2212.07919","date":"2022-12-15","proceeding":null,"authors":["Olga Golovneva","Moya Chen","Spencer Poff","Martin Corredor","Luke Zettlemoyer","Maryam Fazel-Zarandi","Asli Celikyilmaz"],"abstract":"Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.","url_abs":"https://arxiv.org/abs/2212.07919v2","url_pdf":"https://arxiv.org/pdf/2212.07919v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"roscoe-a-suite-of-metrics-for-scoring-step-by","repo_url":"https://github.com/facebookresearch/ParlAI","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"informativeness","task_name":"Informativeness"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[{"slug":"roscoe","name":"ROSCOE","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.07919","atlas_url":"https://app.syntology.ai/?focus=2212.07919","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.07919"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/ParlAI","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"761ab53c3cc0c21a","entry":"find_ngrams","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/dict.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/dict.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"761ab53c3cc0c21a"}},{"code_sha256_prefix":"b6eb8b2b6abaf6ba","entry":"get_metric_display_data","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/metrics.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b6eb8b2b6abaf6ba"}},{"code_sha256_prefix":"60339a7726e86b18","entry":"register_agent","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/loader.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/loader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"60339a7726e86b18"}},{"code_sha256_prefix":"f9ccfd237025f9b0","entry":"register_mutator","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/mutators.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/mutators.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"f9ccfd237025f9b0"}},{"code_sha256_prefix":"eba5f047c4021087","entry":"register_script","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/loader.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/loader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"eba5f047c4021087"}},{"code_sha256_prefix":"b965ad8e264b80f1","entry":"register_teacher","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/loader.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/loader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b965ad8e264b80f1"}},{"code_sha256_prefix":"7e5138ac467c5625","entry":"unescape","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/dict.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/dict.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"7e5138ac467c5625"}},{"code_sha256_prefix":"758e9948df81569f","entry":"escape","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/dict.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/dict.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"758e9948df81569f"}},{"code_sha256_prefix":"da3bc8c42eec18d1","entry":"normalize_answer","repo":"facebookresearch/ParlAI","repo_kind":"official","path":"parlai/core/metrics.py","file_url":"https://github.com/facebookresearch/ParlAI/blob/HEAD/parlai/core/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"da3bc8c42eec18d1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}