{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tigerscore-towards-building-explainable","title":"TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks","arxiv_id":"2310.00752","date":"2023-10-01","proceeding":null,"authors":["Dongfu Jiang","Yishan Li","Ge Zhang","Wenhao Huang","Bill Yuchen Lin","Wenhu Chen"],"abstract":"We present TIGERScore, a \\textbf{T}rained metric that follows \\textbf{I}nstruction \\textbf{G}uidance to perform \\textbf{E}xplainable, and \\textbf{R}eference-free evaluation over a wide spectrum of text generation tasks. Different from other automatic evaluation methods that only provide arcane scores, TIGERScore is guided by natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. The dataset consists of 42K quadruple in the form of (instruction, input, system output $\\rightarrow$ error analysis). We collected the `system outputs' through from a large variety of models to cover different types of errors. To quantitatively assess our metric, we evaluate its correlation with human ratings on 5 held-in datasets, 2 held-out datasets and show that TIGERScore can achieve the open-source SoTA correlation with human ratings across these datasets and almost approaches GPT-4 evaluator. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human evaluation on the generated explanations and found that the explanations are 70.8\\% accurate. Through these experimental results, we believe TIGERScore demonstrates the possibility of building universal explainable metrics to evaluate any text generation task. All the resourced are released in our project website: \\url{https://tiger-ai-lab.github.io/TIGERScore/}.","url_abs":"https://arxiv.org/abs/2310.00752v4","url_pdf":"https://arxiv.org/pdf/2310.00752v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tigerscore-towards-building-explainable","repo_url":"https://github.com/TIGER-AI-Lab/TIGERScore","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"metricinstruct","name":"MetricInstruct","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.00752","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.00752"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TIGER-AI-Lab/TIGERScore","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fb5de83f1a1344d3","entry":"beam_search_step","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/engine.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb5de83f1a1344d3"}},{"code_sha256_prefix":"9dc2e0bf6b1d6789","entry":"cor_pearson","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/common/cor_eval.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/common/cor_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9dc2e0bf6b1d6789"}},{"code_sha256_prefix":"b4f6905960fb1442","entry":"cor_spearman","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/common/cor_eval.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/common/cor_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4f6905960fb1442"}},{"code_sha256_prefix":"27a193806920b38c","entry":"cor_spearman_footrule","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/common/cor_eval.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/common/cor_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27a193806920b38c"}},{"code_sha256_prefix":"eb2c3c1c949f4587","entry":"get_model_size","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/generate_candidates.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/generate_candidates.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb2c3c1c949f4587"}},{"code_sha256_prefix":"81990a380398703e","entry":"get_torch_dtype","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/generate_candidates.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/generate_candidates.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81990a380398703e"}},{"code_sha256_prefix":"ee4bf60b2ad7b772","entry":"load_dataset","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/finetune_base_model.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/finetune_base_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee4bf60b2ad7b772"}},{"code_sha256_prefix":"d2757c38c4c0b3fa","entry":"preprocess_function","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/finetune_base_model.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/finetune_base_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2757c38c4c0b3fa"}},{"code_sha256_prefix":"061cce7187d47649","entry":"build_model","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/model_utils.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"061cce7187d47649"}},{"code_sha256_prefix":"e640bf37ad767102","entry":"build_processor","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/model_utils.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e640bf37ad767102"}},{"code_sha256_prefix":"200483ce1173890a","entry":"build_tokenizer","repo":"TIGER-AI-Lab/TIGERScore","repo_kind":"official","path":"tigerscore/candidates_generation/model_utils.py","file_url":"https://github.com/TIGER-AI-Lab/TIGERScore/blob/HEAD/tigerscore/candidates_generation/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"200483ce1173890a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}