{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/compression-transduction-and-creation-a","title":"Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation","arxiv_id":"2109.06379","date":"2021-09-14","proceeding":"EMNLP 2021 11","authors":["Mingkai Deng","Bowen Tan","Zhengzhong Liu","Eric P. Xing","Zhiting Hu"],"abstract":"Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives and desires different properties of generated text. The complexity makes automatic evaluation of NLG particularly challenging. Previous work has typically focused on a single task and developed individual evaluation metrics based on specific intuitions. In this paper, we propose a unifying perspective that facilitates the design of metrics for a wide range of language generation tasks and quality aspects. Based on the nature of information change from input to output, we classify NLG tasks into compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog). The information alignment, or overlap, between input, context, and output text plays a common central role in characterizing the generation. Using the uniform concept of information alignment, we develop a family of interpretable metrics for various NLG tasks and aspects, often without need of gold reference data. To operationalize the metrics, we train self-supervised models to approximate information alignment as a prediction task. Experiments show the uniformly designed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in each of diverse tasks, including text summarization, style transfer, and knowledge-grounded dialog. With information alignment as the intermediate representation, we deliver a composable library for easy NLG evaluation and future metric design.","url_abs":"https://arxiv.org/abs/2109.06379v2","url_pdf":"https://arxiv.org/pdf/2109.06379v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"compression-transduction-and-creation-a","repo_url":"https://github.com/tanyuqian/ctc-gen-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"style-transfer","task_name":"Style Transfer"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"nlg-evaluation","task_name":"nlg evaluation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.06379","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.06379"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tanyuqian/ctc-gen-eval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e9fc6d93b3c4e519","entry":"get_dataloaders","repo":"tanyuqian/ctc-gen-eval","repo_kind":"official","path":"ctc_score/data_utils/data_utils.py","file_url":"https://github.com/tanyuqian/ctc-gen-eval/blob/HEAD/ctc_score/data_utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9fc6d93b3c4e519"}},{"code_sha256_prefix":"8c2fa4c8bb1a1056","entry":"get_words","repo":"tanyuqian/ctc-gen-eval","repo_kind":"official","path":"ctc_score/data_utils/data_utils.py","file_url":"https://github.com/tanyuqian/ctc-gen-eval/blob/HEAD/ctc_score/data_utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c2fa4c8bb1a1056"}},{"code_sha256_prefix":"c38668427bf4c4ae","entry":"make_vocab","repo":"tanyuqian/ctc-gen-eval","repo_kind":"official","path":"ctc_score/download.py","file_url":"https://github.com/tanyuqian/ctc-gen-eval/blob/HEAD/ctc_score/download.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c38668427bf4c4ae"}},{"code_sha256_prefix":"d84c734630bc36d8","entry":"maybe_download","repo":"tanyuqian/ctc-gen-eval","repo_kind":"official","path":"ctc_score/download.py","file_url":"https://github.com/tanyuqian/ctc-gen-eval/blob/HEAD/ctc_score/download.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d84c734630bc36d8"}},{"code_sha256_prefix":"88b82c33bd19c0e0","entry":"read_words","repo":"tanyuqian/ctc-gen-eval","repo_kind":"official","path":"ctc_score/download.py","file_url":"https://github.com/tanyuqian/ctc-gen-eval/blob/HEAD/ctc_score/download.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88b82c33bd19c0e0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}