{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spor-a-comprehensive-and-practical-evaluation","title":"SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation","arxiv_id":"2405.10650","date":"2024-05-17","proceeding":null,"authors":["Ziyao Xu","Houfeng Wang"],"abstract":"Compositional generalization is an important ability of language models and has many different manifestations. For data-to-text generation, previous research on this ability is limited to a single manifestation called Systematicity and lacks consideration of large language models (LLMs), which cannot fully cover practical application scenarios. In this work, we propose SPOR, a comprehensive and practical evaluation method for compositional generalization in data-to-text generation. SPOR includes four aspects of manifestations (Systematicity, Productivity, Order invariance, and Rule learnability) and allows high-quality evaluation without additional manual annotations based on existing datasets. We demonstrate SPOR on two different datasets and evaluate some existing language models including LLMs. We find that the models are deficient in various aspects of the evaluation and need further improvement. Our work shows the necessity for comprehensive research on different manifestations of compositional generalization in data-to-text generation and provides a framework for evaluation.","url_abs":"https://arxiv.org/abs/2405.10650v8","url_pdf":"https://arxiv.org/pdf/2405.10650v8.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spor-a-comprehensive-and-practical-evaluation","repo_url":"https://github.com/xzy-xzy/spor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"data-to-text-generation","task_name":"Data-to-Text Generation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.10650","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.10650"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/xzy-xzy/SPOR","reach":null}],"summary":{"ran":1,"ran_honours":1,"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ddb0c8dc40d57260","entry":"Entry","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddb0c8dc40d57260"}},{"code_sha256_prefix":"35ccb7763668f558","entry":"coefficient","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35ccb7763668f558"}},{"code_sha256_prefix":"e383b4596e28bf60","entry":"delete_comp","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e383b4596e28bf60"}},{"code_sha256_prefix":"4724d167ece786a6","entry":"select_files","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4724d167ece786a6"}},{"code_sha256_prefix":"2b3ae0adb7304cc9","entry":"select_tfile","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b3ae0adb7304cc9"}},{"code_sha256_prefix":"fb379c167758d8aa","entry":"Benchmark","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb379c167758d8aa"}},{"code_sha256_prefix":"971cfe0a65a8d4fe","entry":"deal_sys","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"971cfe0a65a8d4fe"}},{"code_sha256_prefix":"211c4cf9603bb7db","entry":"get_benchmark_e2e","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"211c4cf9603bb7db"}},{"code_sha256_prefix":"6af49ab40e5e3341","entry":"get_entries","repo":"xzy-xzy/SPOR","repo_kind":"official","path":"code/construction/systematicity.py","file_url":"https://github.com/xzy-xzy/SPOR/blob/HEAD/code/construction/systematicity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6af49ab40e5e3341"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}