{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-generation-with-strictly-proper","title":"Language Generation with Strictly Proper Scoring Rules","arxiv_id":"2405.18906","date":"2024-05-29","proceeding":null,"authors":["Chenze Shao","Fandong Meng","Yijin Liu","Jie zhou"],"abstract":"Language generation based on maximum likelihood estimation (MLE) has become the fundamental approach for text generation. Maximum likelihood estimation is typically performed by minimizing the log-likelihood loss, also known as the logarithmic score in statistical decision theory. The logarithmic score is strictly proper in the sense that it encourages honest forecasts, where the expected score is maximized only when the model reports true probabilities. Although many strictly proper scoring rules exist, the logarithmic score is the only local scoring rule among them that depends exclusively on the probability of the observed sample, making it capable of handling the exponentially large sample space of natural text. In this work, we propose a straightforward strategy for adapting scoring rules to language generation, allowing for language modeling with any non-local scoring rules. Leveraging this strategy, we train language generation models using two classic strictly proper scoring rules, the Brier score and the Spherical score, as alternatives to the logarithmic score. Experimental results indicate that simply substituting the loss function, without adjusting other hyperparameters, can yield substantial improvements in model's generation capabilities. Moreover, these improvements can scale up to large language models (LLMs) such as LLaMA-7B and LLaMA-13B. Source code: \\url{https://github.com/shaochenze/ScoringRulesLM}.","url_abs":"https://arxiv.org/abs/2405.18906v1","url_pdf":"https://arxiv.org/pdf/2405.18906v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-generation-with-strictly-proper","repo_url":"https://github.com/shaochenze/scoringruleslm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"scoring-rule","task_name":"scoring rule"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.18906","atlas_url":"https://app.syntology.ai/?focus=2405.18906","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.18906"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shaochenze/scoringruleslm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"3f6541fe9b57b5c9","entry":"dataset_dest_file","repo":"shaochenze/scoringruleslm","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/shaochenze/scoringruleslm/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f6541fe9b57b5c9"}},{"code_sha256_prefix":"ee4a597ec93e7f03","entry":"dataset_dest_prefix","repo":"shaochenze/scoringruleslm","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/shaochenze/scoringruleslm/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee4a597ec93e7f03"}},{"code_sha256_prefix":"d4192f5e85770fdd","entry":"find_version","repo":"shaochenze/scoringruleslm","repo_kind":"official","path":"release_utils.py","file_url":"https://github.com/shaochenze/scoringruleslm/blob/HEAD/release_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4192f5e85770fdd"}},{"code_sha256_prefix":"88ce019380f10478","entry":"get_symbols_to_strip_from_output","repo":"shaochenze/scoringruleslm","repo_kind":"official","path":"generate.py","file_url":"https://github.com/shaochenze/scoringruleslm/blob/HEAD/generate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88ce019380f10478"}},{"code_sha256_prefix":"26da4f92ba56e785","entry":"label_smoothed_nll_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"26da4f92ba56e785"}},{"code_sha256_prefix":"44b6e8848744468f","entry":"get_next_version","repo":"shaochenze/scoringruleslm","repo_kind":"official","path":"release_utils.py","file_url":"https://github.com/shaochenze/scoringruleslm/blob/HEAD/release_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44b6e8848744468f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}