{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cs4-measuring-the-creativity-of-large","title":"CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints","arxiv_id":"2410.04197","date":"2024-10-05","proceeding":null,"authors":["Anirudh Atmakuru","Jatin Nainani","Rohith Siddhartha Reddy Bheemreddy","Anirudh Lakkaraju","Zonghai Yao","Hamed Zamani","Haw-Shiuan Chang"],"abstract":"Evaluating the creativity of large language models (LLMs) in story writing is difficult because LLM-generated stories could seemingly look creative but be very similar to some existing stories in their huge and proprietary training corpus. To overcome this challenge, we introduce a novel benchmark dataset with varying levels of prompt specificity: CS4 ($\\mathbf{C}$omparing the $\\mathbf{S}$kill of $\\mathbf{C}$reating $\\mathbf{S}$tories by $\\mathbf{C}$ontrolling the $\\mathbf{S}$ynthesized $\\mathbf{C}$onstraint $\\mathbf{S}$pecificity). By increasing the number of requirements/constraints in the prompt, we can increase the prompt specificity and hinder LLMs from retelling high-quality narratives in their training data. Consequently, CS4 empowers us to indirectly measure the LLMs' creativity without human annotations. Our experiments on LLaMA, Gemma, and Mistral not only highlight the creativity challenges LLMs face when dealing with highly specific prompts but also reveal that different LLMs perform very differently under different numbers of constraints and achieve different balances between the model's instruction-following ability and narrative coherence. Additionally, our experiments on OLMo suggest that Learning from Human Feedback (LHF) can help LLMs select better stories from their training data but has limited influence in boosting LLMs' ability to produce creative stories that are unseen in the training corpora. The benchmark is released at https://github.com/anirudhlakkaraju/cs4_benchmark.","url_abs":"https://arxiv.org/abs/2410.04197v1","url_pdf":"https://arxiv.org/pdf/2410.04197v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cs4-measuring-the-creativity-of-large","repo_url":"https://github.com/anirudhlakkaraju/cs4_benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"specificity","task_name":"Specificity"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.04197","atlas_url":"https://app.syntology.ai/?focus=2410.04197","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.04197"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/anirudhlakkaraju/cs4_benchmark","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fbca3fb05f5d729f","entry":"Quc_9VsRcs_7_39","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/coherence_vs_constraint_graph.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/coherence_vs_constraint_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbca3fb05f5d729f"}},{"code_sha256_prefix":"41dfba5ffb866e95","entry":"calculate_quc_and_rcs","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/coherence_vs_constraint_graph.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/coherence_vs_constraint_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41dfba5ffb866e95"}},{"code_sha256_prefix":"b8d6b945452eee55","entry":"calculate_quc_and_rcs","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/quc_and_rcs.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/quc_and_rcs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8d6b945452eee55"}},{"code_sha256_prefix":"9b2db3fdfca284ab","entry":"chat","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/story_quality_eval.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/story_quality_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b2db3fdfca284ab"}},{"code_sha256_prefix":"b547c94262782cf4","entry":"generate_response","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"code_files/storygen.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/code_files/storygen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b547c94262782cf4"}},{"code_sha256_prefix":"f3c3e6083ca33d17","entry":"initialize_openai","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/story_quality_eval.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/story_quality_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3c3e6083ca33d17"}},{"code_sha256_prefix":"4874c30058799290","entry":"load_grouped_dfs_from_json","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/quc_and_rcs.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/quc_and_rcs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4874c30058799290"}},{"code_sha256_prefix":"1a773a549f580915","entry":"main","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/constraint_satisfaction.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/constraint_satisfaction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a773a549f580915"}},{"code_sha256_prefix":"5baffd5407078613","entry":"main","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/diversity_calculation.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/diversity_calculation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5baffd5407078613"}},{"code_sha256_prefix":"c811600eab5c3bf7","entry":"main","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/diversity_graphs.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/diversity_graphs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c811600eab5c3bf7"}},{"code_sha256_prefix":"f6bdebfeb1ddf260","entry":"parse_evaluation","repo":"anirudhlakkaraju/cs4_benchmark","repo_kind":"official","path":"evaluation/story_quality_eval.py","file_url":"https://github.com/anirudhlakkaraju/cs4_benchmark/blob/HEAD/evaluation/story_quality_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6bdebfeb1ddf260"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}