{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spinning-the-golden-thread-benchmarking-long","title":"LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs","arxiv_id":"2409.02076","date":"2024-09-03","proceeding":null,"authors":["Yuhao Wu","Ming Shan Hee","Zhiqing Hu","Roy Ka-Wei Lee"],"abstract":"Current benchmarks like Needle-in-a-Haystack (NIAH), Ruler, and Needlebench focus on models' ability to understand long-context input sequences but fail to capture a critical dimension: the generation of high-quality long-form text. Applications such as design proposals, technical documentation, and creative writing rely on coherent, instruction-following outputs over extended sequences - a challenge that existing benchmarks do not adequately address. To fill this gap, we introduce LongGenBench, a novel benchmark designed to rigorously evaluate large language models' (LLMs) ability to generate long text while adhering to complex instructions. Through tasks requiring specific events or constraints within generated text, LongGenBench evaluates model performance across four distinct scenarios, three instruction types, and two generation-lengths (16K and 32K tokens). Our evaluation of ten state-of-the-art LLMs reveals that, despite strong results on Ruler, all models struggled with long text generation on LongGenBench, particularly as text length increased. This suggests that current LLMs are not yet equipped to meet the demands of real-world, long-form text generation.","url_abs":"https://arxiv.org/abs/2409.02076v7","url_pdf":"https://arxiv.org/pdf/2409.02076v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spinning-the-golden-thread-benchmarking-long","repo_url":"https://github.com/mozhu621/SGT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"spinning-the-golden-thread-benchmarking-long","repo_url":"https://github.com/mozhu621/longgenbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"16k","task_name":"16k"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"form","task_name":"Form"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.02076","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.02076"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mozhu621/SGT","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mozhu621/longgenbench","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"f5f2acaf2973d95c","entry":"check_gpu_idle","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/check_gpu.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/check_gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5f2acaf2973d95c"}},{"code_sha256_prefix":"51d2edaf5a76a386","entry":"check_gpu_idle","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/check_gpu_4GPU_2.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/check_gpu_4GPU_2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51d2edaf5a76a386"}},{"code_sha256_prefix":"3c27f1988b1a83f6","entry":"load_inputs","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/API_Openai.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/API_Openai.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c27f1988b1a83f6"}},{"code_sha256_prefix":"48dcfa964c47e030","entry":"load_inputs","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/inference.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48dcfa964c47e030"}},{"code_sha256_prefix":"d0219388ba0b8779","entry":"process_output","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/API_Openai.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/API_Openai.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0219388ba0b8779"}},{"code_sha256_prefix":"1e69de300111f7f0","entry":"process_output","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/inference.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e69de300111f7f0"}},{"code_sha256_prefix":"def043e93912d864","entry":"process_output","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/inference_API.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/inference_API.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"def043e93912d864"}},{"code_sha256_prefix":"6bb6d8575edb075c","entry":"read_json","repo":"mozhu621/SGT","repo_kind":"official","path":"Evalution/API_Openai.py","file_url":"https://github.com/mozhu621/SGT/blob/HEAD/Evalution/API_Openai.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6bb6d8575edb075c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}