{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/let-llms-take-on-the-latest-challenges-a","title":"Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark","arxiv_id":"2402.19248","date":"2024-02-29","proceeding":null,"authors":["Zhikun Xu","Yinghui Li","Ruixue Ding","Xinyu Wang","Boli Chen","Yong Jiang","Hai-Tao Zheng","Wenlian Lu","Pengjun Xie","Fei Huang"],"abstract":"How to better evaluate the capabilities of Large Language Models (LLMs) is the focal point and hot topic in current LLMs research. Previous work has noted that due to the extremely high cost of iterative updates of LLMs, they are often unable to answer the latest dynamic questions well. To promote the improvement of Chinese LLMs' ability to answer dynamic questions, in this paper, we introduce CDQA, a Chinese Dynamic QA benchmark containing question-answer pairs related to the latest news on the Chinese Internet. We obtain high-quality data through a pipeline that combines humans and models, and carefully classify the samples according to the frequency of answer changes to facilitate a more fine-grained observation of LLMs' capabilities. We have also evaluated and analyzed mainstream and advanced Chinese LLMs on CDQA. Extensive experiments and valuable insights suggest that our proposed CDQA is challenging and worthy of more further study. We believe that the benchmark we provide will become one of the key data resources for improving LLMs' Chinese question-answering ability in the future.","url_abs":"https://arxiv.org/abs/2402.19248v2","url_pdf":"https://arxiv.org/pdf/2402.19248v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"let-llms-take-on-the-latest-challenges-a","repo_url":"https://github.com/alibaba-nlp/cdqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.19248","atlas_url":"https://app.syntology.ai/?focus=2402.19248","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.19248"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alibaba-nlp/cdqa","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"7756c7928a5f4188","entry":"compute_acc","repo":"alibaba-nlp/cdqa","repo_kind":"official","path":"cdqa_eval.py","file_url":"https://github.com/alibaba-nlp/cdqa/blob/HEAD/cdqa_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7756c7928a5f4188"}},{"code_sha256_prefix":"23193098026f155d","entry":"process_string","repo":"alibaba-nlp/cdqa","repo_kind":"official","path":"cdqa_eval.py","file_url":"https://github.com/alibaba-nlp/cdqa/blob/HEAD/cdqa_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23193098026f155d"}},{"code_sha256_prefix":"a208a2b10bcd3ae0","entry":"compute_acc_single","repo":"alibaba-nlp/cdqa","repo_kind":"official","path":"cdqa_eval.py","file_url":"https://github.com/alibaba-nlp/cdqa/blob/HEAD/cdqa_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a208a2b10bcd3ae0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}