{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lateval-an-interactive-llms-evaluation","title":"LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles","arxiv_id":"2308.10855","date":"2023-08-21","proceeding":null,"authors":["Shulin Huang","Shirong Ma","Yinghui Li","Mengzuo Huang","Wuhe Zou","Weidong Zhang","Hai-Tao Zheng"],"abstract":"With the continuous evolution and refinement of LLMs, they are endowed with impressive logical reasoning or vertical thinking capabilities. But can they think out of the box? Do they possess proficient lateral thinking abilities? Following the setup of Lateral Thinking Puzzles, we propose a novel evaluation benchmark, LatEval, which assesses the model's lateral thinking within an interactive framework. In our benchmark, we challenge LLMs with 2 aspects: the quality of questions posed by the model and the model's capability to integrate information for problem-solving. We find that nearly all LLMs struggle with employing lateral thinking during interactions. For example, even the most advanced model, GPT-4, exhibits the advantage to some extent, yet still maintain a noticeable gap when compared to human. This evaluation benchmark provides LLMs with a highly challenging and distinctive task that is crucial to an effective AI assistant.","url_abs":"https://arxiv.org/abs/2308.10855v3","url_pdf":"https://arxiv.org/pdf/2308.10855v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lateval-an-interactive-llms-evaluation","repo_url":"https://github.com/thukelab/lateval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.10855","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.10855"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thukelab/lateval","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":6},"by_repo_kind":{"official":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"5f7de07e75040659","entry":"get_benchmark_results","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/Bloomchat/benchmark.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/Bloomchat/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f7de07e75040659"}},{"code_sha256_prefix":"23648dfbe89d08f8","entry":"get_model_path","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/Bloomchat/models/ds_inference.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/Bloomchat/models/ds_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23648dfbe89d08f8"}},{"code_sha256_prefix":"6d5ba8546b6aad2c","entry":"load_peft_model","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/Llama2_chat/model_utils.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/Llama2_chat/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d5ba8546b6aad2c"}},{"code_sha256_prefix":"1fa97532e4272e78","entry":"gpt35_messages","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/gpt35_api.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/gpt35_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fa97532e4272e78"}},{"code_sha256_prefix":"2bb9ddffcd138b3e","entry":"gpt35_request","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/gpt35_api.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/gpt35_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2bb9ddffcd138b3e"}},{"code_sha256_prefix":"bbb57eff5344e086","entry":"gpt4_messages","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/gpt4_api.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/gpt4_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bbb57eff5344e086"}},{"code_sha256_prefix":"14404140f8c1b2bc","entry":"gpt4_request","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/gpt4_api.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/gpt4_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14404140f8c1b2bc"}},{"code_sha256_prefix":"d9140c3ef7f275d4","entry":"load_file","repo":"thukelab/lateval","repo_kind":"official","path":"src/game.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/src/game.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9140c3ef7f275d4"}},{"code_sha256_prefix":"a5139b11f2293744","entry":"load_model","repo":"thukelab/lateval","repo_kind":"official","path":"chat_api/Llama2_chat/model_utils.py","file_url":"https://github.com/thukelab/lateval/blob/HEAD/chat_api/Llama2_chat/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5139b11f2293744"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}