{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-large-language-models-always-solve-easy","title":"Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?","arxiv_id":"2406.12809","date":"2024-06-18","proceeding":null,"authors":["Zhe Yang","Yichang Zhang","Tianyu Liu","Jian Yang","Junyang Lin","Chang Zhou","Zhifang Sui"],"abstract":"Large language models (LLMs) have demonstrated impressive capabilities, but still suffer from inconsistency issues (e.g. LLMs can react differently to disturbances like rephrasing or inconsequential order change). In addition to these inconsistencies, we also observe that LLMs, while capable of solving hard problems, can paradoxically fail at easier ones. To evaluate this hard-to-easy inconsistency, we develop the ConsisEval benchmark, where each entry comprises a pair of questions with a strict order of difficulty. Furthermore, we introduce the concept of consistency score to quantitatively measure this inconsistency and analyze the potential for improvement in consistency by relative consistency score. Based on comprehensive experiments across a variety of existing models, we find: (1) GPT-4 achieves the highest consistency score of 92.2\\% but is still inconsistent to specific questions due to distraction by redundant information, misinterpretation of questions, etc.; (2) models with stronger capabilities typically exhibit higher consistency, but exceptions also exist; (3) hard data enhances consistency for both fine-tuning and in-context learning. Our data and code will be publicly available on GitHub.","url_abs":"https://arxiv.org/abs/2406.12809v1","url_pdf":"https://arxiv.org/pdf/2406.12809v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-large-language-models-always-solve-easy","repo_url":"https://github.com/QwenLM/ConsisEval","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12809","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12809"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/QwenLM/ConsisEval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":11,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8bec43a95385f997","entry":"are_equal_under_sympy","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"math_check/grader.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/math_check/grader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8bec43a95385f997"}},{"code_sha256_prefix":"b0505332a16b5516","entry":"check_correctness_code","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"code_check/check.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/code_check/check.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0505332a16b5516"}},{"code_sha256_prefix":"d197da35ee544f75","entry":"conflict_make","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"instruction_following_check/instructions_registry.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/instruction_following_check/instructions_registry.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d197da35ee544f75"}},{"code_sha256_prefix":"54417bf5eb16e81f","entry":"count_unknown_letters_in_expr","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"math_check/grader.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/math_check/grader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54417bf5eb16e81f"}},{"code_sha256_prefix":"cdcc85ca09b00f7e","entry":"count_words","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"instruction_following_check/instructions_util.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/instruction_following_check/instructions_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cdcc85ca09b00f7e"}},{"code_sha256_prefix":"b22b268ce5bb5b68","entry":"extract_answer","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"math_check/check.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/math_check/check.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b22b268ce5bb5b68"}},{"code_sha256_prefix":"101b1e54fcdadf90","entry":"normalize_answer","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"math_check/math_normalize.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/math_check/math_normalize.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"101b1e54fcdadf90"}},{"code_sha256_prefix":"dab90bb25f71a531","entry":"process_text","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"code_check/post_process.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/code_check/post_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dab90bb25f71a531"}},{"code_sha256_prefix":"29ee1d8c120f674a","entry":"remove_after_return","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"code_check/post_process.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/code_check/post_process.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29ee1d8c120f674a"}},{"code_sha256_prefix":"f5c7c44887cd8c88","entry":"should_allow_eval","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"math_check/grader.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/math_check/grader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5c7c44887cd8c88"}},{"code_sha256_prefix":"34d346853108011e","entry":"split_into_sentences","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"instruction_following_check/instructions_util.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/instruction_following_check/instructions_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34d346853108011e"}},{"code_sha256_prefix":"f9e0e08dca784150","entry":"check_correctness_instruction_following","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"instruction_following_check/check.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/instruction_following_check/check.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9e0e08dca784150"}},{"code_sha256_prefix":"a13ec071b5563491","entry":"check_correctness_math","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"math_check/check.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/math_check/check.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a13ec071b5563491"}},{"code_sha256_prefix":"f79646ddd0a03162","entry":"count_sentences","repo":"QwenLM/ConsisEval","repo_kind":"official","path":"instruction_following_check/instructions_util.py","file_url":"https://github.com/QwenLM/ConsisEval/blob/HEAD/instruction_following_check/instructions_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f79646ddd0a03162"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}