{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-systematic-evaluation-of-large-language-1","title":"Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning","arxiv_id":"2310.09430","date":"2023-10-13","proceeding":null,"authors":["Qiming Bao","Gael Gendron","Alex Yuxuan Peng","Wanjun Zhong","Neset Tan","Yang Chen","Michael Witbrock","Jiamou Liu"],"abstract":"Large language models (LLMs), such as LLaMA, Alpaca, Vicuna, GPT-3.5 and GPT-4, have advanced the performance of AI systems on various natural language processing tasks to human-like levels. However, their generalisation and robustness when performing logical reasoning has not been sufficiently assessed. To comprehensively evaluate this ability, we develop three new logical reasoning datasets named \"ReClor-plus\", \"LogiQA-plus\" and \"LogiQAv2-plus\" that extend standard logical reasoning datasets to evaluate the robustness of the LLM's reasoning. For each, we create three subsets: the first with randomly shuffled options, the second with the correct choices replaced by \"none of the other options is correct\", and the third with a combination of shuffling and substitution. Experiments on these datasets show that these simple augmentations greatly hinder the models' performance. Despite their high performance on the original publicly available datasets, we find that all models perform poorly on these newly constructed datasets. We also demonstrate that introducing task variations into the training set can markedly improve the model's performance on both the original and our developed datasets. Finally, we show that applying logic-driven data augmentation for fine-tuning and prompting can enhance generalisation in both discriminative and generative models, offering a path to improving their robustness for tasks involving logical reasoning. Source code and data are made publicly available at https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning.","url_abs":"https://arxiv.org/abs/2310.09430v5","url_pdf":"https://arxiv.org/pdf/2310.09430v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-systematic-evaluation-of-large-language-1","repo_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.09430","atlas_url":"https://app.syntology.ai/?focus=2310.09430","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.09430"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"90ea10eb1aa031b1","entry":"convert_examples_to_features","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"models/utils_multiple_choice.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/models/utils_multiple_choice.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90ea10eb1aa031b1"}},{"code_sha256_prefix":"266daa7219ccdd1d","entry":"corrupt_bench","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"src/evaluate/information_gain.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/src/evaluate/information_gain.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"266daa7219ccdd1d"}},{"code_sha256_prefix":"237c07af276bf442","entry":"corrupt_pvr","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"src/evaluate/information_gain.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/src/evaluate/information_gain.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"237c07af276bf442"}},{"code_sha256_prefix":"fbc6ef16f2c20b2a","entry":"get_example_symbolic","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"src/evaluate/causal_induction_identification.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/src/evaluate/causal_induction_identification.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbc6ef16f2c20b2a"}},{"code_sha256_prefix":"a099bb4dec45b409","entry":"get_trial_symbolic","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"src/evaluate/causal_induction_identification.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/src/evaluate/causal_induction_identification.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a099bb4dec45b409"}},{"code_sha256_prefix":"0f3995390250c44f","entry":"parse_input_symbolic","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"src/evaluate/causal_induction_identification.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/src/evaluate/causal_induction_identification.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f3995390250c44f"}},{"code_sha256_prefix":"0a546b305d274996","entry":"select_field","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"models/run_multiple_choice.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/models/run_multiple_choice.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a546b305d274996"}},{"code_sha256_prefix":"3c241ecfe3749a6d","entry":"simple_accuracy","repo":"strong-ai-lab/logical-and-abstract-reasoning","repo_kind":"official","path":"models/run_multiple_choice.py","file_url":"https://github.com/strong-ai-lab/logical-and-abstract-reasoning/blob/HEAD/models/run_multiple_choice.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c241ecfe3749a6d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}