{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measuring-abstract-reasoning-in-neural","title":"Measuring abstract reasoning in neural networks","arxiv_id":"1807.04225","date":"2018-07-11","proceeding":"ICML 2018 7","authors":["David G. T. Barrett","Felix Hill","Adam Santoro","Ari S. Morcos","Timothy Lillicrap"],"abstract":"Whether neural networks can learn abstract reasoning or whether they merely\nrely on superficial statistics is a topic of recent debate. Here, we propose a\ndataset and challenge designed to probe abstract reasoning, inspired by a\nwell-known human IQ test. To succeed at this challenge, models must cope with\nvarious generalisation `regimes' in which the training and test data differ in\nclearly-defined ways. We show that popular models such as ResNets perform\npoorly, even when the training and test sets differ only minimally, and we\npresent a novel architecture, with a structure designed to encourage reasoning,\nthat does significantly better. When we vary the way in which the test\nquestions and training data differ, we find that our model is notably\nproficient at certain forms of generalisation, but notably weak at others. We\nfurther show that the model's ability to generalise improves markedly if it is\ntrained to predict symbolic explanations for its answers. Altogether, we\nintroduce and explore ways to both measure and induce stronger abstract\nreasoning in neural networks. Our freely-available dataset should motivate\nfurther progress in this direction.","url_abs":"http://arxiv.org/abs/1807.04225v1","url_pdf":"http://arxiv.org/pdf/1807.04225v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"measuring-abstract-reasoning-in-neural","repo_url":"https://github.com/shinelink/abstract-reasoning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"measuring-abstract-reasoning-in-neural","repo_url":"https://github.com/mikomel/wild-relation-network","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[{"slug":"abstractreasoning","name":"AbstractReasoning","full_name":"AbstractReasoning"},{"slug":"pgm","name":"PGM","full_name":"Procedurally Generated Matrices (PGM)"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1807.04225","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}