{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-measure-of-intelligence","title":"On the Measure of Intelligence","arxiv_id":"1911.01547","date":"2019-11-05","proceeding":null,"authors":["François Chollet"],"abstract":"To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to \"buy\" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.","url_abs":"https://arxiv.org/abs/1911.01547v2","url_pdf":"https://arxiv.org/pdf/1911.01547v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-measure-of-intelligence","repo_url":"https://github.com/fchollet/ARC","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-measure-of-intelligence","repo_url":"https://github.com/alejandrodemiquel/arc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-measure-of-intelligence","repo_url":"https://github.com/arcprize/arc-agi-2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-measure-of-intelligence","repo_url":"https://github.com/artyompal/kaggle-abstract-reasoning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-measure-of-intelligence","repo_url":"https://github.com/fchollet/arc-agi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-measure-of-intelligence","repo_url":"https://github.com/ilialarchenko/abstract_reasoning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"arc","task_name":"ARC"},{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[{"slug":"arc-the-abstraction-and-reasoning-corpus","name":"ARC (The Abstraction and Reasoning Corpus)","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1911.01547","atlas_url":"https://app.syntology.ai/?focus=1911.01547","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.01547"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alejandrodemiquel/arc","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ilialarchenko/abstract_reasoning","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/artyompal/kaggle-abstract-reasoning","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fchollet/ARC","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fchollet/arc-agi","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arcprize/arc-agi-2","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":10,"unverified":1},"by_repo_kind":{"listed":{"samples":11,"ran":10,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5705f797d6c8ed13","entry":"combine_submission_files","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/submission_utils.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/submission_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5705f797d6c8ed13"}},{"code_sha256_prefix":"2598408104f35ac7","entry":"combine_two_lists","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/functions.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2598408104f35ac7"}},{"code_sha256_prefix":"01b3f8dafbba00b7","entry":"encode_file","repo":"artyompal/kaggle-abstract-reasoning","repo_kind":"listed","path":"python-dsl-code/bake_kernel.py","file_url":"https://github.com/artyompal/kaggle-abstract-reasoning/blob/HEAD/python-dsl-code/bake_kernel.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01b3f8dafbba00b7"}},{"code_sha256_prefix":"cb5b66e357456a69","entry":"find_color_boundaries","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/preprocessing.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/preprocessing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb5b66e357456a69"}},{"code_sha256_prefix":"ab44366ceba04378","entry":"find_grid","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/preprocessing.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/preprocessing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab44366ceba04378"}},{"code_sha256_prefix":"54178cc12c59390e","entry":"generate_submission","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/submission_utils.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/submission_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54178cc12c59390e"}},{"code_sha256_prefix":"9b2b1b6f42de3ffe","entry":"get_color_max","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/preprocessing.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/preprocessing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b2b1b6f42de3ffe"}},{"code_sha256_prefix":"62a7b00a87aebb3b","entry":"matrix2answer","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62a7b00a87aebb3b"}},{"code_sha256_prefix":"b8bf0064ef2c92ca","entry":"swap_two_colors","repo":"ilialarchenko/abstract_reasoning","repo_kind":"listed","path":"src/functions.py","file_url":"https://github.com/ilialarchenko/abstract_reasoning/blob/HEAD/src/functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8bf0064ef2c92ca"}},{"code_sha256_prefix":"e5703b19f49bceeb","entry":"vcat_imgs","repo":"artyompal/kaggle-abstract-reasoning","repo_kind":"listed","path":"python-dsl-code/genetics.py","file_url":"https://github.com/artyompal/kaggle-abstract-reasoning/blob/HEAD/python-dsl-code/genetics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5703b19f49bceeb"}},{"code_sha256_prefix":"ca5e60c977324c1e","entry":"parse_output","repo":"artyompal/kaggle-abstract-reasoning","repo_kind":"listed","path":"unit_tests.py","file_url":"https://github.com/artyompal/kaggle-abstract-reasoning/blob/HEAD/unit_tests.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca5e60c977324c1e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}