{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-by-abstraction-the-neural-state","title":"Learning by Abstraction: The Neural State Machine","arxiv_id":"1907.03950","date":"2019-07-09","proceeding":"NeurIPS 2019 12","authors":["Drew A. Hudson","Christopher D. Manning"],"abstract":"We introduce the Neural State Machine, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning. Given an image, we first predict a probabilistic graph that represents its underlying semantics and serves as a structured world model. Then, we perform sequential reasoning over the graph, iteratively traversing its nodes to answer a given question or draw a new inference. In contrast to most neural architectures that are designed to closely interact with the raw sensory data, our model operates instead in an abstract latent space, by transforming both the visual and linguistic modalities into semantic concept-based representations, thereby achieving enhanced transparency and modularity. We evaluate our model on VQA-CP and GQA, two recent VQA datasets that involve compositionality, multi-step inference and diverse reasoning skills, achieving state-of-the-art results in both cases. We provide further experiments that illustrate the model's strong generalization capacity across multiple dimensions, including novel compositions of concepts, changes in the answer distribution, and unseen linguistic structures, demonstrating the qualities and efficacy of our approach.","url_abs":"https://arxiv.org/abs/1907.03950v4","url_pdf":"https://arxiv.org/pdf/1907.03950v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-by-abstraction-the-neural-state","repo_url":"https://github.com/stanfordnlp/mac-network","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"learning-by-abstraction-the-neural-state","repo_url":"https://github.com/abwilf/social-iq-2.0-challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-by-abstraction-the-neural-state","repo_url":"https://github.com/ceyzaguirre4/NSM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"learning-by-abstraction-the-neural-state","repo_url":"https://github.com/gchaperon/neural-state-machine","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-gqa-test-dev","task":"Visual Question Answering (VQA)","dataset":"GQA test-dev","model":"NSM","rank_in_archive_order":5,"of":17,"metrics":{"Accuracy":"62.95"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-gqa-test-std","task":"Visual Question Answering (VQA)","dataset":"GQA test-std","model":"NSM","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"63.17"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-cp","task":"Visual Question Answering (VQA)","dataset":"VQA-CP","model":"NSM","rank_in_archive_order":8,"of":10,"metrics":{"Score":"45.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1907.03950","atlas_url":"https://app.syntology.ai/?focus=1907.03950","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.03950"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abwilf/social-iq-2.0-challenge","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stanfordnlp/mac-network","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gchaperon/neural-state-machine","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ceyzaguirre4/NSM","reach":null}],"summary":{"ran_honours":2,"ran":7,"ran_fixture":1,"unverified":11},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1},"listed":{"samples":13,"ran":10,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"616b5e82dd2a38f9","entry":"count_params","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"baselines/chatgpt/alex_utils.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/baselines/chatgpt/alex_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"616b5e82dd2a38f9"}},{"code_sha256_prefix":"8312fd020fcf129a","entry":"chatGPT","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"baselines/chatgpt/chatgpt_baseline.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/baselines/chatgpt/chatgpt_baseline.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8312fd020fcf129a"}},{"code_sha256_prefix":"fe946dd42b320c75","entry":"compute_colors_for_labels","repo":"ceyzaguirre4/NSM","repo_kind":"listed","path":"mrcnn-test.py","file_url":"https://github.com/ceyzaguirre4/NSM/blob/HEAD/mrcnn-test.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe946dd42b320c75"}},{"code_sha256_prefix":"56c215932698a004","entry":"fix_cat_grid","repo":"gchaperon/neural-state-machine","repo_kind":"listed","path":"acc_per_cat.py","file_url":"https://github.com/gchaperon/neural-state-machine/blob/HEAD/acc_per_cat.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56c215932698a004"}},{"code_sha256_prefix":"87b3b6d59b5f6cb0","entry":"n_filters","repo":"gchaperon/neural-state-machine","repo_kind":"listed","path":"compare_model_results.py","file_url":"https://github.com/gchaperon/neural-state-machine/blob/HEAD/compare_model_results.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87b3b6d59b5f6cb0"}},{"code_sha256_prefix":"e7f4ccb81d678b17","entry":"process_defaults","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"baselines/chatgpt/alex_utils.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/baselines/chatgpt/alex_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7f4ccb81d678b17"}},{"code_sha256_prefix":"083c67090bc76ac3","entry":"read_uploaded_vtt","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"siq2/youtube_utils.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/siq2/youtube_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"083c67090bc76ac3"}},{"code_sha256_prefix":"cc0eeb809adea837","entry":"sec_to_ts","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"siq2/youtube_utils.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/siq2/youtube_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc0eeb809adea837"}},{"code_sha256_prefix":"d13c6151b26cba67","entry":"to_glove","repo":"ceyzaguirre4/NSM","repo_kind":"listed","path":"NSM.py","file_url":"https://github.com/ceyzaguirre4/NSM/blob/HEAD/NSM.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d13c6151b26cba67"}},{"code_sha256_prefix":"b50957d3d7d06ece","entry":"ts_to_sec","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"siq2/youtube_utils.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/siq2/youtube_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b50957d3d7d06ece"}},{"code_sha256_prefix":"8317ab884576997a","entry":"bcolored","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8317ab884576997a"}},{"code_sha256_prefix":"0c7aaed4c145f580","entry":"bold","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"preprocess.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c7aaed4c145f580"}},{"code_sha256_prefix":"d90ad06e8aff38ed","entry":"build_model","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"extract_features.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/extract_features.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d90ad06e8aff38ed"}},{"code_sha256_prefix":"d4ecbe5474146ede","entry":"compute_correctness","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"baselines/chatgpt/chatgpt_baseline.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/baselines/chatgpt/chatgpt_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4ecbe5474146ede"}},{"code_sha256_prefix":"877a679a8e0c8e14","entry":"compute_correctness","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"baselines/flant5xl/flant5xl_baseline.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/baselines/flant5xl/flant5xl_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"877a679a8e0c8e14"}},{"code_sha256_prefix":"4834aa38d80ec5ee","entry":"filter","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"visualization.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/visualization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4834aa38d80ec5ee"}},{"code_sha256_prefix":"c58f5331f6d04dc6","entry":"getBias","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"ops.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c58f5331f6d04dc6"}},{"code_sha256_prefix":"278ebc1a30cb3ac5","entry":"getKernel","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"ops.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"278ebc1a30cb3ac5"}},{"code_sha256_prefix":"d229950ef10faeee","entry":"getWeight","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"ops.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d229950ef10faeee"}},{"code_sha256_prefix":"f9482a830c57d09c","entry":"run_batch","repo":"stanfordnlp/mac-network","repo_kind":"official","path":"extract_features.py","file_url":"https://github.com/stanfordnlp/mac-network/blob/HEAD/extract_features.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9482a830c57d09c"}},{"code_sha256_prefix":"f1c340b23f7fad32","entry":"tens","repo":"abwilf/social-iq-2.0-challenge","repo_kind":"listed","path":"baselines/chatgpt/alex_utils.py","file_url":"https://github.com/abwilf/social-iq-2.0-challenge/blob/HEAD/baselines/chatgpt/alex_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1c340b23f7fad32"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}