{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neuro-symbolic-language-modeling-with","title":"Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval","arxiv_id":"2201.12431","date":"2022-01-28","proceeding":null,"authors":["Uri Alon","Frank F. Xu","Junxian He","Sudipta Sengupta","Dan Roth","Graham Neubig"],"abstract":"Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time. While effective, a major bottleneck of using these models in practice is the computationally costly datastore search, which can be performed as frequently as every time step. In this paper, we present RetoMaton - retrieval automaton - which approximates the datastore search, based on (1) saving pointers between consecutive datastore entries, and (2) clustering of entries into \"states\". This effectively results in a weighted finite automaton built on top of the datastore, instead of representing the datastore as a flat list. The creation of the automaton is unsupervised, and a RetoMaton can be constructed from any text collection: either the original training corpus or from another domain. Traversing this automaton at inference time, in parallel to the LM inference, reduces its perplexity by up to 1.85, or alternatively saves up to 83% of the nearest neighbor searches over $k$NN-LM (Khandelwal et al., 2020) without hurting perplexity. Our code and trained models are available at https://github.com/neulab/retomaton .","url_abs":"https://arxiv.org/abs/2201.12431v2","url_pdf":"https://arxiv.org/pdf/2201.12431v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neuro-symbolic-language-modeling-with","repo_url":"https://github.com/neulab/retomaton","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neuro-symbolic-language-modeling-with","repo_url":"https://github.com/neulab/knn-transformers","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2201.12431","atlas_url":"https://app.syntology.ai/?focus=2201.12431","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.12431"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/neulab/retomaton","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neulab/knn-transformers","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"5b87ede004fa49b0","entry":"get_members_path","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"5b87ede004fa49b0"}},{"code_sha256_prefix":"ce70a7f3d1a52dba","entry":"KNNWrapper","repo":"neulab/knn-transformers","repo_kind":"official","path":"retomaton.py","file_url":"https://github.com/neulab/knn-transformers/blob/HEAD/retomaton.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce70a7f3d1a52dba"}},{"code_sha256_prefix":"0ebdfa0124e8ed1a","entry":"KNN_Dstore","repo":"neulab/retomaton","repo_kind":"official","path":"fairseq/knnlm.py","file_url":"https://github.com/neulab/retomaton/blob/HEAD/fairseq/knnlm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ebdfa0124e8ed1a"}},{"code_sha256_prefix":"616b7b6be663b3a6","entry":"RetomatonWrapper","repo":"neulab/knn-transformers","repo_kind":"official","path":"retomaton.py","file_url":"https://github.com/neulab/knn-transformers/blob/HEAD/retomaton.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"616b7b6be663b3a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}