Papers › The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention
The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention
Yufa Zhou
Title, abstract, authors and date from arXiv's metadata (CC0); this paper is not in the Papers with Code archive (frozen 2025-07-28).
Human reasoning depends on how objects are related within propositions. How do relations organize the language representations of contextual contents? We give an LLM a list of facts in its context (e.g., Alice eats an apple. Bob eats a pear.) and measure how its hidden state changes when the question switches from what Alice eats to what Bob eats. Averaged over many lists, this change is a steering vector, which we call the ordinal vector. It points to a fact by its order of mention, the order in which the facts were stated in the context. We find that LLMs represent the fact a question asks about by its order of mention, not by the name the question contains. We state this as the ordinal addressing hypothesis: each order of mention has a fact address in the model's state, shared by all contexts, and a question moves the state to the fact address of the fact it asks about, while the context supplies what that fact says. Across Qwen, Gemma, and Llama, fact addresses are (1) ordered by mention: query states are organized by the order of facts, not of names, even when one fact has multiple subjects; (2) steerable: added to a question about the first fact of a new list, the ordinal vector makes the model answer with the second fact of that list; (3) low-rank: they span a low-rank subspace in which the first-mentioned fact is the easiest to reach, surprisingly similar to human recall; and (4) emergent: they are shared in late-middle layers, hold from 1.5B to 32B parameters, and form early in pretraining. Language models reach a stated fact by where it was mentioned, deepening our understanding of LLM reasoning.
In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.
For agents, on Syntology's MCP service (how to connect):
get_citation_path(paper_1="2610.00910", paper_2="…")with another paper's arXiv id or titleget_concepts_for_paper(arxiv_id="2610.00910")
Code
Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Syntology holds the repository link but has not harvested or run code from it.
Results from the paper
The Papers with Code archive ends with its 2025-07-28 snapshot. This paper's arXiv identifier, 2610.00910, was issued in October 2026, after that date, so the archive has no leaderboard rows for it.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections