{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2609-01556","title":"Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories","arxiv_id":"2609.01556","date":"2026-09-01","proceeding":null,"authors":["Nabira Rashid","Manolis Kellis"],"abstract":"We evaluate embedding retrieval where surface form and meaning are pulled apart on purpose: retrieving items that share underlying structure but not wording, in two unrelated domains under one protocol, competition mathematics (MathNet-Retrieve; 500 queries, 117,088-item corpus) and embodied-agent trajectories (ALFWorld-derived; 118 queries, 336 trajectories). In mathematics the failure is complete: strict Hit@1 at the heaviest disguise tier is 0.0% for both production embedders (bootstrap 95% CI [0.0, 0.0]) while the correct item sits in the top 10 nearly always, and in 95.2 to 99.8% of misses the winner is more lexically similar to the query than the correct answer. In trajectories, where surface variation is incidental, the same models land at or near hypergeometric chance when gold must involve a different object, and below chance for all three embedders once gold must differ in object and receptacle: retrieval anchors on literal tokens, not task structure. A lexical reranker control hurts in mathematics and helps in trajectories (closing 26 to 36% of the gap, CIs excluding zero); its sign reveals whether a benchmark's surface variation is adversarial or incidental. An LLM reranker recovers 5 to 63% of the gap in mathematics and 43 to 76% in trajectories; direction replicates across three judges (all 21 cells positive), but effect sizes, tier profiles, and the outlier judge change with domain (paired differences excluding zero everywhere). Mathematics gains concentrate on well-known competitions (+19.8 points, CI [+6.7, +33.2], one of six cells), so part of the recovery is memorization. In a paired downstream experiment (210 queries, graders at 96 to 99% agreement), oracle retrieval was indistinguishable from adversarially bad retrieval (McNemar p = 0.678); the solver's 69.5% zero-shot accuracy is largely a truncation proxy (97 to 100% on finished answers), leaving no headroom.","url_abs":"https://arxiv.org/abs/2609.01556","url_pdf":"https://arxiv.org/pdf/2609.01556","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2609.01556","atlas_url":"https://app.syntology.ai/?focus=2609.01556","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2609.01556"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/nabirarashid/structural-retrieval","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":10},"by_repo_kind":{"found_in_text":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"0d51dbcc3ac4c40f","entry":"call_judge","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/llm_judge_haiku.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/llm_judge_haiku.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0d51dbcc3ac4c40f"}},{"code_sha256_prefix":"c2ff04c5968543a3","entry":"call_solver","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/llm_solver_deepinfra.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/llm_solver_deepinfra.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c2ff04c5968543a3"}},{"code_sha256_prefix":"8f2f9b63249eeacd","entry":"call_solver","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/llm_solver_deepseek.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/llm_solver_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8f2f9b63249eeacd"}},{"code_sha256_prefix":"cc1529e569f22f09","entry":"dumb_score","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/dumb_reranker.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/dumb_reranker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cc1529e569f22f09"}},{"code_sha256_prefix":"88655287e149326b","entry":"load_jsonl","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/data.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"88655287e149326b"}},{"code_sha256_prefix":"79ffe3d7b0c300c3","entry":"load_qrels","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/data.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"79ffe3d7b0c300c3"}},{"code_sha256_prefix":"8d15ffc5c919d9bb","entry":"parse_corpus_id","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/data.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8d15ffc5c919d9bb"}},{"code_sha256_prefix":"205af17267b095fc","entry":"record_call","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/spend_tracker.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/spend_tracker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"205af17267b095fc"}},{"code_sha256_prefix":"534f86c2fb546a3e","entry":"rerank_top10","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/dumb_reranker.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/dumb_reranker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"534f86c2fb546a3e"}},{"code_sha256_prefix":"5f64d8fad6ce70e4","entry":"rerank_top10_llm","repo":"nabirarashid/structural-retrieval","repo_kind":"found_in_text","path":"src/llm_reranker.py","file_url":"https://github.com/nabirarashid/structural-retrieval/blob/HEAD/src/llm_reranker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5f64d8fad6ce70e4"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_daily_20260902.json"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}