{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-10008","title":"Do LLM Recommenders Know When They're Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness","arxiv_id":"2608.10008","date":"2026-08-07","proceeding":null,"authors":["Srijith Ravikumar"],"abstract":"LLM recommenders for top-K item suggestion regularly emit titles outside the target catalog. Prior audits report a binary out-of-domain rate; none ask whether the model knew. We jointly audit hallucination rate (OOD@10) and verbalized-confidence calibration (ECE, Brier, reliability) for four zero-shot LLM recommenders from four independent vendors (Mistral Large, Llama-3.3-70B, GPT-OSS-120B, Claude Sonnet 4.6), not grounded or fine-tuned systems, across three catalogs (MovieLens-25M, Amazon Reviews 2023 Toys, Yelp Open Dataset), stratified by item popularity. Measuring catalog membership is itself the hard part: on identical outputs the reported rate moves by an order of magnitude with the string matcher used, and F1 cannot separate the candidates. We validate the instrument against 201 human judgments and select on net bias, where the adopted one is off by -0.040 against +0.144 for the common fuzzy rule. Hallucination is then strongly catalog-dependent (0.6-2.7% on MovieLens, 11.6-38.7% on Yelp, 49.3-61.0% on Amazon Toys). Each model holds a near-constant confidence level barely responsive to the catalog, while the catalog-hit rate swings 60 points, so the sign of the error is set by where a model's constant lands against a catalog's accuracy: 7 of the twelve cells are under-confident and 5 over-confident, all four under-confident on MovieLens, all four over-confident on Amazon Toys. We read this as an elicitation mismatch: \"Just Ask\" elicits a generic quality rating, not a catalog-membership probability. A conformal abstention threshold over verbalized confidence changes hallucination by at most 1.65 pp across four alpha levels, because the channel cannot separate correct items from hallucinations. We recommend that audits report calibration alongside OOD, validate the matcher producing the OOD number, and use catalog-anchored elicitation.","url_abs":"https://arxiv.org/abs/2608.10008","url_pdf":"https://arxiv.org/pdf/2608.10008","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.10008"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/rsrijith/cikm26-catalog-faithfulness","reach":null}],"summary":{"ran":12,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"0258c93781373449","entry":"assign_popularity_quartile","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/utils.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0258c93781373449"}},{"code_sha256_prefix":"118dd165f4b0636d","entry":"build_user","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/llm_matcher.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/llm_matcher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"118dd165f4b0636d"}},{"code_sha256_prefix":"8ddf280a00303a8d","entry":"cands","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/build_round2_scored.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/build_round2_scored.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8ddf280a00303a8d"}},{"code_sha256_prefix":"d3106d9183a743bf","entry":"kappa","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/instrument_table.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/instrument_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d3106d9183a743bf"}},{"code_sha256_prefix":"3df0ffcce275599c","entry":"load","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"rehydrate.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/rehydrate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3df0ffcce275599c"}},{"code_sha256_prefix":"7a1382f2ea061f72","entry":"load_jsonl","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/prep_yelp.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/prep_yelp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7a1382f2ea061f72"}},{"code_sha256_prefix":"ea4a881fea734c24","entry":"load_jsonl_gz","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/prep_amazon.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/prep_amazon.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ea4a881fea734c24"}},{"code_sha256_prefix":"2eefaf671dee64d1","entry":"normalize_title","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/utils.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2eefaf671dee64d1"}},{"code_sha256_prefix":"35758791ba7f23c9","entry":"parse_reply","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/llm_matcher.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/llm_matcher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"35758791ba7f23c9"}},{"code_sha256_prefix":"24b62e60be6a67bd","entry":"sample_users_with_holdout","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/utils.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"24b62e60be6a67bd"}},{"code_sha256_prefix":"038db614afac5183","entry":"scores","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/instrument_table.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/instrument_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"038db614afac5183"}},{"code_sha256_prefix":"138b5512dc7a0294","entry":"sortkey","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/surface_match.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/surface_match.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"138b5512dc7a0294"}},{"code_sha256_prefix":"b1101b782b70b745","entry":"boot_ci","repo":"rsrijith/cikm26-catalog-faithfulness","repo_kind":"found_in_text","path":"code/instrument_table.py","file_url":"https://github.com/rsrijith/cikm26-catalog-faithfulness/blob/HEAD/code/instrument_table.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b1101b782b70b745"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}