{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-10715","title":"Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents","arxiv_id":"2602.10715","date":"2026-02-11","proceeding":null,"authors":["Yifei Li","Weidong Guo","Lingling Zhang","Rongman Xu","Muye Huang","Hui Liu","Lijiao Xu","Yu Xu","Jun Liu"],"abstract":"Long-term conversational memory is a core capability for LLM-based dialogue systems, yet existing benchmarks and evaluation protocols primarily focus on surface-level factual recall. In realistic interactions, appropriate responses often depend on implicit constraints such as user state, goals, or values that are not explicitly queried later. To evaluate this setting, we introduce \\textbf{LoCoMo-Plus}, a benchmark for assessing cognitive memory under cue--trigger semantic disconnect, where models must retain and apply latent constraints across long conversational contexts. We further show that conventional string-matching metrics and explicit task-type prompting are misaligned with such scenarios, and propose a unified evaluation framework based on constraint consistency. Experiments across diverse backbone models, retrieval-based methods, and memory systems demonstrate that cognitive memory remains challenging and reveals failures not captured by existing benchmarks. Our code and evaluation framework are publicly available at: https://github.com/xjtuleeyf/Locomo-Plus.","url_abs":"https://arxiv.org/abs/2602.10715","url_pdf":"https://arxiv.org/pdf/2602.10715","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.10715","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.10715"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/xjtuleeyf/Locomo-Plus","reach":null}],"summary":{"ran":8,"ran_draft_wrong":2,"ran_honours":1,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":16,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"914a5c41650fad4f","entry":"_build_model_input","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"914a5c41650fad4f"}},{"code_sha256_prefix":"b7f5153386e0f19b","entry":"_compute_summary","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7f5153386e0f19b"}},{"code_sha256_prefix":"30143e37aea019f9","entry":"_get_openai_client","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30143e37aea019f9"}},{"code_sha256_prefix":"dceda189c5455bb9","entry":"_get_task_instruction","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dceda189c5455bb9"}},{"code_sha256_prefix":"4ed4c87680ca0fb9","entry":"_parse_judge_response","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ed4c87680ca0fb9"}},{"code_sha256_prefix":"34802bcaf77d0e07","entry":"_prepend_conv_prefix","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34802bcaf77d0e07"}},{"code_sha256_prefix":"5c5765a0c9007c76","entry":"call_llm","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c5765a0c9007c76"}},{"code_sha256_prefix":"7e92aca251456902","entry":"call_model","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e92aca251456902"}},{"code_sha256_prefix":"ca34b992ed5a0906","entry":"call_test","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca34b992ed5a0906"}},{"code_sha256_prefix":"457fbd7ac27fc0d9","entry":"get_judge_prompt","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"457fbd7ac27fc0d9"}},{"code_sha256_prefix":"06a8fea429acf534","entry":"label_to_score","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"06a8fea429acf534"}},{"code_sha256_prefix":"92054394282a016d","entry":"_judge_one_record","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92054394282a016d"}},{"code_sha256_prefix":"285d282349ed0fc7","entry":"_load_env_local_sh","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"285d282349ed0fc7"}},{"code_sha256_prefix":"c6a4099105374ceb","entry":"_print_summary","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6a4099105374ceb"}},{"code_sha256_prefix":"5d858d0bbfee493c","entry":"call_vllm","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d858d0bbfee493c"}},{"code_sha256_prefix":"1b8262b1426a8d81","entry":"run_judge","repo":"xjtuleeyf/Locomo-Plus","repo_kind":"found_in_text","path":"evaluation_framework/task_eval/llm_as_judge.py","file_url":"https://github.com/xjtuleeyf/Locomo-Plus/blob/HEAD/evaluation_framework/task_eval/llm_as_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b8262b1426a8d81"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}