{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-very-long-term-conversational","title":"Evaluating Very Long-Term Conversational Memory of LLM Agents","arxiv_id":"2402.17753","date":"2024-02-27","proceeding":null,"authors":["Adyasha Maharana","Dong-Ho Lee","Sergey Tulyakov","Mohit Bansal","Francesco Barbieri","Yuwei Fang"],"abstract":"Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.","url_abs":"https://arxiv.org/abs/2402.17753v1","url_pdf":"https://arxiv.org/pdf/2402.17753v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-very-long-term-conversational","repo_url":"https://github.com/danny911kr/REALTALK","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":null,"task_name":"Avg"},{"task_slug":"dialogue-generation","task_name":"Dialogue Generation"},{"task_slug":"multi-modal-dialogue-generation","task_name":"Multi-modal Dialogue Generation"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.17753","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.17753"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/danny911kr/REALTALK","reach":null}],"summary":{"ran":3,"unverified":4},"by_repo_kind":{"listed":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"9f677f6c6abefe77","entry":"LLMMessage","repo":"danny911kr/REALTALK","repo_kind":"listed","path":"eval_memory/qa_tool.py","file_url":"https://github.com/danny911kr/REALTALK/blob/HEAD/eval_memory/qa_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f677f6c6abefe77"}},{"code_sha256_prefix":"369d57b350a789f9","entry":"OpenAIGenerator","repo":"danny911kr/REALTALK","repo_kind":"listed","path":"eval_memory/qa_tool.py","file_url":"https://github.com/danny911kr/REALTALK/blob/HEAD/eval_memory/qa_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"369d57b350a789f9"}},{"code_sha256_prefix":"21024e1be55a2035","entry":"TokenUsage","repo":"danny911kr/REALTALK","repo_kind":"listed","path":"eval_memory/qa_tool.py","file_url":"https://github.com/danny911kr/REALTALK/blob/HEAD/eval_memory/qa_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21024e1be55a2035"}},{"code_sha256_prefix":"7202c1c6001763c6","entry":"LLMJSONResponse","repo":"danny911kr/REALTALK","repo_kind":"listed","path":"eval_memory/qa_tool.py","file_url":"https://github.com/danny911kr/REALTALK/blob/HEAD/eval_memory/qa_tool.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7202c1c6001763c6"}},{"code_sha256_prefix":"6ee6284bd946785c","entry":"LLMResponse","repo":"danny911kr/REALTALK","repo_kind":"listed","path":"eval_memory/qa_tool.py","file_url":"https://github.com/danny911kr/REALTALK/blob/HEAD/eval_memory/qa_tool.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ee6284bd946785c"}},{"code_sha256_prefix":"3e7b151905a2687b","entry":"QATool","repo":"danny911kr/REALTALK","repo_kind":"listed","path":"eval_memory/qa_tool.py","file_url":"https://github.com/danny911kr/REALTALK/blob/HEAD/eval_memory/qa_tool.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e7b151905a2687b"}},{"code_sha256_prefix":"50497c2df91e796d","entry":"StopReasonEnum","repo":"danny911kr/REALTALK","repo_kind":"listed","path":"eval_memory/qa_tool.py","file_url":"https://github.com/danny911kr/REALTALK/blob/HEAD/eval_memory/qa_tool.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50497c2df91e796d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}