{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2609-00103","title":"Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy","arxiv_id":"2609.00103","date":"2026-08-31","proceeding":null,"authors":["Shmuel Berman","Jia Deng"],"abstract":"Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video. However, accuracy alone misses properties that matter for real long-horizon tasks. We introduce ECCBench, a benchmark and evaluation protocol that measures memory beyond a system's capacity--its raw accuracy at a specific budget--via three axes we call ECC: efficiency--the computation, in FLOPs, needed to answer from memory; compression--whether compressible inputs are remembered more accurately or efficiently; and calibration--whether the system abstains in response to its own uncertainty and the cost of an error. We find that pretrained VLMs compress their memory over text but not video and are poorly calibrated on both. Among a broader set of memory backbones, several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers, suggesting they may be useful components for agents operating over long horizons.","url_abs":"https://arxiv.org/abs/2609.00103","url_pdf":"https://arxiv.org/pdf/2609.00103","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2609.00103","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2609.00103"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/princeton-vl/ECCBench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1412f492759b6327","entry":"build_max_memory_map","repo":"princeton-vl/ECCBench","repo_kind":"found_in_text","path":"src/substream/models/device_map_utils.py","file_url":"https://github.com/princeton-vl/ECCBench/blob/HEAD/src/substream/models/device_map_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1412f492759b6327"}},{"code_sha256_prefix":"a4899f0ae9b948d8","entry":"calculate_video_metrics","repo":"princeton-vl/ECCBench","repo_kind":"found_in_text","path":"src/substream/eval/wandb_log.py","file_url":"https://github.com/princeton-vl/ECCBench/blob/HEAD/src/substream/eval/wandb_log.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4899f0ae9b948d8"}},{"code_sha256_prefix":"7236d73956e14544","entry":"count_valid_completed_questions","repo":"princeton-vl/ECCBench","repo_kind":"found_in_text","path":"src/substream/eval/runner.py","file_url":"https://github.com/princeton-vl/ECCBench/blob/HEAD/src/substream/eval/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7236d73956e14544"}},{"code_sha256_prefix":"9a74db9744a8f5a5","entry":"extract_bucket_length","repo":"princeton-vl/ECCBench","repo_kind":"found_in_text","path":"src/substream/eval/runner.py","file_url":"https://github.com/princeton-vl/ECCBench/blob/HEAD/src/substream/eval/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a74db9744a8f5a5"}},{"code_sha256_prefix":"31c14f51035681c9","entry":"load_model_class","repo":"princeton-vl/ECCBench","repo_kind":"found_in_text","path":"src/substream/registry.py","file_url":"https://github.com/princeton-vl/ECCBench/blob/HEAD/src/substream/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31c14f51035681c9"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_daily_20260902.json"},"syntology_extracted_results":null}