{"url":"/dataset/chocolate","name":"CHOCOLATE","full_name":"Captions Have Often ChOsen Lies About The Evidence","description_markdown":"**CHOCOLATE** is a benchmark for detecting and correcting factual inconsistency in generated chart captions. It consists of captions produced by six advanced models, which are categorized into three subsets:\r\n\r\n- **LVLM**: GPT-4V, Bard (before Gemini)\r\n- **LLM-based Pipeline**: DePlot + GPT-4\r\n- **Fine-tuned Model**: ChartT5, MatCha, UniChart\r\n\r\n\r\nThe charts are from two datasets: VisText and the Pew split of Chart-to-Text. In total, **CHOCOLATE** consists of **1,187 examples**. Each instance in **CHOCOLATE** consists of a caption generated by one of the models and the annotations of the factual errors for each caption sentence.\r\n\r\n### Paper Information\r\n\r\n- Paper: [https://arxiv.org/abs/2312.10160](https://arxiv.org/abs/2312.10160)\r\n- Code: [https://github.com/khuangaf/CHOCOLATE/](https://github.com/khuangaf/CHOCOLATE/)\r\n- Project: [https://khuangaf.github.io/CHOCOLATE](https://khuangaf.github.io/CHOCOLATE)\r\n\r\n\r\n### Citation\r\n\r\nIf you use the **CHOCOLATE** dataset in your work, please kindly cite the paper using this BibTeX:\r\n\r\n```\r\n@misc{huang-etal-2023-do,\r\n    title = \"Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning\",\r\n    author = \"Huang, Kung-Hsiang  and\r\n      Zhou, Mingyang and\r\n      Chan, Hou Pong  and\r\n      Fung, Yi R. and\r\n      Wang, Zhenhailong and\r\n      Zhang, Lingyu and\r\n      Chang, Shih-Fu and\r\n      Ji, Heng\",\r\n    year={2023},\r\n    eprint={2312.10160},\r\n    archivePrefix={arXiv},\r\n    primaryClass={cs.CL}\r\n}    \r\n```","description_withheld":null,"homepage":"https://khuangaf.github.io/CHOCOLATE/","introduced_date":"2023-12-15","introduced_date_note":null,"introduced_by":{"paper":"/paper/do-lvlms-understand-charts-analyzing-and","title":"Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning","first_author":"Kung-Hsiang Huang","url":null},"license":{"name":"Apache-2.0 license","url":"https://github.com/khuangaf/CHOCOLATE?tab=Apache-2.0-1-ov-file#readme"},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Factual Inconsistency Detection in Chart Captioning","url":"/task/factual-inconsistency-detection-in-chart","datasets_with_task":"/datasets/task/factual-inconsistency-detection-in-chart"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["CHOCOLATE","CHOCOLATE-LLM","CHOCOLATE-FT","CHOCOLATE-LVLM"],"data_loaders":[{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/khhuang/CHOCOLATE","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/khuangaf/chocolate","url":"https://huggingface.co/datasets/khhuang/CHOCOLATE","frameworks":["pytorch"]}],"num_papers_in_archive":4,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/factual-inconsistency-detection-in-chart-1","task":"Factual Inconsistency Detection in Chart Captioning","dataset_variant":"CHOCOLATE-LLM","rows":5,"metrics":["Kendall's Tau-c"],"first_row_in_archive_order":{"model":"GPT-4V","paper":"/paper/gpt-4-technical-report-1","metrics":{"Kendall's Tau-c":"0.205"},"code_links":[{"title":"openai/evals","url":"https://github.com/openai/evals"},{"title":"shmsw25/factscore","url":"https://github.com/shmsw25/factscore"},{"title":"unispac/visual-adversarial-examples-jailbreak-large-language-models","url":"https://github.com/unispac/visual-adversarial-examples-jailbreak-large-language-models"},{"title":"gpt4life/alpagasus","url":"https://github.com/gpt4life/alpagasus"},{"title":"emrgnt-cmplxty/zero-shot-replication","url":"https://github.com/emrgnt-cmplxty/zero-shot-replication"},{"title":"ethz-privsec/superhuman-ai-consistency","url":"https://github.com/ethz-privsec/superhuman-ai-consistency"},{"title":"ethz-spylab/superhuman-ai-consistency","url":"https://github.com/ethz-spylab/superhuman-ai-consistency"},{"title":"eternityyw/tram-benchmark","url":"https://github.com/eternityyw/tram-benchmark"},{"title":"AUCOHL/RTL-Repo","url":"https://github.com/AUCOHL/RTL-Repo"},{"title":"zach-zhiling-zheng/reticular_chemist","url":"https://github.com/zach-zhiling-zheng/reticular_chemist"},{"title":"lflage/openfactscore","url":"https://github.com/lflage/openfactscore"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/factual-inconsistency-detection-in-chart-2","task":"Factual Inconsistency Detection in Chart Captioning","dataset_variant":"CHOCOLATE-FT","rows":5,"metrics":["Kendall's Tau-c"],"first_row_in_archive_order":{"model":"Bard (before Gemini)","paper":null,"metrics":{"Kendall's Tau-c":"0.291"},"code_links":[]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/factual-inconsistency-detection-in-chart-3","task":"Factual Inconsistency Detection in Chart Captioning","dataset_variant":"CHOCOLATE-LVLM","rows":5,"metrics":["Kendall's Tau-c"],"first_row_in_archive_order":{"model":"ChartVE","paper":"/paper/do-lvlms-understand-charts-analyzing-and","metrics":{"Kendall's Tau-c":"0.178"},"code_links":[{"title":"huggingface/transformers","url":"https://github.com/huggingface/transformers"},{"title":"salesforceairesearch/crmarena","url":"https://github.com/salesforceairesearch/crmarena"},{"title":"khuangaf/chocolate","url":"https://github.com/khuangaf/chocolate"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/factual-inconsistency-detection-in-chart","task":"Factual Inconsistency Detection in Chart Captioning","dataset_variant":"CHOCOLATE","rows":1,"metrics":["Kendall's Tau-c"],"first_row_in_archive_order":{"model":"ChartVE","paper":"/paper/do-lvlms-understand-charts-analyzing-and","metrics":{"Kendall's Tau-c":"0.178"},"code_links":[{"title":"huggingface/transformers","url":"https://github.com/huggingface/transformers"},{"title":"salesforceairesearch/crmarena","url":"https://github.com/salesforceairesearch/crmarena"},{"title":"khuangaf/chocolate","url":"https://github.com/khuangaf/chocolate"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/do-lvlms-understand-charts-analyzing-and","title":"Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning","date":"2023-12-15","rows_on_this_dataset":4,"code_links":3,"syntology":null},{"paper":"/paper/improved-baselines-with-visual-instruction","title":"Improved Baselines with Visual Instruction Tuning","date":"2023-10-05","rows_on_this_dataset":3,"code_links":9,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":9,"samples_ran":6,"samples_unverified":3,"pointer_only_for_licence":8,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/gpt-4-technical-report-1","title":"GPT-4 Technical Report","date":"2023-03-15","rows_on_this_dataset":1,"code_links":11,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":5,"samples_ran":2,"samples_unverified":3,"pointer_only_for_licence":1,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/deplot-one-shot-visual-language-reasoning-by","title":"DePlot: One-shot visual language reasoning by plot-to-table translation","date":"2022-12-20","rows_on_this_dataset":3,"code_links":1,"syntology":null}],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":2,"samples_harvested":14,"samples_ran":8,"samples_unverified":6,"pointer_only_for_licence":9,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}