{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mocha-multi-objective-reinforcement","title":"Mitigating Open-Vocabulary Caption Hallucinations","arxiv_id":"2312.03631","date":"2023-12-06","proceeding":null,"authors":["Assaf Ben-Kish","Moran Yanuka","Morris Alper","Raja Giryes","Hadar Averbuch-Elor"],"abstract":"While recent years have seen rapid progress in image-conditioned text generation, image captioning still suffers from the fundamental issue of hallucinations, namely, the generation of spurious details that cannot be inferred from the given image. Existing methods largely use closed-vocabulary object lists to mitigate or evaluate hallucinations in image captioning, ignoring the long-tailed nature of hallucinations that occur in practice. To this end, we propose a framework for addressing hallucinations in image captioning in the open-vocabulary setting. Our framework includes a new benchmark, OpenCHAIR, that leverages generative foundation models to evaluate open-vocabulary object hallucinations for image captioning, surpassing the popular and similarly-sized CHAIR benchmark in both diversity and accuracy. Furthermore, to mitigate open-vocabulary hallucinations without using a closed object list, we propose MOCHa, an approach harnessing advancements in reinforcement learning. Our multi-objective reward function explicitly targets the trade-off between fidelity and adequacy in generations without requiring any strong supervision. MOCHa improves a large variety of image captioning models, as captured by our OpenCHAIR benchmark and other existing metrics. Code and models can be found at: https://github.com/assafbk/mocha_code","url_abs":"https://arxiv.org/abs/2312.03631v4","url_pdf":"https://arxiv.org/pdf/2312.03631v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mocha-multi-objective-reinforcement","repo_url":"https://github.com/assafbk/mocha_code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"object","task_name":"Object"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[{"slug":"openchair","name":"OpenCHAIR","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.03631","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.03631"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/assafbk/mocha_code","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"54bb363a84810bbc","entry":"apply_ignore_words","repo":"assafbk/mocha_code","repo_kind":"official","path":"OpenCHAIR/evaluate.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/OpenCHAIR/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54bb363a84810bbc"}},{"code_sha256_prefix":"300b5f14c9912b97","entry":"collate_fn","repo":"assafbk/mocha_code","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"300b5f14c9912b97"}},{"code_sha256_prefix":"0c54289ae763b987","entry":"flatten_data","repo":"assafbk/mocha_code","repo_kind":"official","path":"OpenCHAIR/evaluate.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/OpenCHAIR/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c54289ae763b987"}},{"code_sha256_prefix":"ff3f62c52985c41c","entry":"is_concrete","repo":"assafbk/mocha_code","repo_kind":"official","path":"OpenCHAIR/utils.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/OpenCHAIR/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff3f62c52985c41c"}},{"code_sha256_prefix":"84d259ca9b98ac9a","entry":"load_config","repo":"assafbk/mocha_code","repo_kind":"official","path":"utils.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84d259ca9b98ac9a"}},{"code_sha256_prefix":"138d0bd7bf5f2513","entry":"load_llm_pipe","repo":"assafbk/mocha_code","repo_kind":"official","path":"OpenCHAIR/utils.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/OpenCHAIR/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"138d0bd7bf5f2513"}},{"code_sha256_prefix":"aa1b0f9eac04bb9d","entry":"local_coco_collate_fn","repo":"assafbk/mocha_code","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa1b0f9eac04bb9d"}},{"code_sha256_prefix":"cd4ab5612adb4997","entry":"unflatten_responses","repo":"assafbk/mocha_code","repo_kind":"official","path":"OpenCHAIR/evaluate.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/OpenCHAIR/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd4ab5612adb4997"}},{"code_sha256_prefix":"46f8dfafbbb8c9b2","entry":"calc_grad_norm","repo":"assafbk/mocha_code","repo_kind":"official","path":"utils.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46f8dfafbbb8c9b2"}},{"code_sha256_prefix":"91336ac6a4247b60","entry":"calc_mean_entropy","repo":"assafbk/mocha_code","repo_kind":"official","path":"utils.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91336ac6a4247b60"}},{"code_sha256_prefix":"22f204eff2c106a3","entry":"generate","repo":"assafbk/mocha_code","repo_kind":"official","path":"OpenCHAIR/generate_captions.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/OpenCHAIR/generate_captions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22f204eff2c106a3"}},{"code_sha256_prefix":"24f6f2d5b6395922","entry":"get_data_loaders","repo":"assafbk/mocha_code","repo_kind":"official","path":"vlm_rlhf.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/vlm_rlhf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24f6f2d5b6395922"}},{"code_sha256_prefix":"8c95b0551092a2f4","entry":"get_local_coco_data_loaders","repo":"assafbk/mocha_code","repo_kind":"official","path":"vlm_rlhf.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/vlm_rlhf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c95b0551092a2f4"}},{"code_sha256_prefix":"03a80eb50928e6df","entry":"get_url_data_loaders_coco","repo":"assafbk/mocha_code","repo_kind":"official","path":"vlm_rlhf.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/vlm_rlhf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03a80eb50928e6df"}},{"code_sha256_prefix":"258b6ae5a76ca81a","entry":"url_coco_collate_fn","repo":"assafbk/mocha_code","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/assafbk/mocha_code/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"258b6ae5a76ca81a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}