{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/controllable-dense-captioner-with-multimodal","title":"ControlCap: Controllable Region-level Captioning","arxiv_id":"2401.17910","date":"2024-01-31","proceeding":null,"authors":["Yuzhong Zhao","Yue Liu","Zonghao Guo","Weijia Wu","Chen Gong","Fang Wan","Qixiang Ye"],"abstract":"Region-level captioning is challenged by the caption degeneration issue, which refers to that pre-trained multimodal models tend to predict the most frequent captions but miss the less frequent ones. In this study, we propose a controllable region-level captioning (ControlCap) approach, which introduces control words to a multimodal model to address the caption degeneration issue. In specific, ControlCap leverages a discriminative module to generate control words within the caption space to partition it to multiple sub-spaces. The multimodal model is constrained to generate captions within a few sub-spaces containing the control words, which increases the opportunity of hitting less frequent captions, alleviating the caption degeneration issue. Furthermore, interactive control words can be given by either a human or an expert model, which enables captioning beyond the training caption space, enhancing the model's generalization ability. Extensive experiments on Visual Genome and RefCOCOg datasets show that ControlCap respectively improves the CIDEr score by 21.6 and 2.2, outperforming the state-of-the-arts by significant margins. Code is available at https://github.com/callsys/ControlCap.","url_abs":"https://arxiv.org/abs/2401.17910v3","url_pdf":"https://arxiv.org/pdf/2401.17910v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"controllable-dense-captioner-with-multimodal","repo_url":"https://github.com/callsys/controlcap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dense-captioning","task_name":"Dense Captioning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/dense-captioning-on-visual-genome","task":"Dense Captioning","dataset":"Visual Genome","model":"ControlCap","rank_in_archive_order":1,"of":4,"metrics":{"mAP":"18.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.17910","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.17910"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/callsys/controlcap","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/callsys/ControlCap","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"c0f3081d9053373f","entry":"merge_boxes","repo":"callsys/ControlCap","repo_kind":"official","path":"controlcap/common/evaluation/eval_densecap.py","file_url":"https://github.com/callsys/ControlCap/blob/HEAD/controlcap/common/evaluation/eval_densecap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0f3081d9053373f"}},{"code_sha256_prefix":"71ea247422847eea","entry":"pluck_boxes","repo":"callsys/ControlCap","repo_kind":"official","path":"controlcap/common/evaluation/eval_densecap.py","file_url":"https://github.com/callsys/ControlCap/blob/HEAD/controlcap/common/evaluation/eval_densecap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71ea247422847eea"}},{"code_sha256_prefix":"91176208680551c3","entry":"show_mask","repo":"callsys/ControlCap","repo_kind":"official","path":"demo/controlcap_inference.py","file_url":"https://github.com/callsys/ControlCap/blob/HEAD/demo/controlcap_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91176208680551c3"}},{"code_sha256_prefix":"9da754fa690dde4c","entry":"load_dataset_config","repo":"callsys/ControlCap","repo_kind":"official","path":"controlcap/datasets/builder.py","file_url":"https://github.com/callsys/ControlCap/blob/HEAD/controlcap/datasets/builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9da754fa690dde4c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}