{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/injecting-semantic-concepts-into-end-to-end","title":"Injecting Semantic Concepts into End-to-End Image Captioning","arxiv_id":"2112.05230","date":"2021-12-09","proceeding":"CVPR 2022 1","authors":["Zhiyuan Fang","JianFeng Wang","Xiaowei Hu","Lin Liang","Zhe Gan","Lijuan Wang","Yezhou Yang","Zicheng Liu"],"abstract":"Tremendous progress has been made in recent years in developing better image captioning models, yet most of them rely on a separate object detector to extract regional features. Recent vision-language studies are shifting towards the detector-free trend by leveraging grid representations for more flexible model training and faster inference speed. However, such development is primarily focused on image understanding tasks, and remains less investigated for the caption generation task. In this paper, we are concerned with a better-performing detector-free image captioning model, and propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features. For improved performance, we introduce a novel Concept Token Network (CTN) to predict the semantic concepts and then incorporate them into the end-to-end captioning. In particular, the CTN is built on the basis of a vision transformer and is designed to predict the concept tokens through a classification task, from which the rich semantic information contained greatly benefits the captioning task. Compared with the previous detector-based models, ViTCAP drastically simplifies the architectures and at the same time achieves competitive performance on various challenging image captioning datasets. In particular, ViTCAP reaches 138.1 CIDEr scores on COCO-caption Karpathy-split, 93.8 and 108.6 CIDEr scores on nocaps, and Google-CC captioning datasets, respectively.","url_abs":"https://arxiv.org/abs/2112.05230v2","url_pdf":"https://arxiv.org/pdf/2112.05230v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"injecting-semantic-concepts-into-end-to-end","repo_url":"https://github.com/jacobswan1/ViTCAP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"caption-generation","task_name":"Caption Generation"},{"task_slug":"image-captioning","task_name":"Image Captioning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2112.05230","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.05230"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jacobswan1/ViTCAP","reach":null}],"summary":{"ran_draft_wrong":1,"ran_fixture":3},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"086f695727aa10b7","entry":"checkpoint_filter_fn","repo":"jacobswan1/ViTCAP","repo_kind":"official","path":"src/pytorch_image_models/timm/models/vision_transformer.py","file_url":"https://github.com/jacobswan1/ViTCAP/blob/HEAD/src/pytorch_image_models/timm/models/vision_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"086f695727aa10b7"}},{"code_sha256_prefix":"2f1fd31e7f8b7357","entry":"convert_prediction_results_base","repo":"jacobswan1/ViTCAP","repo_kind":"official","path":"src/tools/captioning/captioning_e2e.py","file_url":"https://github.com/jacobswan1/ViTCAP/blob/HEAD/src/tools/captioning/captioning_e2e.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f1fd31e7f8b7357"}},{"code_sha256_prefix":"2cea4d41ba9d3781","entry":"create_image_features","repo":"jacobswan1/ViTCAP","repo_kind":"official","path":"src/tools/captioning/captioning_e2e.py","file_url":"https://github.com/jacobswan1/ViTCAP/blob/HEAD/src/tools/captioning/captioning_e2e.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cea4d41ba9d3781"}},{"code_sha256_prefix":"9fb05704643a3963","entry":"resize_pos_embed","repo":"jacobswan1/ViTCAP","repo_kind":"official","path":"src/pytorch_image_models/timm/models/vision_transformer.py","file_url":"https://github.com/jacobswan1/ViTCAP/blob/HEAD/src/pytorch_image_models/timm/models/vision_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fb05704643a3963"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}