{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/end-to-end-supermask-pruning-learning-to","title":"End-to-End Supermask Pruning: Learning to Prune Image Captioning Models","arxiv_id":"2110.03298","date":"2021-10-07","proceeding":null,"authors":["Jia Huei Tan","Chee Seng Chan","Joon Huang Chuah"],"abstract":"With the advancement of deep models, research work on image captioning has led to a remarkable gain in raw performance over the last decade, along with increasing model complexity and computational cost. However, surprisingly works on compression of deep networks for image captioning task has received little to no attention. For the first time in image captioning research, we provide an extensive comparison of various unstructured weight pruning methods on three different popular image captioning architectures, namely Soft-Attention, Up-Down and Object Relation Transformer. Following this, we propose a novel end-to-end weight pruning method that performs gradual sparsification based on weight sensitivity to the training loss. The pruning schemes are then extended with encoder pruning, where we show that conducting both decoder pruning and training simultaneously prior to the encoder pruning provides good overall performance. Empirically, we show that an 80% to 95% sparse network (up to 75% reduction in model size) can either match or outperform its dense counterpart. The code and pre-trained models for Up-Down and Object Relation Transformer that are capable of achieving CIDEr scores >120 on the MS-COCO dataset but with only 8.7 MB and 14.5 MB in model size (size reduction of 96% and 94% respectively against dense versions) are publicly available at https://github.com/jiahuei/sparse-image-captioning.","url_abs":"https://arxiv.org/abs/2110.03298v1","url_pdf":"https://arxiv.org/pdf/2110.03298v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"end-to-end-supermask-pruning-learning-to","repo_url":"https://github.com/jiahuei/sparse-image-captioning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-captioning","task_name":"Image Captioning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.03298","atlas_url":"https://app.syntology.ai/?focus=2110.03298","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.03298"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiahuei/sparse-image-captioning","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":3,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1535ade78e8d6274","entry":"clones","repo":"jiahuei/sparse-image-captioning","repo_kind":"official","path":"sparse_caption/utils/model_utils.py","file_url":"https://github.com/jiahuei/sparse-image-captioning/blob/HEAD/sparse_caption/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1535ade78e8d6274"}},{"code_sha256_prefix":"51e50297a7cc524a","entry":"register_tokenizer","repo":"jiahuei/sparse-image-captioning","repo_kind":"official","path":"sparse_caption/tokenizer.py","file_url":"https://github.com/jiahuei/sparse-image-captioning/blob/HEAD/sparse_caption/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"51e50297a7cc524a"}},{"code_sha256_prefix":"9620631b06fe8585","entry":"repeat_tensors","repo":"jiahuei/sparse-image-captioning","repo_kind":"official","path":"sparse_caption/utils/model_utils.py","file_url":"https://github.com/jiahuei/sparse-image-captioning/blob/HEAD/sparse_caption/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9620631b06fe8585"}},{"code_sha256_prefix":"936178513bc7b9d7","entry":"split_tensors","repo":"jiahuei/sparse-image-captioning","repo_kind":"official","path":"sparse_caption/utils/model_utils.py","file_url":"https://github.com/jiahuei/sparse-image-captioning/blob/HEAD/sparse_caption/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"936178513bc7b9d7"}},{"code_sha256_prefix":"4943ef90db98a6cc","entry":"get_tokenizer","repo":"jiahuei/sparse-image-captioning","repo_kind":"official","path":"sparse_caption/tokenizer.py","file_url":"https://github.com/jiahuei/sparse-image-captioning/blob/HEAD/sparse_caption/tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4943ef90db98a6cc"}},{"code_sha256_prefix":"b3a5bb38ed85f23d","entry":"parse_opt","repo":"jiahuei/sparse-image-captioning","repo_kind":"official","path":"sparse_caption/opts.py","file_url":"https://github.com/jiahuei/sparse-image-captioning/blob/HEAD/sparse_caption/opts.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b3a5bb38ed85f23d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}