{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/grit-vlp-grouped-mini-batch-sampling-for","title":"GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training","arxiv_id":"2208.04060","date":"2022-08-08","proceeding":null,"authors":["Jaeseok Byun","Taebaek Hwang","Jianlong Fu","Taesup Moon"],"abstract":"Most of the currently existing vision and language pre-training (VLP) methods have mainly focused on how to extract and align vision and text features. In contrast to the mainstream VLP methods, we highlight that two routinely applied steps during pre-training have crucial impact on the performance of the pre-trained model: in-batch hard negative sampling for image-text matching (ITM) and assigning the large masking probability for the masked language modeling (MLM). After empirically showing the unexpected effectiveness of above two steps, we systematically devise our GRIT-VLP, which adaptively samples mini-batches for more effective mining of hard negative samples for ITM while maintaining the computational cost for pre-training. Our method consists of three components: 1) GRouped mIni-baTch sampling (GRIT) strategy that collects similar examples in a mini-batch, 2) ITC consistency loss for improving the mining ability, and 3) enlarged masking probability for MLM. Consequently, we show our GRIT-VLP achieves a new state-of-the-art performance on various downstream tasks with much less computational cost. Furthermore, we demonstrate that our model is essentially in par with ALBEF, the previous state-of-the-art, only with one-third of training epochs on the same training data. Code is available at https://github.com/jaeseokbyun/GRIT-VLP.","url_abs":"https://arxiv.org/abs/2208.04060v1","url_pdf":"https://arxiv.org/pdf/2208.04060v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"grit-vlp-grouped-mini-batch-sampling-for","repo_url":"https://github.com/jaeseokbyun/grit-vlp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"text-matching","task_name":"Text Matching"}],"methods":[{"method_slug":"albef","method_name":"ALBEF"},{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.04060","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.04060"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jaeseokbyun/grit-vlp","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/jaeseokbyun/GRIT-VLP","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"cb3b9a90beb432a1","entry":"chunks","repo":"jaeseokbyun/GRIT-VLP","repo_kind":"official","path":"models/GRIT_utils.py","file_url":"https://github.com/jaeseokbyun/GRIT-VLP/blob/HEAD/models/GRIT_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb3b9a90beb432a1"}},{"code_sha256_prefix":"34a461af62bc20d6","entry":"mini_batch_level_shuffle","repo":"jaeseokbyun/GRIT-VLP","repo_kind":"official","path":"models/GRIT_utils.py","file_url":"https://github.com/jaeseokbyun/GRIT-VLP/blob/HEAD/models/GRIT_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34a461af62bc20d6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}