{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pevl-position-enhanced-pre-training-and","title":"PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models","arxiv_id":"2205.11169","date":"2022-05-23","proceeding":null,"authors":["Yuan YAO","Qianyu Chen","Ao Zhang","Wei Ji","Zhiyuan Liu","Tat-Seng Chua","Maosong Sun"],"abstract":"Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre-training, and also enables flexible prompt tuning for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL.","url_abs":"https://arxiv.org/abs/2205.11169v2","url_pdf":"https://arxiv.org/pdf/2205.11169v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pevl-position-enhanced-pre-training-and","repo_url":"https://github.com/thunlp/pevl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object","task_name":"Object"},{"task_slug":"phrase-grounding","task_name":"Phrase Grounding"},{"task_slug":null,"task_name":"Position"},{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"visual-commonsense-reasoning","task_name":"Visual Commonsense Reasoning"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-relationship-detection","task_name":"Visual Relationship Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/phrase-grounding-on-flickr30k-entities-dev","task":"Phrase Grounding","dataset":"Flickr30k Entities Dev","model":"PEVL","rank_in_archive_order":2,"of":3,"metrics":{"R@1":"84.1"},"uses_additional_data":false},{"leaderboard":"/sota/phrase-grounding-on-flickr30k-entities-test","task":"Phrase Grounding","dataset":"Flickr30k Entities Test","model":"PEVL","rank_in_archive_order":4,"of":18,"metrics":{"R@1":"84.4"},"uses_additional_data":true},{"leaderboard":"/sota/visual-commonsense-reasoning-on-vcr-q-a-dev","task":"Visual Commonsense Reasoning","dataset":"VCR (Q-A) dev","model":"PEVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"75.1"},"uses_additional_data":false},{"leaderboard":"/sota/visual-commonsense-reasoning-on-vcr-q-a-test","task":"Visual Commonsense Reasoning","dataset":"VCR (Q-A) test","model":"PEVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"76.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-commonsense-reasoning-on-vcr-q-ar-dev","task":"Visual Commonsense Reasoning","dataset":"VCR (Q-AR) dev","model":"PEVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"57.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-commonsense-reasoning-on-vcr-q-ar-test","task":"Visual Commonsense Reasoning","dataset":"VCR (Q-AR) test","model":"PEVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"58.6"},"uses_additional_data":false},{"leaderboard":"/sota/visual-commonsense-reasoning-on-vcr-qa-r-dev","task":"Visual Commonsense Reasoning","dataset":"VCR (QA-R) dev","model":"PEVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"76.4"},"uses_additional_data":false},{"leaderboard":"/sota/visual-commonsense-reasoning-on-vcr-qa-r-test","task":"Visual Commonsense Reasoning","dataset":"VCR (QA-R) test","model":"PEVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"76.7"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-gqa","task":"Visual Question Answering (VQA)","dataset":"GQA","model":"PEVL+","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"77"},"uses_additional_data":false},{"leaderboard":"/sota/visual-relationship-detection-on-visual","task":"Visual Relationship Detection","dataset":"Visual Genome","model":"PEVL","rank_in_archive_order":1,"of":1,"metrics":{"R@100":"66.3","R@50":"64.4","mR@100":"23.5","mR@50":"21.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.11169","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.11169"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/pevl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/thunlp/PEVL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"thunlp/PEVL","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/thunlp/PEVL/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"thunlp/PEVL","repo_kind":"official","path":"models/vit.py","file_url":"https://github.com/thunlp/PEVL/blob/HEAD/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"e7fbc7a74a3457c7","entry":"load_vocab","repo":"thunlp/PEVL","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/thunlp/PEVL/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7fbc7a74a3457c7"}},{"code_sha256_prefix":"92bb6d42cf25a3c2","entry":"computeIoU","repo":"thunlp/PEVL","repo_kind":"official","path":"run_grounding_train.py","file_url":"https://github.com/thunlp/PEVL/blob/HEAD/run_grounding_train.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92bb6d42cf25a3c2"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"thunlp/PEVL","repo_kind":"official","path":"models/model_grounding.py","file_url":"https://github.com/thunlp/PEVL/blob/HEAD/models/model_grounding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"181ef24861d6a70a","entry":"get_mask_posnum_prob","repo":"thunlp/PEVL","repo_kind":"official","path":"models/model_vcr.py","file_url":"https://github.com/thunlp/PEVL/blob/HEAD/models/model_vcr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"181ef24861d6a70a"}},{"code_sha256_prefix":"26be70dca3249c0b","entry":"load_tf_weights_in_bert","repo":"thunlp/PEVL","repo_kind":"official","path":"models/xbert.py","file_url":"https://github.com/thunlp/PEVL/blob/HEAD/models/xbert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26be70dca3249c0b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}