{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/b-prop-bootstrapped-pre-training-with","title":"B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval","arxiv_id":"2104.09791","date":"2021-04-20","proceeding":null,"authors":["Xinyu Ma","Jiafeng Guo","Ruqing Zhang","Yixing Fan","Yingyan Li","Xueqi Cheng"],"abstract":"Pre-training and fine-tuning have achieved remarkable success in many downstream natural language processing (NLP) tasks. Recently, pre-training methods tailored for information retrieval (IR) have also been explored, and the latest success is the PROP method which has reached new SOTA on a variety of ad-hoc retrieval benchmarks. The basic idea of PROP is to construct the \\textit{representative words prediction} (ROP) task for pre-training inspired by the query likelihood model. Despite its exciting performance, the effectiveness of PROP might be bounded by the classical unigram language model adopted in the ROP task construction process. To tackle this problem, we propose a bootstrapped pre-training method (namely B-PROP) based on BERT for ad-hoc retrieval. The key idea is to use the powerful contextual language model BERT to replace the classical unigram language model for the ROP task construction, and re-train BERT itself towards the tailored objective for IR. Specifically, we introduce a novel contrastive method, inspired by the divergence-from-randomness idea, to leverage BERT's self-attention mechanism to sample representative words from the document. By further fine-tuning on downstream ad-hoc retrieval tasks, our method achieves significant improvements over baselines without pre-training or with other pre-training methods, and further pushes forward the SOTA on a variety of ad-hoc retrieval tasks.","url_abs":"https://arxiv.org/abs/2104.09791v4","url_pdf":"https://arxiv.org/pdf/2104.09791v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"b-prop-bootstrapped-pre-training-with","repo_url":"https://github.com/Albert-Ma/PROP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.09791","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.09791"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Albert-Ma/PROP","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran_honours":2,"unverified":11},"by_repo_kind":{"official":{"samples":16,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/modeling.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/tokenization.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/tokenization.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/modeling.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"5eff22fa0a651276","entry":"url_to_filename","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/file_utils.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5eff22fa0a651276"}},{"code_sha256_prefix":"e0960afdb0d64aa7","entry":"warmup_linear","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/optimization.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/optimization.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0960afdb0d64aa7"}},{"code_sha256_prefix":"cf9c448c96f48428","entry":"cached_path","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/file_utils.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9c448c96f48428"}},{"code_sha256_prefix":"31309d37109b349a","entry":"convert_example_to_features","repo":"Albert-Ma/PROP","repo_kind":"official","path":"run_pretraining.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/run_pretraining.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31309d37109b349a"}},{"code_sha256_prefix":"84da8d20deaf09f5","entry":"convert_examples_to_features","repo":"Albert-Ma/PROP","repo_kind":"official","path":"bprop/extract_bert_attention.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/bprop/extract_bert_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84da8d20deaf09f5"}},{"code_sha256_prefix":"bfc2b26a3e70c602","entry":"create_masked_lm_predictions","repo":"Albert-Ma/PROP","repo_kind":"official","path":"prop/multiprocessing_generate_pairwise_instances.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/prop/multiprocessing_generate_pairwise_instances.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfc2b26a3e70c602"}},{"code_sha256_prefix":"a6aab787fcf8b10d","entry":"filename_to_url","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/file_utils.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6aab787fcf8b10d"}},{"code_sha256_prefix":"b1a7180868dd8d6a","entry":"load_tf_weights_in_bert","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/modeling.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a7180868dd8d6a"}},{"code_sha256_prefix":"a4e183f24a5ad1a7","entry":"load_vocab","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/tokenization.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4e183f24a5ad1a7"}},{"code_sha256_prefix":"3fb4a496ddda4d00","entry":"read_examples","repo":"Albert-Ma/PROP","repo_kind":"official","path":"bprop/extract_bert_attention.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/bprop/extract_bert_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3fb4a496ddda4d00"}},{"code_sha256_prefix":"2871908e26a30647","entry":"softmax","repo":"Albert-Ma/PROP","repo_kind":"official","path":"prop/multiprocessing_generate_word_sets.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/prop/multiprocessing_generate_word_sets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2871908e26a30647"}},{"code_sha256_prefix":"59e4e730a9dc5dc6","entry":"warmup_constant","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/optimization.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/optimization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59e4e730a9dc5dc6"}},{"code_sha256_prefix":"007a8d2955fb555d","entry":"warmup_cosine","repo":"Albert-Ma/PROP","repo_kind":"official","path":"pytorch_pretrain_bert/optimization.py","file_url":"https://github.com/Albert-Ma/PROP/blob/HEAD/pytorch_pretrain_bert/optimization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"007a8d2955fb555d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}