{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/passage-re-ranking-with-bert","title":"Passage Re-ranking with BERT","arxiv_id":"1901.04085","date":"2019-01-13","proceeding":null,"authors":["Rodrigo Nogueira","Kyunghyun Cho"],"abstract":"Recently, neural models pretrained on a language modeling task, such as ELMo (Peters et al., 2017), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et al., 2018), have achieved impressive results on various natural language processing tasks such as question-answering and natural language inference. In this paper, we describe a simple re-implementation of BERT for query-based passage re-ranking. Our system is the state of the art on the TREC-CAR dataset and the top entry in the leaderboard of the MS MARCO passage retrieval task, outperforming the previous state of the art by 27% (relative) in MRR@10. The code to reproduce our results is available at https://github.com/nyu-dl/dl4marco-bert","url_abs":"https://arxiv.org/abs/1901.04085v5","url_pdf":"https://arxiv.org/pdf/1901.04085v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"passage-re-ranking-with-bert","repo_url":"https://github.com/nyu-dl/dl4marco-bert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"passage-re-ranking-with-bert","repo_url":"https://github.com/airKlizz/MsMarco","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"passage-re-ranking-with-bert","repo_url":"https://github.com/castorini/pygaggle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"passage-re-ranking-with-bert","repo_url":"https://github.com/lukevs/charity-explorer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"passage-re-ranking-with-bert","repo_url":"https://github.com/mrjleo/ranking-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"passage-re-ranking-with-bert","repo_url":"https://github.com/yang-ze-kang/PassageReranking","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"passage-re-ranking","task_name":"Passage Re-Ranking"},{"task_slug":"passage-retrieval","task_name":"Passage Retrieval"},{"task_slug":"re-ranking","task_name":"Re-Ranking"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"bilstm","method_name":"BiLSTM"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"elmo","method_name":"ELMo"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/passage-re-ranking-on-ms-marco","task":"Passage Re-Ranking","dataset":"MS MARCO","model":"BERT + Small Training","rank_in_archive_order":3,"of":4,"metrics":{"MRR":"0.359"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1901.04085","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.04085"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/castorini/pygaggle","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mrjleo/ranking-models","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yang-ze-kang/PassageReranking","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lukevs/charity-explorer","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nyu-dl/dl4marco-bert","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/airKlizz/MsMarco","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"unverified":11},"by_repo_kind":{"official":{"samples":13,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1923fc05163d207d","entry":"convert_to_unicode","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"tokenization.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/tokenization.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1923fc05163d207d"}},{"code_sha256_prefix":"5681f7dd9b6e3679","entry":"get_activation","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"modeling.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5681f7dd9b6e3679"}},{"code_sha256_prefix":"1e33e4465ad9760e","entry":"NDCG","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1e33e4465ad9760e"}},{"code_sha256_prefix":"45e8829b47e5a1b7","entry":"average_precision","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"45e8829b47e5a1b7"}},{"code_sha256_prefix":"895873eb21616a05","entry":"convert_to_bert_input","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"tokenization.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/tokenization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"895873eb21616a05"}},{"code_sha256_prefix":"ecab128238ebf253","entry":"gelu","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"modeling.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ecab128238ebf253"}},{"code_sha256_prefix":"50958618b65e514e","entry":"get_assignment_map_from_checkpoint","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"modeling.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"50958618b65e514e"}},{"code_sha256_prefix":"e010030ee57fe701","entry":"input_fn_builder","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"run_msmarco.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/run_msmarco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e010030ee57fe701"}},{"code_sha256_prefix":"9fbf56276f399498","entry":"input_fn_builder","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"run_treccar.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/run_treccar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9fbf56276f399498"}},{"code_sha256_prefix":"9eb250ad7f25c79e","entry":"load_qrels","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"convert_treccar_to_tfrecord.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/convert_treccar_to_tfrecord.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9eb250ad7f25c79e"}},{"code_sha256_prefix":"86c70a8984ca78b7","entry":"load_run","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"convert_treccar_to_tfrecord.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/convert_treccar_to_tfrecord.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"86c70a8984ca78b7"}},{"code_sha256_prefix":"54573d73f899d22f","entry":"metrics","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"54573d73f899d22f"}},{"code_sha256_prefix":"0e5615f8994003cf","entry":"printable_text","repo":"nyu-dl/dl4marco-bert","repo_kind":"official","path":"tokenization.py","file_url":"https://github.com/nyu-dl/dl4marco-bert/blob/HEAD/tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0e5615f8994003cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}