{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/abnirml-analyzing-the-behavior-of-neural-ir","title":"ABNIRML: Analyzing the Behavior of Neural IR Models","arxiv_id":"2011.00696","date":"2020-11-02","proceeding":null,"authors":["Sean MacAvaney","Sergey Feldman","Nazli Goharian","Doug Downey","Arman Cohan"],"abstract":"Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc search. However, it is not yet well-understood why these methods are so effective, what makes some variants more effective than others, and what pitfalls they may have. We present a new comprehensive framework for Analyzing the Behavior of Neural IR ModeLs (ABNIRML), which includes new types of diagnostic probes that allow us to test several characteristics -- such as writing styles, factuality, sensitivity to paraphrasing and word order -- that are not addressed by previous techniques. To demonstrate the value of the framework, we conduct an extensive empirical study that yields insights into the factors that contribute to the neural model's gains, and identify potential unintended biases the models exhibit. Some of our results confirm conventional wisdom, like that recent neural ranking models rely less on exact term overlap with the query, and instead leverage richer linguistic information, evidenced by their higher sensitivity to word and sentence order. Other results are more surprising, such as that some models (e.g., T5 and ColBERT) are biased towards factually correct (rather than simply relevant) texts. Further, some characteristics vary even for the same base language model, and other characteristics can appear due to random variations during model training.","url_abs":"https://arxiv.org/abs/2011.00696v2","url_pdf":"https://arxiv.org/pdf/2011.00696v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"abnirml-analyzing-the-behavior-of-neural-ir","repo_url":"https://github.com/allenai/abnirml","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"abnirml-analyzing-the-behavior-of-neural-ir","repo_url":"https://github.com/allenai/ir_datasets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sensitivity","task_name":"Sensitivity"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2011.00696","atlas_url":"https://app.syntology.ai/?focus=2011.00696","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2011.00696"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/ir_datasets","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/abnirml","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bb05ea380c1dec92","entry":"un_subbatch","repo":"allenai/abnirml","repo_kind":"official","path":"abnirml/scorers/vanilla_bert.py","file_url":"https://github.com/allenai/abnirml/blob/HEAD/abnirml/scorers/vanilla_bert.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb05ea380c1dec92"}},{"code_sha256_prefix":"106897b462cdf28a","entry":"binmat","repo":"allenai/abnirml","repo_kind":"official","path":"abnirml/scorers/conv_knrm.py","file_url":"https://github.com/allenai/abnirml/blob/HEAD/abnirml/scorers/conv_knrm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"106897b462cdf28a"}},{"code_sha256_prefix":"4a23efa271d03ccb","entry":"cos_simmat","repo":"allenai/abnirml","repo_kind":"official","path":"abnirml/scorers/conv_knrm.py","file_url":"https://github.com/allenai/abnirml/blob/HEAD/abnirml/scorers/conv_knrm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a23efa271d03ccb"}},{"code_sha256_prefix":"396065e8d8077ced","entry":"lens2mask","repo":"allenai/abnirml","repo_kind":"official","path":"abnirml/scorers/vanilla_bert.py","file_url":"https://github.com/allenai/abnirml/blob/HEAD/abnirml/scorers/vanilla_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"396065e8d8077ced"}},{"code_sha256_prefix":"320b1b24b3961a38","entry":"prepare_candidate","repo":"allenai/abnirml","repo_kind":"official","path":"abnirml/scorers/s2.py","file_url":"https://github.com/allenai/abnirml/blob/HEAD/abnirml/scorers/s2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"320b1b24b3961a38"}},{"code_sha256_prefix":"ed35fabf74131886","entry":"subbatch","repo":"allenai/abnirml","repo_kind":"official","path":"abnirml/scorers/vanilla_bert.py","file_url":"https://github.com/allenai/abnirml/blob/HEAD/abnirml/scorers/vanilla_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed35fabf74131886"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}