{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neighborhood-contrastive-learning-for-1","title":"Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings","arxiv_id":"2202.06671","date":"2022-02-14","proceeding":null,"authors":["Malte Ostendorff","Nils Rethmeier","Isabelle Augenstein","Bela Gipp","Georg Rehm"],"abstract":"Learning scientific document representations can be substantially improved through contrastive learning objectives, where the challenge lies in creating positive and negative training samples that encode the desired similarity semantics. Prior work relies on discrete citation relations to generate contrast samples. However, discrete citations enforce a hard cut-off to similarity. This is counter-intuitive to similarity-based learning, and ignores that scientific papers can be very similar despite lacking a direct citation - a core problem of finding related research. Instead, we use controlled nearest neighbor sampling over citation graph embeddings for contrastive learning. This control allows us to learn continuous similarity, to sample hard-to-learn negatives and positives, and also to avoid collisions between negative and positive samples by controlling the sampling margin between them. The resulting method SciNCL outperforms the state-of-the-art on the SciDocs benchmark. Furthermore, we demonstrate that it can train (or tune) models sample-efficiently, and that it can be combined with recent training-efficient methods. Perhaps surprisingly, even training a general-domain language model this way outperforms baselines pretrained in-domain.","url_abs":"https://arxiv.org/abs/2202.06671v2","url_pdf":"https://arxiv.org/pdf/2202.06671v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neighborhood-contrastive-learning-for-1","repo_url":"https://github.com/malteos/scincl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"citation-prediction","task_name":"Citation Prediction"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"document-classification","task_name":"Document Classification"},{"task_slug":"document-embedding","task_name":"Document Embedding"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/document-classification-on-scidocs-mag","task":"Document Classification","dataset":"SciDocs (MAG)","model":"SciNCL","rank_in_archive_order":2,"of":2,"metrics":{"F1 (micro)":"81.4"},"uses_additional_data":false},{"leaderboard":"/sota/document-classification-on-scidocs-mesh","task":"Document Classification","dataset":"SciDocs (MeSH)","model":"SciNCL","rank_in_archive_order":1,"of":2,"metrics":{"F1 (micro)":"88.7"},"uses_additional_data":false},{"leaderboard":"/sota/representation-learning-on-scidocs","task":"Representation Learning","dataset":"SciDocs","model":"SciNCL","rank_in_archive_order":1,"of":7,"metrics":{"Avg.":"81.8"},"uses_additional_data":false},{"leaderboard":"/sota/representation-learning-on-scidocs","task":"Representation Learning","dataset":"SciDocs","model":"Sci-DeCLUTR","rank_in_archive_order":4,"of":7,"metrics":{"Avg.":"66.6"},"uses_additional_data":false},{"leaderboard":"/sota/representation-learning-on-scidocs","task":"Representation Learning","dataset":"SciDocs","model":"CiteBERT","rank_in_archive_order":7,"of":7,"metrics":{"Avg.":"58.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.06671","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.06671"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/malteos/scincl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6fed79e4b6b544af","entry":"get_from_ids","repo":"malteos/scincl","repo_kind":"official","path":"s2_scraper.py","file_url":"https://github.com/malteos/scincl/blob/HEAD/s2_scraper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6fed79e4b6b544af"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}