{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/structure-aware-language-model-pretraining","title":"Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data","arxiv_id":"2305.19912","date":"2023-05-31","proceeding":null,"authors":["Xinze Li","Zhenghao Liu","Chenyan Xiong","Shi Yu","Yu Gu","Zhiyuan Liu","Ge Yu"],"abstract":"This paper presents Structure Aware Dense Retrieval (SANTA) model, which encodes user queries and structured data in one universal embedding space for retrieving structured data. SANTA proposes two pretraining methods to make language models structure-aware and learn effective representations for structured data: 1) Structured Data Alignment, which utilizes the natural alignment relations between structured data and unstructured data for structure-aware pretraining. It contrastively trains language models to represent multi-modal text data and teaches models to distinguish matched structured data for unstructured texts. 2) Masked Entity Prediction, which designs an entity-oriented mask strategy and asks language models to fill in the masked entities. Our experiments show that SANTA achieves state-of-the-art on code search and product search and conducts convincing results in the zero-shot setting. SANTA learns tailored representations for multi-modal text data by aligning structured and unstructured data pairs and capturing structural semantics by masking and predicting entities in the structured data. All codes are available at https://github.com/OpenMatch/OpenMatch.","url_abs":"https://arxiv.org/abs/2305.19912v1","url_pdf":"https://arxiv.org/pdf/2305.19912v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"structure-aware-language-model-pretraining","repo_url":"https://github.com/openmatch/openmatch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"structure-aware-language-model-pretraining","repo_url":"https://github.com/NEUIR/CONAN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"code-search","task_name":"Code Search"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.19912","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.19912"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NEUIR/CONAN","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openmatch/openmatch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b46a95cff9bf41b5","entry":"has_answers","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/qa_utils.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/qa_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b46a95cff9bf41b5"}},{"code_sha256_prefix":"222c1daa5a08da51","entry":"load_beir_positives","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/utils.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"222c1daa5a08da51"}},{"code_sha256_prefix":"8d2e5d58238cd53f","entry":"load_from_trec","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/utils.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d2e5d58238cd53f"}},{"code_sha256_prefix":"9e4c17e0bdb9dcd5","entry":"load_positives","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/utils.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e4c17e0bdb9dcd5"}},{"code_sha256_prefix":"93b0eef32604c3f7","entry":"regex_match","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/qa_utils.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/qa_utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93b0eef32604c3f7"}},{"code_sha256_prefix":"82420f6350858142","entry":"get_current_ckpt_step","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/driver/async_eval_dr.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/driver/async_eval_dr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82420f6350858142"}},{"code_sha256_prefix":"90f77daf63ac2f1f","entry":"get_idx","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/dataset/inference_dataset.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/dataset/inference_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90f77daf63ac2f1f"}},{"code_sha256_prefix":"3d25cb3600eef8dc","entry":"load_beir_qrels","repo":"openmatch/openmatch","repo_kind":"official","path":"src/openmatch/dataset/beir_dataset.py","file_url":"https://github.com/openmatch/openmatch/blob/HEAD/src/openmatch/dataset/beir_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d25cb3600eef8dc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}