{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sbert-studies-meaning-representations","title":"SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features","arxiv_id":"2206.07023","date":"2022-06-14","proceeding":null,"authors":["Juri Opitz","Anette Frank"],"abstract":"Models based on large-pretrained language models, such as S(entence)BERT, provide effective and efficient sentence embeddings that show high correlation to human similarity ratings, but lack interpretability. On the other hand, graph metrics for graph-based meaning representations (e.g., Abstract Meaning Representation, AMR) can make explicit the semantic aspects in which two sentences are similar. However, such metrics tend to be slow, rely on parsers, and do not reach state-of-the-art performance when rating sentence similarity. In this work, we aim at the best of both worlds, by learning to induce $S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our S$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize various semantic sentence features (e.g., semantic roles, negation, or quantification). We show how to i) learn a decomposition of the sentence embeddings into semantic features, through approximation of a suite of interpretable AMR graph metrics, and how to ii) preserve the overall power of the neural embeddings by controlling the decomposition learning process with a second objective that enforces consistency with the similarity ratings of an SBERT teacher model. In our experimental studies, we show that our approach offers interpretability -- while fully preserving the effectiveness and efficiency of the neural sentence embeddings.","url_abs":"https://arxiv.org/abs/2206.07023v2","url_pdf":"https://arxiv.org/pdf/2206.07023v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sbert-studies-meaning-representations","repo_url":"https://github.com/flipz357/S3BERT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"abstract-meaning-representation","task_name":"Abstract Meaning Representation"},{"task_slug":"negation","task_name":"Negation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-embeddings","task_name":"Sentence Embeddings"},{"task_slug":"sentence-similarity","task_name":"Sentence Similarity"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sbert","method_name":"SBERT"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.07023","atlas_url":"https://app.syntology.ai/?focus=2206.07023","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.07023"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flipz357/S3BERT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b7d99c0d0c53c89d","entry":"cosine_sim","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/prediction_helpers.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/prediction_helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7d99c0d0c53c89d"}},{"code_sha256_prefix":"866cc73fc6c3e77e","entry":"dist_sim","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/custom_losses.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/custom_losses.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"866cc73fc6c3e77e"}},{"code_sha256_prefix":"e1930a781ca4e475","entry":"get_preds","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/prediction_helpers.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/prediction_helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1930a781ca4e475"}},{"code_sha256_prefix":"1779ed436ac80094","entry":"jload","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/data_helpers.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/data_helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1779ed436ac80094"}},{"code_sha256_prefix":"a2fb5897f0d97930","entry":"manhattan_sim","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/prediction_helpers.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/prediction_helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2fb5897f0d97930"}},{"code_sha256_prefix":"8a73b88b2ee8016a","entry":"prod_sim","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/custom_losses.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/custom_losses.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a73b88b2ee8016a"}},{"code_sha256_prefix":"9e1c02152506aa40","entry":"co_sim","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/custom_losses.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/custom_losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e1c02152506aa40"}},{"code_sha256_prefix":"b67a6e0d4b3fe113","entry":"freeze_all_layers","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/model_freeze.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/model_freeze.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b67a6e0d4b3fe113"}},{"code_sha256_prefix":"23fe518aded0d9fb","entry":"freeze_except_last_layers","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/model_freeze.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/model_freeze.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23fe518aded0d9fb"}},{"code_sha256_prefix":"f94769cc07de90c4","entry":"jsave","repo":"flipz357/S3BERT","repo_kind":"official","path":"src/data_helpers.py","file_url":"https://github.com/flipz357/S3BERT/blob/HEAD/src/data_helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f94769cc07de90c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}