{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measuring-semantic-similarity-of-clinical","title":"Measuring semantic similarity of clinical trial outcomes using deep pre-trained language representations","arxiv_id":null,"date":"2019-10-17","proceeding":"Journal of Biomedical Informatics: X 2019 10","authors":["Anna Korolevaa","Sanjay Kamatha","Patrick Paroubeka"],"abstract":"Background: Outcomes are variables monitored during a clinical trial to assess the impact of an intervention on\r\nhumans’ health.Automatic assessment of semantic similarity of trial outcomes is required for a number of tasks,\r\nsuch as detection of outcome switching (unjustified changes of pre-defined outcomes of a trial) and implementation of Core Outcome Sets (minimal sets of outcomes that should be reported in a particular medical\r\ndomain).\r\nObjective: We aimed at building an algorithm for assessing semantic similarity of pairs of primary and reported\r\noutcomes.We focused on approaches that do not require manually curated domain-specific resources such as\r\nontologies and thesauri.\r\nMethods: We tested several approaches, including single measures of similarity (based on strings, stems and\r\nlemmas, paths and distances in an ontology, and vector representations of phrases), classifiers using a combination of single measures as features, and a deep learning approach that consists in fine-tuning pre-trained deep\r\nlanguage representations.We tested language models provided by BERT (trained on general-domain texts),\r\nBioBERT and SciBERT (trained on biomedical and scientific texts, respectively).We explored the possibility of\r\nimproving the results by taking into account the variants for referring to an outcome (e.g.the use of a measurement tool name instead on the outcome name; the use of abbreviations).We release an open corpus with\r\nannotation for similarity of pairs of outcomes.\r\nResults: Classifiers using a combination of single measures as features outperformed the single measures, while\r\ndeep learning algorithms using BioBERT and SciBERT models outperformed the classifiers.BioBERT reached the\r\nbest F-measure of 89.75%.The addition of variants of outcomes did not improve the results for the best-performing single measures nor for the classifiers, but it improved the performance of deep learning algorithms:\r\nBioBERT achieved an F-measure of93.38%.\r\nConclusions: Deep learning approaches using pre-trained language representations outperformed other approaches for similarity assessment of trial outcomes, without relying on any manually curated domain-specific\r\nresources (ontologies and other lexical resources). Addition of variants of outcomes further improved the performance of deep learning algorithms.","url_abs":"https://www.sciencedirect.com/science/article/pii/S2590177X19300575","url_pdf":"https://www.sciencedirect.com/science/article/pii/S2590177X19300575/pdfft?md5=b75b2ad0e5e3f49f15b6052110fbe906&pid=1-s2.0-S2590177X19300575-main.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-4","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (expanded corpus)","model":"BioBERT (pre-trained on PubMed abstracts + PMC, fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, expanded corpus\")","rank_in_archive_order":1,"of":5,"metrics":{"F1":"93.38","Precision":"92.98","Recall":"93.85"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-4","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (expanded corpus)","model":"SciBERT uncased (SciVocab, fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, expanded corpus\")","rank_in_archive_order":2,"of":5,"metrics":{"F1":"91.51","Precision":"91.3","Recall":"91.79"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-4","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (expanded corpus)","model":"SciBERT cased (SciVocab, fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, expanded corpus\")","rank_in_archive_order":3,"of":5,"metrics":{"F1":"90.69","Precision":"89","Recall":"92.54"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-4","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (expanded corpus)","model":"BERT-Base uncased (fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, expanded corpus\")","rank_in_archive_order":4,"of":5,"metrics":{"F1":"89.16","Precision":"89.31","Recall":"89.12"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-4","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (expanded corpus)","model":"BERT-Base cased (fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, expanded corpus\")","rank_in_archive_order":5,"of":5,"metrics":{"F1":"89.12","Precision":"88.25","Recall":"90.1"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-3","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (original corpus)","model":"BioBERT\n(pre-trained on PubMed abstracts + PMC, fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, original corpus\")","rank_in_archive_order":1,"of":5,"metrics":{"F1":"89.75","Precision":"88.93","Recall":"90.76"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-3","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (original corpus)","model":"SciBERT uncased\n(SciVocab, fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, original corpus\")","rank_in_archive_order":2,"of":5,"metrics":{"F1":"89.3","Precision":"87.99","Recall":"90.78"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-3","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (original corpus)","model":"SciBERT cased\n(SciVocab, fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, original corpus\")","rank_in_archive_order":3,"of":5,"metrics":{"F1":"89.3","Precision":"87.31","Recall":"91.53"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-3","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (original corpus)","model":"BERT-Base uncased\n(fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, original corpus\")","rank_in_archive_order":4,"of":5,"metrics":{"F1":"86.8","Precision":"85.76","Recall":"88.15"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-embeddings-for-biomedical-texts-on-3","task":"Semantic Similarity","dataset":"Annotated corpus for semantic similarity of clinical trial outcomes (original corpus)","model":"BERT-Base cased\n(fine-tuned on \"Annotated corpus for semantic similarity of clinical trial outcomes, original corpus\")","rank_in_archive_order":5,"of":5,"metrics":{"F1":"84.21","Precision":"83.36","Recall":"85.2"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}