ALTA 2021 Shared Task (Automatic Grading of Evidence, 10 years later)

This dataset is described in the ALTA 2021 Shared Task website and associated CodaLab competition.

The basic task is to build an automatic evidence grading system for evidence-based medicine. Evidence-based medicine is a medical practice which requires practitioners to search medical literature for evidence when making clinical decisions. The practitioners are also required to grade the quality of extracted evidence on some chosen scale. The goal of the grading system is to automatically determine the grade of an evidence given the article abstract(s) from which the evidence is extracted.

The grading scale used for this task is the Strength of Recommendation Taxonomy (SORT). This taxonomy has 3 grades - A (strong), B (moderate) and C (weak). The grade of an evidence depends on multiple factors and information about this grading scale can be found in the paper by Ebell et al. (2004).