{"url":"/dataset/recam","name":"ReCAM","full_name":"SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning","description_markdown":"Tasks\r\nOur shared task has three subtasks. Subtask 1 and 2 focus on evaluating machine learning models' performance with regard to two definitions of abstractness (Spreen and Schulz, 1966; Changizi, 2008), which we call imperceptibility and nonspecificity, respectively. Subtask 3 aims to provide some insights to their relationships.\r\n\r\n• Subtask 1: ReCAM-Imperceptibility\r\n\r\nConcrete words refer to things, events, and properties that we can perceive directly with our senses (Spreen and Schulz, 1966; Coltheart 1981; Turney et al., 2011), e.g., donut, trees, and red. In contrast, abstract words refer to ideas and concepts that are distant from immediate perception. Examples include objective, culture, and economy. In subtask 1, the participanting systems are required to perform reading comprehension of abstract meaning for imperceptible concepts.\r\n\r\nBelow is an example. Given a passage and a question, your model needs to choose from the five candidates the best one for replacing @placeholder.\r\n\r\n\r\n\r\n \r\n\r\n• Subtask 2: ReCAM-Nonspecificity\r\n \r\nSubtask 2 focuses on a different type of definition. Compared to concrete concepts like groundhog and whale, hypernyms such as vertebrate are regarded as more abstract (Changizi, 2008). \r\n \r\n• Subtask 3: ReCAM-Intersection\r\nSubtask 3 aims to provide more insights to the relationship of the two views on abstractness, In this subtask, we test the performance of a system that is trained on one definition and evaluted on the other.","description_withheld":null,"homepage":"https://competitions.codalab.org/competitions/26153#learn_the_details-overview","introduced_date":"2021-05-31","introduced_date_note":null,"introduced_by":{"paper":"/paper/semeval-2021-task-4-reading-comprehension-of","title":"SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning","first_author":"Boyuan Zheng","url":null},"license":null,"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Reading Comprehension","url":"/task/reading-comprehension","datasets_with_task":"/datasets/task/reading-comprehension"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["ReCAM"],"data_loaders":[],"num_papers_in_archive":11,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/reading-comprehension-on-recam","task":"Reading Comprehension","dataset_variant":"ReCAM","rows":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"NAL","paper":"/paper/zjuklab-at-semeval-2021-task-4-negative","metrics":{"Accuracy":"87.9/92.8"},"code_links":[{"title":"zjunlp/SemEval2021Task4","url":"https://github.com/zjunlp/SemEval2021Task4"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/zjuklab-at-semeval-2021-task-4-negative","title":"ZJUKLAB at SemEval-2021 Task 4: Negative Augmentation with Language Model for Reading Comprehension of Abstract Meaning","date":"2021-02-25","rows_on_this_dataset":1,"code_links":1,"syntology":null}],"syntology_totals":{"read_at":"2026-09-25T09:33:49+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}