{"url":"/dataset/probability-words-nli","name":"Probability words NLI","full_name":"Natural language inference with words estimative of probability (WEP)","description_markdown":"This dataset tests the capabilities of language models to correctly capture the meaning of words denoting probabilities (WEP), e.g. words like \"probably\", \"maybe\", \"surely\", \"impossible\".\r\n\r\nWe used probabilitic soft logic to combine probabilistic statements expressed with WEP (WEP-Reasoning) and we also used the UNLI dataset (https://nlp.jhu.edu/unli/) to directly check whether models can detect the WEP matching human-annotated probabilities. The dataset can be used as natural langauge inference data (context, premise, label) or multiple choice question answering (context,valid_hypothesis, invalid_hypothesis).","description_withheld":null,"homepage":"https://huggingface.co/datasets/sileod/probability_words_nli","introduced_date":"2022-11-07","introduced_date_note":null,"introduced_by":{"paper":"/paper/probing-neural-language-models-for","title":"Probing neural language models for understanding of words of estimative probability","first_author":"Damien Sileo","url":null},"license":{"name":"Apache-2.0","url":null},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Natural Language Inference","url":"/task/natural-language-inference","datasets_with_task":"/datasets/task/natural-language-inference"},{"name":"Multiple-choice","url":"/task/multiple-choice","datasets_with_task":"/datasets/task/multiple-choice"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["Probability words NLI"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/natural-language-inference-on-probability","task":"Natural Language Inference","dataset_variant":"Probability words NLI","rows":1,"metrics":["1:1 Accuracy"],"first_row_in_archive_order":{"model":"roberta-base-mnli","paper":"/paper/probing-neural-language-models-for","metrics":{"1:1 Accuracy":"48.5"},"code_links":[]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/probing-neural-language-models-for","title":"Probing neural language models for understanding of words of estimative probability","date":"2022-11-07","rows_on_this_dataset":1,"code_links":0,"syntology":null}],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}