{"url":"/dataset/civil-comments","name":"Civil Comments","full_name":"Jigsaw Unintended Bias in Toxicity Classification","description_markdown":"At the end of 2017 the Civil Comments platform shut down and chose make their ~2m public comments from their platform available in a lasting open archive so that researchers could understand and improve civility in online conversations for years to come. Jigsaw sponsored this effort and extended annotation of this data by human raters for various toxic conversational attributes.\r\n\r\nIn the data supplied for this competition, the text of the individual comment is found in the comment_text column. Each comment in Train has a toxicity label (target), and models should predict the target toxicity for the Test data. This attribute (and all others) are fractional values which represent the fraction of human raters who believed the attribute applied to the given comment.\r\n\r\nThe data also has several additional toxicity subtype attributes. Models do not need to predict these attributes for the competition, they are included as an additional avenue for research. Subtype attributes are:\r\n\r\n* `severe_toxicity`\r\n* `obscene`\r\n* `threat`\r\n* `insult`\r\n* `identity_attack`\r\n* `sexual_explicit`\r\n\r\nAdditionally, a subset of comments have been labelled with a variety of identity attributes, representing the identities that are mentioned in the comment. The columns corresponding to identity attributes are listed below. Only identities with more than 500 examples in the test set (combined public and private) will be included in the evaluation calculation. These identities are shown in bold.\r\n\r\n* `male`\r\n* `female`\r\n* `transgender`\r\n* `other_gender`\r\n* `heterosexual`\r\n* `homosexual_gay_or_lesbian`\r\n* `bisexual`\r\n* `other_sexual_orientation`\r\n* `christian`\r\n* `jewish`\r\n* `muslim`\r\n* `hindu`\r\n* `buddhist`\r\n* `atheist`\r\n* `other_religion`\r\n* `black`\r\n* `white`\r\n* `asian`\r\n* `latino`\r\n* `other_race_or_ethnicity`\r\n* `physical_disability`\r\n* `intellectual_or_learning_disability`\r\n* `psychiatric_or_mental_illness`\r\n* `other_disability`","description_withheld":null,"homepage":"https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification","introduced_date":"2019-03-11","introduced_date_note":null,"introduced_by":{"paper":"/paper/nuanced-metrics-for-measuring-unintended-bias","title":"Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification","first_author":"Daniel Borkan","url":null},"license":{"name":"Public domain (CC0)","url":null},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Hate Speech Detection","url":"/task/hate-speech-detection","datasets_with_task":"/datasets/task/hate-speech-detection"},{"name":"Toxic Comment Classification","url":"/task/toxic-comment-classification","datasets_with_task":"/datasets/task/toxic-comment-classification"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["Civil Comments"],"data_loaders":[{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/Bourdin/test3","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/1-800-SHARED-TASKS/civil_comments_Safety","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/google/civil_comments","frameworks":["tf","pytorch","jax"]}],"num_papers_in_archive":156,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/toxic-comment-classification-on-civil","task":"Toxic Comment Classification","dataset_variant":"Civil Comments","rows":22,"metrics":["GMB BPSN","GMB Subgroup","Micro F1","Precision","Recall","Macro F1","AUROC","GMB BNSP"],"first_row_in_archive_order":{"model":"RoBERTa Focal Loss","paper":"/paper/a-benchmark-for-toxic-comment-classification","metrics":{"AUROC":"0.9818","GMB BNSP":"0.9581","GMB BPSN":"0.901","GMB Subgroup":"0.8807","Macro F1":"0.4648","Micro F1":"0.5524","Precision":"0.4017","Recall":"0.8839"},"code_links":[{"title":"Nigiva/hatespeech-detection-models","url":"https://github.com/Nigiva/hatespeech-detection-models"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/pytorch-frame-a-modular-framework-for-multi","title":"PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learning","date":"2024-03-31","rows_on_this_dataset":6,"code_links":1,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":2,"samples_ran":2,"samples_unverified":0,"pointer_only_for_licence":0,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/palm-2-technical-report-1","title":"PaLM 2 Technical Report","date":"2023-05-17","rows_on_this_dataset":2,"code_links":1,"syntology":null},{"paper":"/paper/a-benchmark-for-toxic-comment-classification","title":"A benchmark for toxic comment classification on Civil Comments dataset","date":"2023-01-26","rows_on_this_dataset":14,"code_links":1,"syntology":null}],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":1,"samples_harvested":2,"samples_ran":2,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}