Papers › Exploring Multi-Level Threats in Telegram Data with AI-Human Annotation: A Preliminary Study
Exploring Multi-Level Threats in Telegram Data with AI-Human Annotation: A Preliminary Study
Kamalakkannan Ravi, Adan Ernesto Vela, Elizabeth Jenaway, Steven Windisch
This research addresses the crucial challenge of effectively measuring threats in social media comments targeting voting, public officials, and institutions in the United States. Our understanding of these online threats and their links to real-world risks is limited, making it difficult to assess their seriousness. To overcome these limitations, we propose a comprehensive threat level scale from 0 to 5 and collect a dataset of 1.3 million Telegram responses for developing and rigorously testing these threat levels. Additionally, we explore OpenAI-human annotation to efficiently label this vast dataset. Our innovative two-step transfer learning approach initially employs a pre-existing, pre-trained model for labeling, followed by expert validation. Next, we use the AI-annotated samples to develop independent models, and expert annotators verify their predictions. Notably, our findings demonstrate that the GPT-2 model, despite its fewer annotated training set, performs comparably to OpenAI's anno-tations, showcasing its potential for cost-effective threat detection with more annotated samples. With the long-term objective of establishing continuous threat-level monitoring, we identify the strengths and limitations of our current approach and propose a roadmap for enhancing threat detection.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Text Classification | ThreatGram 101 - Extreme Telegram Data | GPT-2 | weighted-F1 score | 66.2 | #1 of 4 | Archive leaderboard | report |
| Text Classification | ThreatGram 101 - Extreme Telegram Data | SVM | weighted-F1 score | 64.3 | #2 of 4 | Archive leaderboard | report |
| Text Classification | ThreatGram 101 - Extreme Telegram Data | fastText | weighted-F1 score | 60.2 | #3 of 4 | Archive leaderboard | report |
| Text Classification | ThreatGram 101 - Extreme Telegram Data | ULMFit | weighted-F1 score | 55.7 | #4 of 4 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections