Papers › Inferring the source of official texts: can SVM beat ULMFiT?
Inferring the source of official texts: can SVM beat ULMFiT?
Pedro Henrique Luz de Araujo, Teófilo Emidio de Campos, Marcelo Magalhães Silva de Sousa
Official Gazettes are a rich source of relevant information to the public. Their careful examination may lead to the detection of frauds and irregularities that may prevent mismanagement of public funds. This paper presents a dataset composed of documents from the Official Gazette of the Federal District, containing both samples with document source annotation and unlabeled ones. We train, evaluate and compare a transfer learning based model that uses ULMFiT with traditional bag-of-words models that use SVM and Naive Bayes as classifiers. We find the SVM to be competitive, its performance being marginally worse than the ULMFiT while having much faster train and inference time and being less computationally expensive. Finally, we conduct ablation analysis to assess the performance impact of the ULMFiT parts.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Text Classification | DODF Data | ULMFiT (pre-trained vocab, no gradual unfreezing) | Average F1 | 0.8918 | #1 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | ULMFiT (pre-trained vocab, no gradual unfreezing) | Weighted F1 | 0.9257 | #1 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | SVM + word counts (pre-trained vocab) | Average F1 | 0.8782 | #2 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | SVM + word counts (pre-trained vocab) | Weighted F1 | 0.9049 | #2 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | SVM + tf-idf (no pre-trained vocab) | Average F1 | 0.8755 | #3 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | SVM + tf-idf (no pre-trained vocab) | Weighted F1 | 0.8917 | #3 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | ULMFiT (no pre-trained vocab) | Average F1 | 0.8469 | #4 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | ULMFiT (no pre-trained vocab) | Weighted F1 | 0.8974 | #4 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | ULMFiT (pre-trained vocab) | Average F1 | 0.8374 | #5 of 5 | Archive leaderboard | report |
| Text Classification | DODF Data | ULMFiT (pre-trained vocab) | Weighted F1 | 0.9088 | #5 of 5 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections