Papers › LlamBERT: Large-scale low-cost data annotation in NLP

LlamBERT: Large-scale low-cost data annotation in NLP

23 Mar 2024arXiv:2403.15938archive 2025-07-28

Bálint Csanády, Lajos Muzsai, Péter Vedres, Zoltán Nádasdy, András Lukács

Large Language Models (LLMs), such as GPT-4 and Llama 2, show remarkable proficiency in a wide range of natural language processing (NLP) tasks. Despite their effectiveness, the high costs associated with their use pose a challenge. We present LlamBERT, a hybrid approach that leverages LLMs to annotate a small subset of large, unlabeled databases and uses the results for fine-tuning transformer encoders like BERT and RoBERTa. This strategy is evaluated on two diverse datasets: the IMDb review dataset and the UMLS Meta-Thesaurus. Our results indicate that the LlamBERT approach slightly compromises on accuracy while offering much greater cost-effectiveness.

PaperPDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

aielte-research/llambert officialmentioned in paperSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Sentiment AnalysisText Classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Sentiment Analysis IMDb RoBERTa-large with LlamBERT Accuracy 96.68 #1 of 49 Archive leaderboard report
Sentiment Analysis IMDb RoBERTa-large Accuracy 96.54 #2 of 49 Archive leaderboard report
Sentiment Analysis IMDb Llama-2-70b-chat (0-shot) Accuracy 95.39 #17 of 49 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionAttention DropoutBERTBPEDense ConnectionsDropoutGPT-4LLaMALabel SmoothingLayer NormalizationLinear LayerLinear Warmup With Linear DecayMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionRoBERTaSoftmaxTransformerWeight DecayWordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections