Papers › Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language

Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language

13 Oct 2020arXiv:2010.06447archive 2025-07-28

Dan John Velasco

Low-resource languages such as Filipino suffer from data scarcity which makes it challenging to develop NLP applications for Filipino language. The use of Transfer Learning (TL) techniques alleviates this problem in low-resource setting. In recent years, transformer-based models are proven to be effective in low-resource tasks but faces challenges in accessibility due to its high compute and memory requirements. For this reason, there's a need for a cheaper but effective alternative. This paper has three contributions. First, release a pre-trained AWD-LSTM language model for Filipino language. Second, benchmark AWD-LSTM in the Hate Speech classification task and show that it performs on par with transformer-based models. Third, analyze the the performance of AWD-LSTM in low-resource setting using degradation test and compare it with transformer-based models. ----- Ang mga low-resource languages tulad ng Filipino ay gipit sa accessible na datos kaya't mahirap gumawa ng mga applications sa wikang ito. Ang mga Transfer Learning (TL) techniques ay malaking tulong para sa low-resource setting o mga pagkakataong gipit sa datos. Sa mga nagdaang taon, nanaig ang mga transformer-based TL techniques pagdating sa low-resource tasks ngunit ito ay mataas na compute and memory requirements kaya nangangailangan ng mas mura pero epektibong alternatibo. Ang papel na ito ay may tatlong kontribusyon. Una, maglabas ng pre-trained AWD-LSTM language model sa wikang Filipino upang maging tuntungan sa pagbuo ng mga NLP applications sa wikang Filipino. Pangalawa, mag benchmark ng AWD-LSTM sa Hate Speech classification task at ipakita na kayang nitong makipagsabayan sa mga transformer-based models. Pangatlo, suriin ang performance ng AWD-LSTM sa low-resource setting gamit ang degradation test at ikumpara ito sa mga transformer-based models.

Paper PDF Code

Code

danjohnvelasco/Filipino-ULMFiT officialmentioned in papermentioned on GitHub report

jcblaisecruz02/Filipino-Text-Benchmarks mentioned on GitHubtf report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Language Modeling Language Modelling Transfer Learning

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

AWD-LSTM Activation Regularization Discriminative Fine-Tuning DropConnect Dropout Embedding Dropout LSTM Sigmoid Activation Slanted Triangular Learning Rates Tanh Activation Temporal Activation Regularization ULMFiT Variational Dropout Weight Tying

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections