Papers › pNLP-Mixer: an Efficient all-MLP Architecture for Language

pNLP-Mixer: an Efficient all-MLP Architecture for Language

9 Feb 2022arXiv:2202.04350archive 2025-07-28

Francesco Fusco, Damian Pascual, Peter Staar, Diego Antognini

Large pre-trained language models based on transformer architecture have drastically changed the natural language processing (NLP) landscape. However, deploying those models for on-device applications in constrained devices such as smart watches is completely impractical due to their size and inference cost. As an alternative to transformer-based architectures, recent work on efficient NLP has shown that weight-efficient models can attain competitive performance for simple tasks, such as slot filling and intent classification, with model sizes in the order of the megabyte. This work introduces the pNLP-Mixer architecture, an embedding-free MLP-Mixer model for on-device NLP that achieves high weight-efficiency thanks to a novel projection layer. We evaluate a pNLP-Mixer model of only one megabyte in size on two multi-lingual semantic parsing datasets, MTOP and multiATIS. Our quantized model achieves 99.4% and 97.8% the performance of mBERT on MTOP and multi-ATIS, while using 170x fewer parameters. Our model consistently beats the state-of-the-art of tiny models (pQRNN), which is twice as large, by a margin up to 7.8% on MTOP.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

mindslab-ai/pnlp-mixer mentioned on GitHubpytorchBSD-3-Clause report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

All Intent Classification Semantic Parsing Slot Filling intent-classification slot-filling

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Adam Attention Attention Dropout Average Pooling BERT Dense Connections Dropout Global Average Pooling Layer Normalization Linear Layer Linear Warmup With Linear Decay MLP-Mixer Multi-Head Attention Residual Connection RoBERTa Softmax Weight Decay WordPiece mBERT

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections