Papers › Multimodal Side-Tuning for Document Classification

Multimodal Side-Tuning for Document Classification

16 Jan 2023arXiv:2301.07502archive 2025-07-28

Stefano Pio Zingaro, Giuseppe Lisanti, Maurizio Gabbrielli

In this paper, we propose to exploit the side-tuning framework for multimodal document classification. Side-tuning is a methodology for network adaptation recently introduced to solve some of the problems related to previous approaches. Thanks to this technique it is actually possible to overcome model rigidity and catastrophic forgetting of transfer learning by fine-tuning. The proposed solution uses off-the-shelf deep learning architectures leveraging the side-tuning framework to combine a base model with a tandem of two side networks. We show that side-tuning can be successfully employed also when different data sources are considered, e.g. text and images in document classification. The experimental results show that this approach pushes further the limit for document classification accuracy with respect to the state of the art.

Paper PDF Code

Code

thezingaro/multimodal-side-tuning officialpytorchGPL-3.0 report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Classification Document Classification Document Image Classification Transfer Learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Document Image Classification	RVL-CDIP	Multimodal (ResNet50)	Accuracy	92.7%	#21 of 31	Archive leaderboard	report
Document Image Classification	RVL-CDIP	Multimodal (ResNet50)	Parameters	57M	#21 of 31	Archive leaderboard	report
Document Image Classification	RVL-CDIP	Multimodal (MobileNetV2)	Accuracy	92.2%	#25 of 31	Archive leaderboard	report
Document Image Classification	RVL-CDIP	Multimodal (MobileNetV2)	Parameters	12M	#25 of 31	Archive leaderboard	report
Document Image Classification	Tobacco-3482	Multimodal Side-Tuning (MobileNetV2)	Accuracy	90.50	#4 of 10	Archive leaderboard	report
Document Image Classification	Tobacco-3482	Multimodal Side-Tuning (ResNet50)	Accuracy	90.30	#5 of 10	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

1x1 Convolution Average Pooling Batch Normalization Bottleneck Residual Block Convolution Depthwise Convolution Depthwise Separable Convolution Global Average Pooling Inverted Residual Block Kaiming Initialization Max Pooling Pointwise Convolution ReLU Residual Block Residual Connection fastText

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections