Papers › Multimodal Fusion via Teacher-Student Network for Indoor Action Recognition
Multimodal Fusion via Teacher-Student Network for Indoor Action Recognition
Bruce X.B. Yu, Yan Liu, Keith C.C. Chan
Indoor action recognition plays an important role in modern society, such as intelligent healthcare in large mobile cabin hospitals. With the wide usage of depth sensors like Kinect, multimodal information including skeleton and RGB modalities brings a promising way to improve the performance. However, existing methods are either focusing on a single data modality or failed to take the advantage of multiple data modalities. In this paper, we propose a Teacher-Student Multimodal Fusion (TSMF) model that fuses the skeleton and RGB modalities at the model level for indoor action recognition. In our TSMF, we utilize a teacher network to transfer the structural knowledge of the skeleton modality to a student network for the RGB modality. With extensive experiments on two benchmarking datasets: NTU RGB+D and PKU-MMD, results show that the proposed TSMF consistently performs better than state-of-the-art single modal and multimodal methods. It also indicates that our TSMF could not only improve the accuracy of the student network but also significantly improve the ensemble accuracy.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Action Recognition | NTU RGB+D | TSMF (RGB + Pose) | Accuracy (CS) | 92.5 | #18 of 28 | Archive leaderboard | report |
| Action Recognition | NTU RGB+D | TSMF (RGB + Pose) | Accuracy (CV) | 97.4 | #18 of 28 | Archive leaderboard | report |
| Action Recognition In Videos | PKU-MMD | TSMF | X-Sub | 95.8 | #4 of 5 | Archive leaderboard | report |
| Action Recognition In Videos | PKU-MMD | TSMF | X-View | 97.8 | #4 of 5 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections