Papers › MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection
MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection
Heitor R. Medeiros, David Latortue, Eric Granger, Marco Pedersoli
In real-world scenarios, using multiple modalities like visible (RGB) and infrared (IR) can greatly improve the performance of a predictive task such as object detection (OD). Multimodal learning is a common way to leverage these modalities, where multiple modality-specific encoders and a fusion module are used to improve performance. In this paper, we tackle a different way to employ RGB and IR modalities, where only one modality or the other is observed by a single shared vision encoder. This realistic setting requires a lower memory footprint and is more suitable for applications such as autonomous driving and surveillance, which commonly rely on RGB and IR data. However, when learning a single encoder on multiple modalities, one modality can dominate the other, producing uneven recognition results. This work investigates how to efficiently leverage RGB and IR modalities to train a common transformer-based OD vision encoder, while countering the effects of modality imbalance. For this, we introduce a novel training technique to Mix Patches (MiPa) from the two modalities, in conjunction with a patch-wise modality agnostic module, for learning a common representation of both modalities. Our experiments show that MiPa can learn a representation to reach competitive results on traditional RGB/IR benchmarks while only requiring a single modality during inference. Our code is available at: https://github.com/heitorrapela/MiPa.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Multispectral Object Detection | FLIR | MiPa | mAP | 44.8% | #7 of 18 | Archive leaderboard | report |
| Multispectral Object Detection | FLIR | MiPa | mAP50 | 81.3% | #7 of 18 | Archive leaderboard | report |
| Object Detection | FLIR | MiPa | AP 0.5 | 0.813 | #1 of 1 | Archive leaderboard | report |
| Object Detection | LLVIP | MiPa | AP | 0.665 | #1 of 1 | Archive leaderboard | report |
| Pedestrian Detection | LLVIP | MiPa | AP | 0.665 | #3 of 15 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections