Spectral-masked bidirectional fusion transformer for hyperspectral image classification

Abstract

Hyperspectral image (HSI) classification requires robust learning from high-dimensional spectral signatures and spatial context, but remains challenging due to inter-band redundancy, limited labeled samples, class imbalance, and spatial heterogeneity. This paper proposes the Spectral-Masked Bidirectional Fusion Transformer (SMBFT), a dual-stream architecture for spatio-spectral representation learning. SMBFT combines a lightweight 3D convolutional frontend with transformer-based global modeling and constructs parallel spectral and spatial token streams to preserve modality-specific information. A bidirectional fusion module with symmetric cross-attention enables mutual interaction between the two streams, while a learnable gate regulates spatial-to-spectral information transfer to reduce over-fusion. In addition, a training-only band-masked spectral reconstruction objective is introduced as a tunable regularizer for improving spectral representation learning under limited supervision. Experiments on Houston 2013, Pavia University, Salinas, and Indian Pines show that SMBFT achieves competitive accuracy under the conventional random pixel protocol. To address spatial leakage concerns in patch-based HSI evaluation, we further quantify train-test patch overlap and report a buffer-constrained Houston 2013 evaluation with zero train-test patch overlap, where SMBFT maintains strong performance. Additional shared-configuration, ablation, and efficiency analyses indicate that the proposed components provide complementary gains while preserving low computational cost.