Deep learning has transformed the analysis of physiological signals, but multimodal wearable modeling still faces some major challenges, namely in handling different sampling rates across modalities, exploiting cross–modal relationships, managing the computational and deployment cost of individual sensors, and generalizing across subjects under limited training data. We propose Leave–One–Out Multimodal Network (LOOM Net), an importance-aware CNN–Transformer for multimodal physiological signal processing, demonstrated on binary stress detection (stress vs. no stress). LOOM Net resamples modalities to a unified rate and applies early fusion to preserve short and long term cross–modal patterns, then uses a Transformer encoder with self-attention. We integrate Leave–One–Modality–Out analysis to quantify sensor value and guide deployment choices. We evaluate on the WESAD dataset, which contains multiple physiological modalities and provides ground truth labels. Under Leave–One–Subject-Out, LOOM Net shows accurate stress detection with short 4–8 sec windows, while Leave–One–Modality–Out reveals the relative contribution of each modality, highlighting respiration, electrocardiogram, and electrodermal activity as most influential modalities