Neonatal jaundice is a prevalent condition whose delayed diagnosis can lead to severe neurological complications. Conventional diagnostic approaches such as total serum bilirubin (TSB) testing are invasive and resource intensive, while the existing non-invasive approaches often suffer from limited generalization and interpretability. To address these challenges, this study proposes a lightweight bi-modal framework (BiNJD) trained via knowledge distillation for accurate, explainable, and efficient neonatal jaundice diagnosis. The approach integrates spatial visual representation with language-based clinical semantics. A Vision Mamba visual teacher and an LLM + CLIP-based textual teacher jointly transfer complementary knowledge to a lightweight ResNet-50 student through multi-level cross-modal knowledge distillation, enabling rich feature learning without increased inference complexity. Model explainability is achieved using Grad-CAM visualizations and vision-language alignment to reinforce clinically meaningful reasoning. The proposed approach achieves state-of-the-art performance on the NJN dataset (97.37% accuracy, 98.25% F1-score) and demonstrates strong cross-dataset generalization on the externally curated JaundiSet-NG dataset, which contains darker skin tones from African populations. Computational evaluation shows significant reductions in parameters, FLOPS, and inference latency, highlighting the framework’s suitability for real-world deployment in resource-constrained clinical environments.