A major step in lung cancer diagnosis is the classification of nodule malignancy, but in early stages, benign and malignant nodules appear very similar, leading to frequent misdiagnoses. This study developed a novel multimodal image-based CNN (MIB-CNN) model architecture to classify pulmonary nodules as either benign or malignant, performing multimodal learning on only computed tomography (CT) images, without the need for other clinical data like genomic tests. MIB-CNN takes in CT images of nodules, convolutionally extracts chosen semantic features from the images to obtain numeric data, and integrates it with the image data using a novel method, improving model performance and uncovering the mechanisms of the “black box” of this deep learning task. The results showed that the MIB-CNN model achieved 0.94 AUC on the LIDC-IDRI dataset compared to 0.90 AUC with a basic image CNN, and 0.91 specificity in comparison to 0.86 specificity of the basic model, indicating a significant decrease in the number of false positives. This study also identifies the primary causes of inaccurate predictions: small airways and other thoracic organs cause noise in the image data and decrease visibility of small nodules. Furthermore, the premise of MIB-CNN is not limited to this lung nodule malignancy classification task, as this methodology can be applied to other medical image-based deep learning tasks to overcome the challenge of limited multimodal data availability.