Ultrasound is a practical, non-invasive clinical method, which is the first choice for screening and detecting many diseases in current medical examination and treatment, especially breast cancer and thyroid cancer. In this study, a novel model based on stacking attention blocks and depth-wise convolutional blocks is proposed. This hybrid deep learning architecture enables highly accurate detection of breast and thyroid cancer, outperforming state-of-the-art models. The model employs window attention and coordinate attention mechanisms together with large depth-wise convolution kernels to reduce the number of parameters and expand the receptive field. The depth-wise convolutional blocks extract local features, while the attention blocks capture long-range dependencies; their combination produces feature representations rich in both texture and contextual information, thereby improving detection performance. Two datasets of breast cancer, BUSI, and thyroid cancer, TN5000 are selected experimentally to evaluate the proposed model and state-of-the-art models. Our method achieves 96.13% accuracy on the BUSI dataset and 89.99% on the TN5000 set, outperforming the SOTA models. The proposed model has 14.5 million parameters and 5.02 Gflops, which is optimal in both the number of parameters and computational performance compared to state-of-the-art models.