In recent days, recognizing handwritten digits in Arabic and English has been useful for several applications. This paper presents an efficient method to recognize the unlimited variation in human handwriting. We have used freely available datasets, MNIST and MADBase, for English and Arabic handwritten digits, respectively. Each dataset involves enough number of images with ten classes from 0 to 9, so that there are 70,000 images in total, 60,000 images are used for training and 10,000 images are used for testing the models. A Deep Learning-based methodology is suggested for recognizing handwritten digits by using various transfer learning types such as; AlexNet, ResNet-18, GoogleNet, and DensNet-201 aimed at deep feature extractions. Moreover, we utilized three types of classifiers: Decision Tree (DT), k-nearest neighbors (KNN), and Support Vector Machine (SVM) and compared their performances. The results show that the AlexNet features with SVM classifiers provide the best results for both datasets, with error rates of 0.96% and 0.9997% for Arabic and English databases, respectively.