The quality and safety of food is a great concern to the whole society because it is the most basic guarantee for human health and social development and stability. Ensuring food quality and safety is a complex process, and all stages of food processing must be considered, from cultivating, harvesting and storage to preparation and consumption. Grading is one of the essential processes to control food quality. This paper proposed a two-layer image processing system based on machine learning for banana grading. Support Vector Machine is the first layer to classify bananas based on an extracted feature vector that is composed of colour and texture features and YOLOv3 follows up for further locating the defected area on the peel and determining if the inputs belong to mid-ripened or well-ripened class. The performance of the first layer achieved an accuracy of 98.5\% and exceeded other algorithms such as KNN, Naive Bayes, and Random Forest. The classification accuracy of the second layer is 85.7\% and the overall accuracy is 96.4\%