Defects on the surface of railway tracks pose risks to safety and require quick and reliable identification and classification that are not limited to traditional human inspections. This study provides an instance segmentation framework based on YOLOv12 for the detection and classification of crack, flaking, shelling, spalling, and joint defects. The process included the preprocessing, annotation, and augmentation of the size-restricted, high-resolution dataset collected at the NCRA MUET site to develop a dataset that addressed the class imbalance in the defect classes. YOLOv12 performed well overall, achieving an mAP@0.5 of 0.976 with stable precision–recall behavior across all defect classes, besides joint defects. The system presented provides an accurate, real-time, and scalable method for automated inspections of railway track defects. and accurate automated inspection solution for railway tracks.