Continuous condition monitoring of railway track and infrastructure is necessary to ensure safe and comfortable operation of the railway vehicle. Automated visual inspection of the track is highly desirable. Due to its advantages over its counterpart (manual inspection). Adaptation of the deep learning algorithms for visual inspection of railway track condition monitoring have yielded great results. However, most of the research uses deep learning algorithms in a supervised setting. Manual labeling of the acquired image data is a very tedious task and requires in depth domain expertise. Here we adapt an unsupervised approach using Auto-encoder in an anomaly detection setting. Auto encoder is trained on the healthy data acquired on site and later subjected to images with faults. Reconstruction loss yielded by Auto-encoder is used as the metric to detect anomalies. Results obtained show that faults on rail track surface can be identified without labeling of the image data.