Condition based monitoring aims to measure the situation of industrial equipment or processes. This is important for detecting equipment changes or damages. Servicing mechanical devices has traditionally been relying heavily on aural observations. Accurate audio diagnostics require profound understanding and experience. Such skills are not always available on site, especially at remote locations. Therefore automated or assisted diagnostics are very interesting topics in predictive maintenance. Automating the diagnostics may involve using machine learning classifiers. Classifiers are trained using finite training sets. Trained classifiers are then expected to diagnose situations in varying operating conditions. If the classifier is being overfit, it would not perform well in production use. In this work we investigated ways to stress test the audio classifiers by using compression techniques to degrade audio quality for evaluating robustness of trained audio classifiers. We found a practical procedure to provide an additional measure to prevent selecting an overfitted classifier.