This paper will demonstrate a solution for detecting damage to a bridge structure from measured displacements gathered using a roving vision sensor based approach. The measurement of displacement was accomplished using a synchronised multi-camera contactless vision based multiple point displacement measurement system using wireless action cameras. Displacement measurements can provide a valuable insight into the structural condition and service behaviour of bridges under live loading. Computer Vision systems have been validated as a means of displacement calculation, the research developed here is intended to form the basis of a real time damage detection system. This is done through the use of unsupervised deep learning methods for anomaly detection which could form the basis of a low cost durable alternative which is rapidly deployable in the field. The performance of the system was evaluated in a series of controlled laboratory tests. This research provides a means of detecting changes to a bridge structure through use of minimal sensor installation, reducing potential sources of error and allowing for potential live rating of bridge structures.