Computer vision has been utilized to analyze minimally invasive surgery videos and aid with polyp detection, tool localization, and organ 3D modelling tasks. However, irregular light patterns such as saturation, specular highlights, or extreme contrasts occlude texture and hinder these tasks. In this work, specular highlights were removed and the occluded data was reconstructed. To do that, an unsupervised temporal generative adversarial network (GAN) was used to inpaint specular highlights spatially and temporally. Due to the absence of a dataset with ground truth occluded textures, the network was trained on the in-vivo gastric endoscopy dataset with specular highlight masks that were automatically created and processed to act as pseudo ground truths. Ablation studies and direct comparison with other methods were used to show the improved results of our system. In addition, the results on various datasets show the generalizability of our network on different environments and procedures. Finally, experiments also show the positive effect of inpainting on other computer vision tasks under the umbrella of 3D reconstruction and localization in endoscopy including feature matching as well as optical flow and disparity estimation.