Augmented reality/mixed reality (AR/MR/XR) technologies are poised to create compelling and immersive user experiences by combining computer vision and computer graphics. Imagine users interacting with the world around them through their AR device. Visual search tells them what they are seeing, while computer graphics augments reality by overlaying real objects with virtual objects. AR/VR can have a far-ranging impact across many applications, such as retail, virtual prototyping, and entertainment.

In this talk, I will describe my group's research on these complementary areas: graphics models for realistic visual appearance, and visual search and fine-grained recognition for scene understanding. We will also see how these technologies can go beyond XR/AR/VR applications to enable visual discovery—using recognition as a core building block, we can mine social media images at a global scale to discover visual patterns and trends across geography and time.