RICA: Real-Time Image Captioning Application

EasyChair Preprint no. 5740

Abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. Image caption generator is a task that involves computer vision and natural language processing concepts to recognize the context of an image and describe them in a natural language like English. The recent advances in Deep Learning based Machine Translation and Computer Vision have led to excellent Image Captioning models using advanced techniques like Deep Reinforcement Learning. While these models are very accurate, these often rely on the use of expensive computation hardware making it difficult to apply these models in real time scenarios, where their actual applications can be realised. In this paper, we carefully follow some of the core concepts of Image Captioning and its common approaches and present our simplistic encoder and decoder based implementation with significant modifications and optimizations which enable us to run these models on low-end hardware of hand-held devices. We also compare our results evaluated using various metrics with state-of-the-art models and analyze why and where our model trained on MSCOCO dataset lacks due to the trade-off between computation speed and quality. Using the state-of-the-art TensorFlow framework by Google, we also implement a first of its kind Android application to demonstrate the real time applicability and optimizations of our approach.

Keyphrases: Captioning, image, RICA

@Booklet{EasyChair:5740,
  author = {Suraj Dahake and Aditya Ohekar and Shubham Ilag and Aasim Shah},
  title = {RICA: Real-Time Image Captioning Application},
  howpublished = {EasyChair Preprint no. 5740},

  year = {EasyChair, 2021}}