A survey of evolution of image captioning techniques

Abstract

Automatic captioning of Images has been explored extensively in the past 10 to 15 years. It is one of the elementary problems in Computer Vision and Natural Language Processing and has vast array of applications in the real world. In this survey, we aim to study different approaches used for the generation of image captions in a chronological manner starting from the basic template based caption generation model to using Neural Networks combined with external world knowledge. We review existing models in detail, highlighting the involved methodologies and improvements in the same that have occurred in time. We gave an overview to the standard image datasets and the evaluation measures developed to discern the quality of generated image captions. Apart from the basic benchmarks we also note speed and accuracy improvements in all the different approaches. Finally, we investigate further possibilities in automatic image caption generation.

Keywords

Computer Vision image captioning deep learning object recognition Natural Language Processing

1. Introduction

The last two decades have seen great improvements and enthusiasm in the fields of Computer Vision and Natural Language Processing. Main target of these problems is studying and generating automatic text descriptions, and comprehending images and videos. These problems find their roots in AI and ML which themselves are in early phases considering their vast potential. Moreover, these fields have been investigated and researched separately, making it extremely important to study their combined scope and further investigating their possibilities.

The image captioning problem has been viewed for long as a challenging problem because of the need to identify different segments of the image correctly, identifying the connection between them and finally weaving them together in a syntactically and semantically correct manner to describe the most salient aspect of the image. Therefore to accomplish this task, we need the technology which can fully understand the image and should be capable to apply external worldly knowledge to generate a description which is most true to the image.In other words, it is equivalent to mimicking a human capability to compress the most salient features of an image into the most descriptive, but true-to-the-image description. This seems a herculean task relative to normal Computer Vision Evaluation Systems simply identifying what is present in the image.

However, with most of the world population connecting online, the need for a image-caption generation system has surged to attention-seeking levels. We have lots of multi-modal data arriving online every second in the form of millions of tagged photographs on Facebook and cloud storage. Google Photos image classifying and story making feature is a great example. Also, with lots of video content, the need for automatic subtitles for the videos has seen an appraisal. Automatic Image Captioning will help in organizing the millions of unstructured and unorganized and unclassified images on the Internet and on humanitarian grounds, will also aid the visually impaired to sense the images. Therefore, the dire needs for image-caption generation can not go unnoticed and hence, lots of research papers and conferences in this regard are taking place throughout the world.

At the heart of this technology are the power of neural networks which makes it seems like magic. On one hand, where Convolution Neural Network (CNN) Layer performs the prime task of identifying the salient features of the images, the Recurrent Neural Networks makes it possible to construct (almost) meaningful captions from the identified visual features.We will see how the technology advanced from basic tree parsing to use of neural networks utilizing hundreds of millions of features to make the results competent enough to be compared to those of human generated captions.We will be unwinding these technologies one by one in brief in this survey paper.

So, our aim of this survey paper is to present, analyze and compare the research happening throughout the last decade in this ground-breaking field, in a matter of few pages. We aim at chronologically identifying the technological developments, the involved approaches, their drawbacks, how well they perform at various metrics and further investigating the future scope of automatic image generation.

2. Evolution of image captioning techniques

The paper is structured in a chronological manner, where we start from the basic techniques used, discuss its methodology and shortcomings and then move to a newer approach that solved the problems in the previous one. Figure 1 shows the timeline we have developed to help keep track of the advancements in this field and better understand the need and basis of development of newer models.

Figure 1.

Timeline indicating the progress in automated image captioning.

The years 2005 to 2010 saw the birth of the major approaches dealing with computer vision and mapping the detected objects with words and weaving them into a meaningful or stylish description. Related work include those of Li et al. [5], Farhadi et al. [9] and Li et al. [6]. Also approaches involving adding some additional knowledge from the corresponding domain which normal computer vision can not see (Yang et al. [16]) as well as using some originally existing captions for the locally (Ordonez et al. [17]) and globally similar (Farhadi et al. [7]) and images to get rid off the need for structuring the self made sentences was observed during this period.

In years 2010 and 2011 some work still focussed on using primitive techniques for text generation for eg. Kulkarni et al. [15] used template-based description generation, Farhadi et al. [7] grouped the computer vision detections in triplets and then used them to generate descriptions based on templates. Li et al. [14] generated description by merging the computer vision detected objects using proper semantic relationships.

The year 2012 was a highlight in the era of automated Image Captioning Techniques as Imagenet classification with deep CNN (convolutional neural networks) was done using 60 million parameters and half million neurons, it consisted of five convolutional layers.

Year 2013 onwards, techniques involving the use of recurrent network started gaining momentum. Mao et al. [28] Vinyals et al. [48], Kiros et al. [25], Fang et al. [30], Chen et al. [47] used a recurrent NN and Kiros et al. [23] used a feedforward one. Also, Kiros et al. [25] proposed to create a â€œmultimodal embedding spaceâ€ by using a vision model and LSTM to encode text.

Years 2014 and 2015 saw the evolution of rich feature hierarchies for accurate and high quality object detection going deeper with convolutions. Fast progress in object detection [18, 49, 31, 32, 33] was identified with models which labeled multiple regions of image in image captioning.

Years 2015 and 2016, saw the evolution of very â€œdeep convolutional networksâ€ for large scale image recognition with multimodal R-CNN and various other techniques [24, 48, 34, 35, 36, 37, 38, 39, 40]. We discuss in detail 8 prominent categories of approaches belonging to different time periods. Starting with one of the very first techniques to the very recent one.

2.1 Template Based (Tree parser)

The template based technique is discussed using the work of Margaret Mitchell et al. [21] which was published in 2012.

2.1.1 Problems faced till now

Template Based Caption Generation was used earlier in – Kulkarni et al. [15], Yang et al. [16] system substitutes probable prepositions, verbs and interjections on parsing UIUC Pascal-VOC dataset (Farhadi et al. [9]) and choosing head nouns and their dependents using maximum likelihood calculated by taking the ratio of their individual logs.However, only predictable consistent sentences can be generated using template based techniques, but not novice captions. Ordonez et al. [17] matched the query image from a much larger set of existing captioned photographs followed by local reordering. Although natural, but these captions are not true to the image. They mainly describe the similar images and may miss out unique features of the query image.

2.1.2 Overview

Midge uses syntactical knowledge of the probability distribution of the next words which should appear after a given sequence of words. The generator uses constraints to filter out the noisy output from the vision system to generate syntactic trees to describe the image which computer vision detected.

Figure 2.

Tree generated using Midge approach during the tree growth process.

2.1.3 Model/Methodology

For training, 700,000 images from Flickr dataset were taken with respective descriptions in dataset used in Ordonez et al. [17].Before parsing , normalization of description was done. Parsing was done using Berkeley parser (Petrov [12]).Once a head noun was selected, for formulating a description , probability was calculated for determiners (the, a, an) and pre-nominal modifiers/adjectives.

Head nouns were identified and physical objects were distinguished using WordNet (Miller [1]) from the detections of the vision system. Maximum 3 objects were kept in a description of a single sentence. Caption generating process was dealt as a problem of growing a syntactically and semantically informed tree based on detected object nouns. Tree growth was achieved using lexicalized syntactic derivations using head noun anchors detected above.

A three step growth process was followed (Reiter and Dale [3]) that involved utilising content determination for grouping followed by ordering of the object nouns, generating their local subtrees, and filtering irregular detections. Micro planning was done to generate full syntactic trees around the noun objects detected, and modifiers are selected and classified in post-nominal and pre-nominal in surface realisation step and choosing the final outputs. The system followed an approach where multiple trees are grown and then, the best one is chosen as the result. Contexts (nouns) for adjectives were weighted using Point-wise Mutual Information and for any adjective, only the best 1000 nouns are selected.

In Micro-planning, fully-grown trees are generated by taking the intersection of the subtrees created in Content Determination. Subtrees surrounding a noun in position 1 are directly merged with subtrees surrounding a noun in position 2 because the nouns are ordered.In the surface realisation stage, the most probable single tree is chosen by the system from all generated possible trees and mark-up is removed to produce a final string. Different strings may be generated depending on different specifications from the user. The final string is then the one with the most words.

2.1.4 Evaluation metrics

5-point Likert scale, Human decisions accumulated using Amazonâ€™s Mechanical Turk (Amazon, 2011) were used as evaluation metrics.It was also evaluated against Kulkarni et al. system, the Yang et al. system, and human-generated descriptions on the same dataset (images). Other metrics include the parameters of grammar, main aspects, correctness, order, human likeness. Results Analysis were done using the non-parametric Wilcoxon Signed-Rank test where the parameter for comparing different systems is the median values.

2.1.5 Dataset

Training dataset: 700,000 images with their associated descriptions from the Flickr dataset in Ordonez et al. [17]. Testing dataset : 840 PASCAL images.

2.1.6 Results

Midge performed better than all earlier automatic approaches on criteria of correctness and order. And additionally performed better than Yang et al. on the criteria of close proximity of sentences with the human generated ones.

Figure 3 shows captions generated by this method.

Figure 3.

Captions generated as in [21] L to R: The bus by the road with a clear blue sky; People with a bottle at the table; A person in black with a black dog by potted plants.

2.2 Encoder – Decoder based Caption Generation

This technique is discussed using the work of Kiros, Salakhutdinov, and Zemel – Unifying visual semantic embeddings with multimodal neural language models [25] which was published in 2014.

2.2.1 Problems faced till now

Descriptions, by earlier strategies were more machine type in nature and failed to adapt to the fluidness of captions written by humans. Bleu and Rouge [22] evaluation ways were unreliable and did not match human perceptions.

2.2.2 Overview

Encoder (LSTM) ranks captions and pictures and develops sensible grading functions, and the decoder (SC-NLM) optimises the grading functions as some way of generating and grading new descriptions via the learnt representations.

2.2.3 Model/Methodology

Encoding of sentences was done by taking Long short-term memory (LSTM) recurrent neural netw-orks [2]. Projection of features of the image were taken from a deep CNN into the embedding region of the LSTM hidden states. Joint image-sentence embeddings were learnt and minimization of pairwise ranking loss so as to learn to rank pictures and their descriptions was performed. Images and descriptions were ranked. Using a decoder, a structure-content neural language model (SC-NLM), where the sentence structure was disentangled to its content and an encoder which created the conditioning on distributed representations. Sensible image captions were generated if sampling from SC-NLM was used. Decoder generated new captions from base.

Problem of ranking pictures and captions was used as alternate for generation. Optimising this task would lead to an enhancement in generation technique, because any generation system makes a grading function to analyse how well a caption and picture match.

2.2.4 Evaluation metrics

Med r; R@K.

2.2.5 Dataset

Flickr30K and Flickr8K.

2.2.6 Results

The ways delineated during this paper generated descriptions with quality greater than the that time state-of-the-art methods which were based on composition-based strategies. Authors worked on attention-based models which could learn to align the parts of captions to pictures and determining where to attend next by using these alignments, thus modifying dynamically the decoder conditioning vectors. Figure 4 shows captions generated by this method.

Figure 4.

Captions generated as in [25] L to R: A parked car while driving down the road; A little boy with a bunch of friends on the street; There is a cat sitting on a shelf.

2.3 Extracting Visual Features (RNN)

+

Maximum Entropy Language Model-Caption Generation

The approach is discussed using Minds Eye: A Recurrent Visual Representation for Image Caption Generation [47].

2.3.1 Problems faced till now

Many previous papers experimented projecting the image features and their associated description in common space [6, 7, 13] which find their uses in image search or image captions ranking.To learn these projections, various approached were used:Kernel Canonical Correlation Analysis (KCCA) [22], Recursive neural networks [29] and Deep neural networks [24]. While these techniques projected both visual features and associated semantics to joint embedding, they failed to perform the inverse projection. That is, they could not make fresh sentences or visual depictions from those joint embeddings.

2.3.2 Overview

This paper explored the bi-directional mapping between images and their sentence-based descriptions using a recurrent neural network. A new recurrent visual memory was deployed that mechanically learned to remember long-term visual concepts to help in both sentence production and visual feature reconstruction.

2.3.3 Model/Methodology

To accomplish the bidirectional mapping, a set of latent variables $U_{t-1}$ were introduced that encoded the visual features of the previously read/generated words $W_{t-1}$ .The latent variable $U$ played an important role of behaving as a long-term visual memory for the previously generated/read words (which was the heart of this paper). $U$ was used to calculate $P(w_{t}|V,W_{t-1},U_{t-1})$ and $P(V|W_{t-1},U_{t-1})$ . Combining these two probabilities together the author aimed to maximize, $P(w_{t},V|W_{t-1},U-{t-1})=P(w_{t}|V,W_{t-1},U_{t-1})P(V|W_{t-1},U_{t-1})$ . That is, given the previous words and their visual interpretation, author aimed to maximize the possibility of the word $w_{t}$ and the observed visual features $V$ .

Figure 5.

1. Part of the model needed for generating sentences from visual features and vice versa. 2. Sentences to Visual Features. 3. Visual Features to sentence.

Language Model: This system was able to generate 3000 to 20000 words using word classing approach [11] $P(w_{t}|)=P(c_{t}|)P(w_{t}|c_{t};)P(w_{t}|)$ is the probability of the word, $P(c_{t}|)$ is the probability of the class. Using frequency distribution, words were categorized into classes using clustering technique. In order to overcome the uncertainty of which word to be generated, the author took the the RNN models output and combined it with the Maximum Entropy language model [50] output, which was trained simultaneously. The context was kept short by limiting the words to look back in Maximum Entropy model to three for all the experiments.

Learning Model consisted of BPTT (Back-propagation Through Time) Algorithm to revise the weights online. Activation Function used for all units was sigmoid function $(s)=1=(1+\exp(s))$ and for word predictions, the clipping Activation Function Used was soft-max. Author used the RNN code(open-sourced) of [11] and the Caffe framework [51] to implement this model. Author used pre-trained 1000-class ImageNet [8] model ,rather than starting from scratch to prevent overfitting.

The recurrent hidden state $s$ supplies context on the basis of observed previous words. $v$ represents the set of observed (assumed to be constant) visual features. These visual features help in making an informed selection of words. For example, if a girl was detected, the probability of appearing next of the word girl automatically gets higher. $U$ , represents the hidden recurrent layer to reconstruct the visual features from the given word (so that bi-directional mapping can be felicitated ). $w_{t}$ is the next word predicted using this visual hidden layer, i.e. the network utilises its visual memory, $u$ , along with the currently observed visual features $v$ , to make the next word prediction. Before any words are observed, $U$ makes pure random guesses of the visual features. However, with more words being predicted, the visual feature probabilities are revised to predict words, which more closely depict the actual scene.

2.3.4 Evaluation metrics

Perplexity (PPL), BLEU, METEOR (METR), Human Subjects, Recall@1,5,10.

2.3.5 Dataset

PASCAL 1K, Flickr 8K and 30K, MS COCO.

2.3.6 Results

Figure 6 shows captions generated by this method.

Figure 6.

Captions generated as in [47] L to R: A train is stopped at a train station; A group of people standing on a snow covered slope; A group of people that are standing in front of a building.

Figure 7.

Captions generated as in [48] L to R: A red motorcycle parked on the side of the road; A group of young people playing a game of frisbee; Two dogs play in the grass.

2.4 Object Detection (CNN)

+

Caption Generation Model (RNN)

This technique is discussed using the work of Vinyals, Toshev, Bengio, and Erhan, Show and tell: A neural image caption generator [48], which was published in 2014.

2.4.1 Problems faced till now

Text generation in previous works was rigid and excessively handcrafted. It couldn’t create descriptions of previously unobserved arrangements of objects, even if separate objects were detected in the training set.

2.4.2 Overview

End-to-end system that combined newfangled subnetworks for object detection and caption generation models was proposed. This neural network was extensively trained using stochastic gradient descent and described the subject matter of an image using accurately built English sentences.

2.4.3 Model/Methodology

It was based on a neural and probabilistic architecture to produce image captions given an image as input and applying the principle of translation for generating its description. (similar to how we translate text between two languages).

The model first uses the following formula to maximize the probability of the correct description:

$\theta^{*}=\text{arg max}_{\theta}\sum_{I,S}\text{log p}(S|I;\theta)$ (1)

Here, $\theta$ represents the model parameters, $I$ , the input image and S the generated sentence. Chain rule is then applied to calculate the joint probability over all words in the sentence $S_{0},\ldots,S_{N}$ :

$\textit{log p}(S|I)=\sum_{t=0}^{N}\textit{log p}(S_{t}|I,S_{0},\ldots,S_{t-1})$ (2)

A constant length hidden state ht expresses the number of words we consider upto $t-1$ and it updated using a non linear function whenever it sees a new input $x_{t}$ .

$h_{t+1}=f(h_{t},x_{t})$ (3)

This non linear function $f$ was specified by a LSTM network which used words and images as inputs $x_{t}$ and was so trained that it would predict one word of the sentence at a time considering the context of image observed and also all the preceding words $p(S_{t}|I,S_{0},\ldots,S_{t-1})$ .

The loss was given by the the summation of the negative log likelihood of the right word generated at each time step as given below and was minimized with respect to all parameters of the LSTM network, word embeddings $W_{e}$ and the top layer of the Convolutional neural network.

$L(I,S)=-\sum_{t=1}^{N}\textit{log p}_{t}(S_{t})$ (4)

This paper used the BeamSearch approach with a beam of size 20. They also experimented using greedy search by taking beam size equal to 1 only to find out that it degraded the results by and average of 2 BLEU points (the other technique explored was Sampling).

Figure 8.

Captions generated as in [28] L to R: A square with burning street lamps and a street in the foreground; Tourists are sitting at a long table with a white table cloth and are eating; A blue sky in the background.

2.4.4 Evaluation metrics

Bleu-4, METEOR, CIDER. Ranking Metric Recall@k (@1 and @10).

2.4.5 Dataset

Pascal VOC 2008. Flickr8k, Flickr30k, MSCOCO, SBU.

2.4.6 Results

NIC is performed better than various other approaches e.g. Tri5Sem, Im2Text, BabyTalk, SOTA etc. and was quite close to the ground truth. Figure 7 shows captions generated by this method.

Figure 9.

Overview of fully convolutional localization network for dense captioning. The localization layer presents regions and extracts smoothly, batch of corresponding activations with the help of bi-linear interpolations.

2.5 Deep RNN

+

Deep CNN

+

multimodal layer interactions

This technique is discussed using the work of Mao et al. – Explain images with multimodal recurrent neural networks [28] which was published in 2014.

2.5.1 Problems faced till now

Earlier works extracted features for sentences and pictures, and mapped them into embedding space of same semantics. These strategies addressed tasks such as retrieval of sentences when the image is given or retrieval of images when the sentences are given but when they are existing within the database already, and lacking the flexibility to caption new pictures that consists of objects and scenes that are previously unseen.

2.5.2 Overview

The model contains 2 sub-networks: deep RNN for sentences and a deep CNN [20] for images where, RNN is Recurrent Neural Network and CNN is Convolutional Neural Network. These two sub-networks communicate with one other in a multimodal layer and this complete model is known as m-RNN model. It takes out probability distribution for generating a word provided previous words and picture are given and finally when this distribution is sampled, image descriptions are generated.

2.5.3 Model/Methodology

There are 6 layers in each time frame : first one is input word layer, then next two are Word Embedding layers, then there is Recurrent layer, then the layer where connection is made: Multimodal layer, and the last layer : Softmax layer.

$\displaystyle\textit{log}_{2}\textit{PPL}(w_{1:L}|I)=-\frac{1}{L}\sum_{n=1}^{L}$

(5) $\displaystyle\quad\textit{log}_{2}P(w_{n}|w_{1:n-1},I)$ $\displaystyle c=\frac{1}{N}\sum_{i=1}^{N}L\cdot\textit{log}_{2}\textit{PPL}(w_% {1:L}^{(i)}|I^{(i)})+\left\|\theta\right\|_{2}^{2}$ (6)

where PPL is the Perplexity of the sentence and $c$ is Cost calculated for the model.

Sentence Generation involved starting from the start sign ##START##, the model calculated the probability distribution for the upcoming word, given previous words and picture. Then the upcoming word was picked by sampling previously obtained probability distribution. But, the word which had the maximum probability was found out, since this method performed better, though slightly, than sampling. After that, the picked word was input to the model and the process is continued until the end sign ##END## is taken as output from the model

While doing retrieval of image, top ranked images were the output, where ranking was done on the basis of their perplexity with the query sentence. Sentence Retrieval used Normalized probability for each sentence.
2.5.4 Evaluation metrics

Sentence Perplexity & BLEU scores (B-1, B-2, B-3), RK (K $=$ 1, 5, 10) and Med r.For IAPR TC-12: Recall Accuracy Curve and (##RK (K $=$ 1, 5, 10) and Med r).

2.5.5 Dataset

Flickr 8K; Flickr 30K; IAPR TC-12.

2.5.6 Results

This was the first work in which RNN in a deep multimodal architecture was incorporated.

Figure 8 shows captions generated by this method.

2.6 Object Detection (R-CNN) $+$ Localization Layer $+$ Caption Generation Model (RNN)

This technique is discussed using the work of Johnson, Karpathy, Li – DenseCap: Fully Convolutional Localisation Networks for Dense Captioning [46] which was published in 2014.

2.6.1 Problems faced till now

Predictions based on earlier region CNN-RNN models did not include context outside of each region. Those were inefficient as each region had to be forwarded independently. Localization layer is was proposed due to these difficulties.

Figure 10.

The sequence of images shows Dense image captioning task using a model that generates rich and dense captions.

2.6.2 Overview

The paper consists of the work in the detection of objects, Image captioning, and the processing of particular regions of the image.

2.6.3 Model/Methodology

Convolutional Localization Network for Dense Captioning of the image was based on CNN- RNN models for image captioning but also included a differentiable localization layer that could be inserted in the neural network to enable localized predictions of the region proposals.

CNN consisted of 13 layers of 3 $\times$ 3 convolutions and formed the input for localization layer. Localization Layer classified convolutional anchor boxes according to their transformation and confidence scores and aligned region proposals that were proposed at the starting of object detection to the ground truth boxes.

Positive proposals were the ones that were matched and hence increased confidence scores while training, while negative proposals decreased the confidence scores.

Recognition network processed features of each region from the localization layer. The features of each region were flattened to be made into a vector and then passed through fully connected layers. Position was redefined and confidence scores of each region were proposed.

2.6.4 Evaluation metrics

METEOR, mean Average Precision (AP).

2.6.5 Dataset

MSCOCO, YFCC100M, Visual Genome (VG) Dataset.

2.6.6 Results

FLCN model performed better than the Region RNN in both ranking and localization under all metrics in a way that median rank reduced from 7 to 5 and localization recall from 0.5IoU to 0.153.

Figure 10 shows how captions are generated by this method.

2.7 Semantic Alignment Models (R-CNN and B-RNN) $+$ Description Generation Model (M-RNN)

This technique is discussed using the work of Karpathy and Li – Deep Visual-Semantic Alignments for Generating Image Descriptions [41] which was published in 2015.

2.7.1 Problems faced till now

The focus of most of the works so far has been on condensing elaborate visual depictions in an image to just one single sentence. However, this requirement is nothing but an unnecessary restriction.

2.7.2 Overview

This approach consists of two separate models, an alignment model for inferring the latent alignment between continuous group of words in a sentence and the region of the image that they correspond to and the second model which is trained on the inferred correlations.

Figure 11.

Flowchart of proposed description generation model.

Figure 12.

Captions generated as in [41] L to R: A man in black shirt is playing guitar; Two young girls are playing with lego toy; Construction worker in orange safety vest is working on road.

2.7.3 Model/Methodology

To detect objects in an input image the Alignment model used a Regional Convolutional Neural Network (RCNN). CNN was prepared by training it before hand on images in the ImageNet dataset and finally tuning it on the 200 classes of the ImageNet Challenge. In addition to the whole image 19 top detected locations were used. The objects were identified based on the pixels present inside each bounding box. It also used Bidirectional recurrent neural network (BRNN) to compute word representations in the sentence. An Image Sentence Score, $S k l$ aligned every word of a sentence to one best image region.

The ultimate goal was to associate snippets of text instead of single word to each bounding box. Therefore, the concept of Markov Random Field (MRF) and latent alignment variables was used to generate a number of image regions explained with segments of text. (for e.g. wooden table for table, messy pile of documents for documents)

M-RNN Model trained on the dataset of region-level annotations from the previous model took as inputs a series of input vectors and the image I. It then found out a series of hidden states and consequently a series of outputs by using a recurrence relation thereby generated a dense descriptions of images.

2.7.4 Evaluation metrics

Bleu-1,2,3,4, METEOR, CIDER. Ranking Metric Recall@1,5,10,Med r.

2.7.5 Dataset

Flickr8k, Flickr30k, MSCOCO.

2.7.6 Results

This model used very few hard-coded assumptions to formulate captions of individual image regions using the conventional dataset of images and sentences. Figure 12 shows captions generated by this method.

2.8 Object Detection (CNN) $+$ Description Generation Model (RNN) $+$ External Knowledge

This technique is discussed using the work of image Captioning and Visual Question Answering Based on Attributes and External Knowledge – Wu et al. [42] which was published in 2016.

2.8.1 Problems faced till now

The previous papers didnt take into account the external knowledge for generating the captions. Also the importance of introducing an intermediate attribute prediction layer was neglected by almost all previous work.

2.8.2 Overview

An intermediate attribute prediction layer is introduced into the predominant CNN-LSTM framework, which was neglected by almost all previous work.

Figure 13.

Image Caption Generated: A man with bat readies to swing at the pitch while the umpire looks on. External Knowledge: A pitch is a place used to play various sports such as cricket. The umpire is present to review the match.

2.8.3 Model/Methodology

Attributes predicted by the CNN-based attribute prediction model were used to generate the captions for the image. In the image captioning, the gaps in the caption templates were filled by the attributes predicted by the model. The model for caption generation was trained by maximizing the probability for the correct description of the image. The semantic attribute prediction value $V_{att}$ was used rather than using image features directly. The predicted attributes and generated captions were combined with the external knowledge in the knowledge base, and then put to the LSTM for providing answers to various questions. Figure 13 shows captions generated by attribute based captioning model mined with external knowledge.

2.8.4 Evaluation metrics

BLEU, METEOR and CIDER.

2.8.5 Dataset

Flickr8k, Flickr30k and Microsoft COCO.

2.8.6 Results

Att-Region CNN $+$ LSTM is so far one of the most suitable approach for generating image captions. Figure 14 question answering examples using the method used in [42].

Figure 14.

Examples where the attribute-Region-CNN $+$ LSTM gives the most appropriate answer while the baseline model gives wrong answer. Figure from paper [42].

3. Datasets

3.1 PASCAL 1K [43]

The images found in this dataset are a subset of images collected from PASCAL VOC Challenge. It has 20 categories of images, for each of which, it chooses 50 images at random as sample along with their descriptions which is generated by Amazon’s Mechanical Turk.

3.2 Flickr8K & 30K [10]

There are 8000 and 31,783 images respectively in Flickr 8K and 30K datasets which are gathered from Flickr. Majority of these images represent participation of human beings in various tasks. Every image has 5 sentences describing it. These datasets are split for training, testing as well as validation following some approved standards.

Table 1
Image captioning techniques and their scores on IAPRTC12 dataset

Approaches	Year	B-1	B-2	B-3	PPL
IAPRTC12
BACK-OFF GT2	2007	32.3	14.5	5.9	55.4
BACK-OFF GT3	2007	31.2	13.1	5.9	55.6
LBL	2007	32.7	14.4	1	20.1
Gupta et al. [56]	2012	15	6	7
Gupta & Mannem [19]	2012	33	18	9.8
MLBL-B-DeCAF [13]	2014	37.3	18.7	9.8	24.7
MLBL-F-DeCAF [13]	2014	36.1	17.6	9.2	21.8
m-RNN Baseline [28]	2014	31.34	11.68	8.03	7.77
m-RNN [28]	2014	39.51	18.28	13.11	6.92

Table 2

Image captioning techniques and their scores on PASCAL dataset

Approaches	Year	B-1	B-2	B-3	PPL
PASCAL
BabyTalk [15]	2011	25	0.49	9.69
Im2Text [17]	2011	25
LBL	2012	32.7	14.4	1	20.1
Midge [21]	2012		2.89	8.80
RNN	2013		2.79	10.08	36.79
RNN $+$ IF	2013		10.16	16.43	30.04
RNN $+$ IF $+$ FT	2013		10.18	16.45	29.43
Tri5Sem [22]	2013	25
Microsoft Bidirectional Retrieval	2014		10.48	16.69	27.97
Microsoft Bidirectional Retrieval $+$ FT	2014		10.77	16.87	26.95
TreeTalk [26]	2014	25
m-RNN [28]	2014	25
NIC [48]	2015	59
HUMAN	2015	70

Table 3

Image captioning techniques and their scores on Flikr8k dataset

Approaches	Year	B-1	B-2	B-3	B-4	BLEU	M	PPL
Flikr8k
RNN	2013					4.86	11.81	21.88
RNN $+$ IF	2013					12.04	17.10	20.43
Tri5Sem [22]	2013	48
Microsoft Bidirectional Retrieval	2014					14.10	17.97	19.24
m-RNN [28]	2014	58
MNLM [25]	2014	51
Mao et al. [28]	2014	58	28	23				24.39
Google NIC [48]	2014	63	41	27
Chen and Zitnik [47]	2014				14.1
NIC [48]	2015	63
Karpathy et al. [Deep Visual Semantic]	2015	57.9	38.3	24.5	16.0
Xu et al. (Hard-Attention)	2015	67	46	31	21
Att-SVM $+$ LSTM	2016	73	53	38	26			12.63
Att-GlobalCNN $+$ LSTM	2016	72	53	38	27			12.63
Att-RegionCNN $+$ LSTM	2016	74	54	38	27			12.60
Att-GT $+$ LSTM	2016	76	57	41	29			12.52
HUMAN	2016	70				22.51	26.31

Table 4

Image captioning techniques and their scores on Flikr30k dataset

Approaches	Year	B-1	B-2	B-3	B-4	BLEU	M	PPL
Flikr30k
RNN	2013					6.29	12.34	26.94
RNN $+$ IF	2013					12.59	15.56	23.74
Microsoft Bidirectional Retrieval	2014					12.60	16.42	22.51
m-RNN [28]	2014	55
MNLM [25]	2014	56
Mao et al. [28]	2014	55	24	20				35.11
Google NIC [48]	2014	66.3	42.3	27.7	18.3
Chen and Zitnik [47]	2014				12.6
LRCN [35]	2014	58.8	39.1	25.1	16.5
NIC [48]	2015	66
Karpathy and Li [41]	2015	57.3	36.9	24	15.7
Xu et al. (Hard-Attention)	2015	67	44	30	20
Att-SVM $+$ LSTM	2016	68	49	33	23
Att-GlobalCNN $+$ LSTM	2016	70	50	35	27
Att-RegionCNN $+$ LSTM	2016	73	55	40	28
Att-GT $+$ LSTM	2016	78	57	42	30
HUMAN	2016	68				19.62	23.76

Table 5

Image captioning techniques and their scores on MSCOCO dataset

Approaches	Year	B-1	B-2	B-3	B-4	BLEU	M	C	PPL
MSCOCO
Random	2005				4.6		9.0	5.1
Nearest Neighbour	2006	48.0	28.1	16.6	9.9		15.7	36.5
RNN	2013					4.63	11.47		18.96
RNN $+$ IF	2013					16.60	19.24		15.39
RNN $+$ IF $+$ FT	2013					16.77	19.41		14.90
Microsoft Bidirectional Retrieval	2014					18.35	20.04		14.23
Microsoft Bidirectional Retrieval $+$ FT	2014					18.99	20.42		13.98
Google NIC [47]	2014	66.6	46.1	32.9	24.6
Chen and Zitnik [47]	2014				19		20.4
LRCN [35]	2014	62.8	44.2	30.4
NIC [48]	2015				27.7		23.7	85.5
Karpathy and Li [41]	2015	62.5	45	32.1	23		19.5	66.0
Xu et al. (Hard-Attention)	2015	72	50	36	25		23
Att-SVM $+$ LSTM	2016	69	52	38	28		23	82	12.62
Att-GlobalCNN $+$ LSTM	2016	72	54	40	30		25	83	11.39
Att-RegionCNN $+$ LSTM	2016	74	56	42	31		26	94	10.49
Att-GT $+$ LSTM	2016	80	64	50	40		28	107	9.6
HUMAN	2016				21.7	20.19	24.94	85.4

3.3 MS COCO [27]

This is Microsoft Coco dataset which contains 82,000 training images and 40,000 validation images complemented by 5 sentences for their description. These images are sourced from Flickr by finding common/famous object categories and generally, they contain variety of objects with important information pertaining to their context.

3.4 IAPR TC [4]

In this dataset, there are 20,000 still natural pictures from various locations all around the world. Pictures from various categories like – sports, actions, cities, and shapes, animals, people and many other aspects of modern life. There are captions related with every image, in 3 specific languages English, German, Spanish. These 20000 images are of high resolution and strict image selection rules are followed while choosing images for this dataset.

3.5 VISUAL GENOME (VG) [52]

It is a dataset built by experts mainly from Stanford, Yahoo. It is a knowledge base which is basically a persistent effort to relate the image concepts to their natural language description in a structured manner. It is currently the largest dataset of image based question and answers with approximately over 17,000,000 question-answer pairs. Every image is supplemented with an average of 17 question-answer pairs.

4. Evaluation and ranking metrics

We have prepared a table for various datasets in chronological order showing various approaches used for the task of description generation and their corresponding scores using a variety of different metrics such as Bleu 1, 2, 3, 4. Meteor, Cider [45] etc. The results compiled in such a manner allow us to clearly see how over the years, image captioning techniques have evolved over time and also observe the large amounts of positive change in evaluated scores.

BLEU (bilingual evaluation understudy) [43] and METEOR (Metric for Evaluation of Translation with Explicit Ordering) [44] are metrics generally for the evaluation of machine translation output. R@K: recall rates for the first retrieved ground truth sentences or images. Some spaces in table are left empty as the corresponding scores were not calculated. Tables 1, 2, 3, 4, 5, 6 and 7 show the comparison of scores for various dataset using different techniques in a chronological order.

Table 6
Image captioning techniques and their recall scores on Flikr8k dataset

Flikr8k		Image Annotations				Image Search
APPROACHES	YEAR	R $@$ 1	R $@$ 5	R $@$ 10	med r	R $@$ 1	R $@$ 5	R $@$ 10	med r
Random	2005	0.1	0.5	1	631	0.1	0.5	1	500
DeFrag [24]	2014	13	33	44	14	10	30	43	15
m-RNN [28]	2014	15	37.2	49	11	12	31	42	15
MNLM [25]	2014	18		55	8	13		52	10
Socher-decaf [53]	2014	4.5	18	28.6	32	6.1	18.5	29	29
Socher-avg-rcnn [53]	2014	6	22.7	34.0	23	6.6	21.6	31.7	25
DeViSE-avg-rcnn [55]	2014	4.8	16.5	27.3	28	5.9	20.1	29.6	29
DeepFE decaf [54]	2014	5.9	19.2	27.3	34	5.2	17.6	26.5	32
DeepFE-rcnn [24]	2014	12.6	32.9	44	14	9.7	29.6	42.5	15
m-RNN-decafe [28]	2014	14.5	37.2	48.5	11	11.5	31.0	42.4	15
NIC [48]	2015	20		61	6	19		64	5
MNLM [25]	2014	13.5	36.2	45.7	13	10.4	31	43.7	14
MNLM [25] (oxford-net)	2014	18	40.9	55	8	12.5	37	51.5	10

Table 7

Image captioning techniques and their recall scores on Flikr30k dataset

Flikr30k		Image Annotations				Image Search
APPROACHES	YEAR	R $@$ 1	R $@$ 5	R $@$ 10	med r	R $@$ 1	R $@$ 5	R $@$ 10	med r
Random	2005	0.1	0.6	1.1	631	0.1	0.5	1	500
DeFrag [24]	2014	16	40.2	55	8	10	31.4	45	13
m-RNN [28]	2014	18	40.2	51	10	13	31.2	42	16
MNLM [25]	2014	23		63	5	17		57	8
DeViSE-avg-rcnn [54]	2014	4.8	16.5	27.3	28	5.9	20.1	29.6	29
DeepFE-rcnn [24]	2014	16.4	40.2	54.7	8	10.3	31.4	44.5	13
m-RNN-decafe [28]	2014	18.4	40.2	50.9	10	12.6	31.2	41.5	16
SDT-RNN (Socher et al. [57])	2014	9.6	29.8	41.1	16	8.9	29.8	41.1	16
Kiros et al. [25]	2014	14.8	39.2	50.9	10	11.8	34.0	46.3	13
Donahue et al. [35]	2014	17.5	40.3	50.8	9
Vinyals et al.	2014	23		63	5	17		57	8
NIC [48]	2015	17		56	7	17		57	7
DeFrag (Karpathy et al. [24])	2015	19.2	44.5	58.0	6	12.9	35.4	47.5	10.8
DepTree edges(Karpathy et al.)	2015	20	46.6	59.4	5.4	15	36.5	48.2	10.4
BRNN (Karpathy et al.)	2015	22.2	48.2	61.4	4.8	15.2	37.7	50.5	9.2
MNLM [25]	2014	14.8	39.2	50.9	10	11.8	34	46.3	13
MNLM [25](oxfordnet)	2014	23	50.7	62.9	5	16.8	42	56.5	8

5. Challenges

The possibility of developing intelligent computer programs that could correctly interpret and caption photos have been intriguing machine learning experts since decades. However, it was only a few years ago some significant progress in this field has been made. We have come a long way from template based techniques to deep learning ones with attention models but still there are a lot of challenges that need to be overcome.

One of the challenges is the prudent use of an attention system which would describe individual components of an image rather than just the image as a whole in order to create a holistic description of the complete picture. The challenge here is to incorporate more knowledge than just what the model is trained on. This includes understanding the context of the image and incorporating worldly knowledge while generating captions, just as humans would do. Only since last year, a few researches have started working on this issue however significant improvements have not yet surfaced. Better performance can be expected by choosing a superior image encoder, fine-tuning it and setting up ensemble models.

Performance of a system can be judged better if we have better evaluation and ranking metrics. While most of the above discussed approaches use BLEU scores to compare their results to the ground truth suggesting this metric to be a benchmark of evaluation and having some obvious advantages, a number of shortcomings have been noticed. It has been noted that BLEU cannot deal with languages lacking word boundaries. Another problem is its bias towards shorter translations. We could use other automated metrics involving human effort such as HyTER, however it is still just an approximation.

6. Future scope

The field of image captioning has been researched for decades as we just saw. However there is an immense scope still left to explore. Though most of the recent studies have been pretty successful in describing the image correctly, but still, human level accuracy and descriptiveness seems a far fetched idea.

This all boils down to one thing, knowledge. Humans, while thinking for a caption, use their entire knowledge base which they have been acquiring for years. Hence, the emotions, the extra worldly knowledge, the power to express which humans possess is sufficient enough for any human to fail a machine in this so-simple-for-human task.

So, the need of the future is to have an excellent knowledge base, the hardware power to train any model to use that entire knowledge feasibly, in order for that machine to develop an entire multi-dimensional context(s), so that any open-ended question related to the image can be answered irrespective of the attributes simply detected using any computer vision system.

This is the reason, why the major search engines’ corporations, like Google and Microsoft (Bing) have the best cards with them to utilize the power of humongous databases to turn them into knowledge-bases and realize the future of this technology. Microsoft’s “CAPTION BOT” is an excellent example of this initiative which uses the power of Emotions, Computer Vision and most importantly the power of Bing to really give fantastic results.

7. Conclusion

We classified and discussed 8 major approaches used for image captioning according to the order in which they developed. We discussed how and why an approach evolved so as to solve the shortcomings of the previous one. We then explained each of the approaches in detail with the help of a particular study and lastly, compared the results of various experiments conducted so far using various popular metrics such as BLEU scores, METEOR, CIDER etc. We were able to observe clearly the large amount of positive difference the scores.

Footnotes

Acknowledgments

A very sincere thanks to Shubham Thakkar, Saumya Gupta and Shubham Singh who helped us throughout the formulation of this survey paper. This would not have been possible without their constant support.

References

Miller

G.A.

, WordNet: A lexical database for English, Communications of the ACM 38(11) (1995), 39–41.

Hochreiter

and Schmidhuber

, Long short-term memory, Neural computation, 1997.

Reiter

and Dale

, Building Natural Language Generation Systems, Cambridge University Press, 2000.

Grubinger

Clough

Mu ller

and Deselaers

, The iaprtc-12 benchmark: A new evaluation resource for visual information systems.

L.-J.

and Fei-Fei

, What, where and who? classifying events by scene and object recognition, ICCV, 2007.

L.-J.

Socher

and Li

F.-F.

, in: Towards total scene understanding: Classification, annotation and segmentation in an automatic framework, Vision

and Recognition

, CVPR. IEEE Conference on, IEEE, 2009, pp. 2036–2043.

Farhadi

Endres

Hoiem

and Forsyth

, Describing objects by their attributes, Proceedings of CVPR, 2009.

Deng

Dong

Socher

L.-J.

and Li

F.-F.

, in: Imagenet: A large-scale hierarchical image database, Vision

and Recognition

, CVPR(2009). IEEE Conference on, IEEE, 2009, pp. 248–255.

Farhadi

Hejrati

Sadeghi

M.A.

Young

Rashtchian

Hockenmaier

and Forsyth

, Every picture tells a story: Generating sentences from images, In ECCV, 2010.

10.

Rashtchian

Young

Hodosh

and Hockenmaier

, Collecting image annotations using amazons mechanical turk, In NAACL HLT Workshop on Creating Speech and Language Data with Amazons Mechanical Turk, 2010, pp. 139–147.

11.

Mikolov

Karafiat

Burget

Cernocky

and Khudanpur

, Recurrent neural network based language model, In INTERSPEECH, 2010.

12.

Petrov

, Berkeley parser, GNU General Public License v.2, 2010.

13.

Kiros

and Salakhutdinov

R.Z.R.

, Multimodal neural language models, In NIPS Deep Learning Workshop, 2013.

14.

Kulkarni

Berg

T.L.

Berg

A.C.

and Choi

, Composing simple image descriptions using web-scale n-grams, In Conference on Computational Natural Language Learning, 2011.

15.

Kulkarni

Premraj

Dhar

Choi

Berg

A.C.

and Berg

T.L.

, Baby talk: Understanding and generating simple image descriptions, In CVPR, 2011.

16.

Yang

Teo

C.L.

Daume

III and Aloimonos

, Corpus-guided sentence generation of natural images, In EMNLP, 2011.

17.

Ordonez

Kulkarni

and Berg

T.L.

, Im2text: Describing images using 1 million captioned photographs, In NIPS, 2011.

18.

Feng

and Lapata

, Automatic Caption Generation for News Images, IEEE Transactions on Pattern Analysis and Machine Intelligence 35(4) (April 2013), 797–812. doi: 10.1109/TPAMI.2012.118.

19.

Gupta

and Mannem

, From image annotation to image description, In Neural information processing, Springer, 2012.

20.

Krizhevsky

Sutskever

and Hinton

G.E.

, Imagenet classification with deep convolutional neural networks, In NIPS, 2012.

21.

Mitchell

Han

Dodge

Mensch

Goyal

Berg

Yamaguchi

Berg

Stratos

and Daume

III, Midge: Generating image descriptions from computer vision detections, In EACL, Association for Computational Linguistics, 2012, pp. 747–756.

22.

Hodosh

Young

and Hockenmaier

, Framing image description as a ranking task: Data, models and evaluation metrics, JAIR 47 (2013).

23.

Kiros

and Salakhutdinov

R.Z.R.

, Multimodal neural language models, In NIPS Deep Learning Workshop, 2013.

24.

Karpathy

Joulin

and Li

F.-F.

, Deep fragment embeddings for bidirectional image sentence mapping, NIPS, 2014.

25.

Kiros

Salakhutdinov

and Zemel

R.S.

, Unifying visual-semantic embeddings with multimodal neural language models, In arXiv:1411.2539(2014).

26.

Kuznetsova

Ordonez

Berg

and Choi

, Treetalk: Composition and compression of trees for image descriptions, ACL 2(10) (2014).

27.

Lin

T.-Y.

Maire

Belongie

Hays

Perona

Ramanan

Dolla r

and Zitnick

C.L.

, Microsoft coco: Common objects in context, arXiv:14050312(2014).

28.

Mao

Yang

Wang

and Yuille

, Explain images with multimodal recurrent neural networks, In arXiv:1410.1090(2014).

29.

Socher

Karpathy

Q.V.

Manning

C.D.

and Ng

A.Y.

, Grounded compositional semantics for finding and describing images with sentences, TACL, 2014.

30.

Fang

Gupta

Iandola

Srivastava

Deng

Dollar

Gao

Mitchell

Platt

et al., From captions to visual concepts and back, arXiv preprint arXiv:1411. 4952(2014).

31.

Szegedy

Reed

Erhan

and Anguelov

, Scalable, high-quality object detection, arXiv preprint arXiv:1412. 1441(2014).

32.

Girshick

Donahue

Darrell

and Malik

, Rich feature hierarchies for accurate object detection and semantic segmentation, 2014.

33.

Russakovsky

Deng

Krause

Satheesh

Huang

Karpathy

Khosla

Bernstein

Berg

A.C.

and Li

F.-F.

, ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision (IJCV), April 2015, p. 142.

34.

Chenand

Lawrence Zitnick

, Minds Eye: A Recurrent Visual Representation for Image Caption Generation in Proc. IEEE Conf. Comp. Vis. Patt. Recogn, 2015.

35.

Donahue

Hendricks

L.A.

Guadarrama

Rohrbach

Venugopalan

Saenko

and Darrell

, Long-term recurrent convolutional networks for visual recognition and description in Proc, IEEE Conf. Comp. Vis. Patt. Recogn, 2015.

36.

Mao

Yang

Wang

and Yuille

, Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) in Proc. Int. Conf. Learn. Representations, 2015.

37.

Yao

Torabi

Cho

Ballas

Pal

Larochelle

and Courville

, Describing videos by exploiting temporal structure in Proc. IEEE Int. Conf. Comp. Vis., 2015.

38.

Devlin

Cheng

Fang

Gupta

Deng

Zweig

and Mitchell

, Language models for image captioning: The quirks and what works, arXiv preprint arXiv:1505. 01809(2015).

39.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition. in Proc. Int. Conf. Learn. Representations, 2015.

40.

Szegedy

Liu

Jia

Sermanet

Reed

Anguelov

Erhan

Vanhoucke

and Rabinovich

, Going deeper with convolutions, in Proc. IEEE Conf. Comp. Vis. Patt. Recogn, 2015.

41.

Karpathy

and Li

F.-F.

, Deep Visual-Semantic Alignments for Generating Image Descriptions, arXiv:1412. 2306v2. (2015).

42.

Wang

Shen

Dick

and Hengel

A.V.D.

, Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources in Proc. IEEE Conf. Comp. Vis. Patt. Recogn, 2016.

43.

Papineni

Roukos

Ward

and Zhu

W.-J.

, Bleu: A method for automatic evaluation of machine translation, In Proceedings of the 40th annual meeting on association for computational linguistics, Association for Computational Linguistics, 2002, pp, 311–318.

44.

Banerjee

and Lavie Meteor

, An automatic metric for mt evaluation with improved correlation with human judgments, In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005, pp. 65–72.

45.

Vedantam

Zitnick

C.L.

and Parikh

, Cider: Consensus-based image description evaluation, CVPR, 2015.

46.

Johnson

Karpathy

and Li

F.-F.

, DenseCap: Fully Convolutional Localization Networks for Dense Captioning CoRR, abs/1511.07571(2015).

47.

Chen

and Zitnick

C.L.

, Mind’s eye: A recurrent visual representation for image caption generation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 2422–2431.

48.

Vinyals

Toshev

Bengio

and Erhan

, Show and tell: A neural image caption generator, arXiv preprint arXiv:1411.4555(2014).

49.

Sermane

Eigen

Zhang

Mathieu

Fergus

and LeCun

, OverFeat: Integrated recognition, localization and detection using convolutional networks, ICLR, 2014.

50.

Mikolov

Deoras

Povey

Burget

and Cernocky

, Strategies for training large scale neural network language models, In Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, IEEE, 2011, pp. 196–201.

51.

Jia

Shelhamer

Donahue

Karayev

Long

Girshick

Guadarrama

and Darrell

, Caffe: Convolutional architecture for fast feature embedding, arXiv preprint arXiv:1408.5093(2014).

52.

Ranjay

Zhu

Groth

Johnson

Hata

Kravitz

Chen

Kalantidis

L.-J.

Shamma

D.A.

Bernstein

and Li

F.-F.

, Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations, arXiv:1602.07332(2016).

53.

Socher

Manning

and Ng

, Grounded compositional semantics for finding and describing im- ages with sentences, In NIPS Deep Learning Workshop, 2013.

54.

Frome

Corrado

G.S.

Shlens

Bengio

Dean

Mikolov

et al., Devise: A deep visual-semantic embedding model, In Advances in Neural Information Processing Systems, 2013, pp. 2121–2129.

55.

Karpathy

Joulin

and Li

F.-F.

, Deep fragment embeddings for bidirectional image sentence mapping, arXiv preprint arXiv:1406.5679(2014).

56.

Gupta

Verma

and Jawahar

, Choosing linguistics over vision to describe images, In AAAI, 2012.

57.

Socher

Karpathy

Q.V.

Manning

C.D.

and Ng

A.Y.

, Grounded compositional semantics for finding and describing images with sentences, TACL, 2014.

A survey of evolution of image captioning techniques

Abstract

Keywords

1. Introduction

2. Evolution of image captioning techniques

2.1.1 Problems faced till now

2.1.2 Overview

2.1.4 Evaluation metrics

2.1.5 Dataset

2.1.6 Results

2.2.1 Problems faced till now

2.2.2 Overview

2.2.3 Model/Methodology

2.2.4 Evaluation metrics

2.2.5 Dataset

2.2.6 Results

2.3.1 Problems faced till now

2.3.2 Overview

2.3.3 Model/Methodology

2.3.5 Dataset

2.3.6 Results

2.4.1 Problems faced till now

2.4.2 Overview

2.4.3 Model/Methodology

2.4.5 Dataset

2.4.6 Results

2.5.1 Problems faced till now

2.5.2 Overview

2.5.3 Model/Methodology

2.5.5 Dataset

2.5.6 Results

2.6 Object Detection (R-CNN) + Localization Layer + Caption Generation Model (RNN)

2.6.1 Problems faced till now

2.6.3 Model/Methodology

2.6.4 Evaluation metrics

2.6.5 Dataset

2.6.6 Results

2.7 Semantic Alignment Models (R-CNN and B-RNN) + Description Generation Model (M-RNN)

2.7.1 Problems faced till now

2.7.2 Overview

2.7.4 Evaluation metrics

2.7.5 Dataset

2.7.6 Results

2.8 Object Detection (CNN) + Description Generation Model (RNN) + External Knowledge

2.8.1 Problems faced till now

2.8.2 Overview

2.8.4 Evaluation metrics

2.8.5 Dataset

2.8.6 Results

3.1 PASCAL 1K [43]

3.2 Flickr8K & 30K [10]

Table 1 Image captioning techniques and their scores on IAPRTC12 dataset

3.4 IAPR TC [4]

3.5 VISUAL GENOME (VG) [52]

4. Evaluation and ranking metrics

Table 6 Image captioning techniques and their recall scores on Flikr8k dataset

6. Future scope

7. Conclusion

Footnotes

Acknowledgments

References

2.6 Object Detection (R-CNN) $+$ Localization Layer $+$ Caption Generation Model (RNN)

2.7 Semantic Alignment Models (R-CNN and B-RNN) $+$ Description Generation Model (M-RNN)

2.8 Object Detection (CNN) $+$ Description Generation Model (RNN) $+$ External Knowledge

Table 1
Image captioning techniques and their scores on IAPRTC12 dataset

Table 6
Image captioning techniques and their recall scores on Flikr8k dataset