Variable-length sequence model for attribute detection in the image

Abstract

Holistic scene understanding is a challenging problem in computer vision. Most recent researches in this field were focusing on the object detection, the semantic segmentation and the relationship detection tasks. The attribute can provide meaningful information for the object instance, thus the object instance can be expressed more detail in the scene understanding. However, most researches in this field have been limited to several special conditions. Such as, several researches were just focusing on the attribute of special object class, because their solutions were aimed at a limited-scenarios, their methods are hardly to generalize in other scenarios. We also find that most of the research for multi-attribute detection task were only regarding each attribute as binary class and simply use the multi-binary-classifier method for the attribute detection. But these strategies above not consider the relation between each pair of the attributes, they will fall into trouble in the “imperfect” attribute dataset (which is labeled with the missing and incomplete annotations), and they will have low performance in the long-tail attribute class (which has lower rank of annotation and more missing labels). In this paper, we focus on the multi-attribute detection for a variant of object classes and take the relation between attributes into consideration. We propose a GRU-based model to detect a variable-length attribute sequence with a customized loss compute method to solve the “imperfect” attribute dataset problem. Furthermore, we perform ablative studies to prove the effectiveness of each part of our method. Finally, we compare our model with several existed multi-attribute detection methods on VG (Visual Genome) and CUB200 bird datasets to prove the superior performance of the proposed model.

Keywords

Attribute detection scene understanding variable-length attribute detection VADN

1. Introduction

Holistic scene understanding is a challenging task for computer vision, autonomous robot and other intelligent agent fields of research. To achieve the goal of scene understanding for the intelligent agent, great efforts have been made in the recent years, such as the object detection Redmon et al. et al. [19, 20], image segmentation Long et al. [15], Liang et al. [13], the relationship detection Zhang et al. [27], Dai et al. [4], Li et al. [10] and the image classification Daniels and Metaxas (2018).

We think the attribute detection is also a key task of the holistic scene understanding. The attribute is a fine grained description of the object instance, and the attribute can describe the object from many perspectives, such as color, material, shape, action and so on. Several other tasks have shown to achieve a better performance with the use of object attributes, e.g., question and answer Antol et al. [1], Yu et al. [25] automatic caption generation Johnson et al. [7], Liang et al. [11], fine-grained recognition Farhadi et al. [5]and face recognition Iranmanesh, Kazemi, Soleymani, Dabouei and Nasrabadi (2018) Taherkhani et al. [22].

We consider object detection is one of the classic tasks of data mining. Analyzing anomalous data helps enterprises or users understand the formation mechanism behind it, so that they can make corresponding decisions and avoid losses. With the development of the network, anomaly detection for structured data, that is, graph anomaly detection, has received more and more attention. However, general-purpose graph neural networks (such as Graph Convolutional Networks, etc.) are mainly designed for normal data, and are prone to encounter the problem of “over-smoothing” in anomaly detection tasks, the expression of abnormal nodes and normal nodes is difficult to distinguish, which affects the accuracy of anomaly detection. Thus, object detection is usually a complex system engineering, but choosing the appropriate graph neural network is a key factor affecting system performance.

We regard the attribute as a set of the collection for an object instance, and the order in the attribute annotation for each object instance is nothing else matters. The relationship between attributes is no doubt, and the relevance of relationship between different pair of attributes are varied. Such as, “yellow” and “orange” are both color attributes, and their relationship should be stronger than the relationship between “yellow” and “big”. Several attributes are abstract, the vision feature is inefficient to represent these attributes, but with the relationship between attributes, these attributes may easier to be detected. We also consider that there is relationship between attributes and object class, such as “wet” is highly related to “sand”. We find that only several researches can take the relation described above in to consideration, such as Deep variation-structured reinforcement learning (DVRL) Liang et al. [12] and Multi-Label Image Recognition with Graph Convolutional Networks (ML-GCN) Chen et al. [3].

A large part of researches of multi-attribute detection were only focusing on the attributes for a special object class to reduce the annotation workload, and the attribute classes are pre-defined to ensure the high accuracy of the attribute annotation. Then, they used the multi-binary-classifier to solve the multi-attribute detection problem. The above method works well on the attributes for a special object class with well annotated attribute dataset, but if we do not limit the object class and cannot ensure all the attributes are carefully annotated, the multi-binary-classifier method will fall into trouble.

The public attribute detection dataset has more challenging compared to the object detection dataset. For each object instance, more than one attributes can be labeled by an annotator and the workload will be multiplied. Enumerating all the attribute for the given object instance and labeling them are difficult for annotators, that makes the attribute dataset have more uncertainty. The unlabeled attribute may be not really existing or annotator’s negligence, thus, we can only ensure that the annotated attributes are positive samples, but not all the unlabeled attribute (especially the longtail attribute) are negative samples. We call this an “imperfect” characteristic for many attribute datasets. Moreover, the number of the attributes for different object was vary significantly, which make the traditional classifier method incompetent.

The Visual Genome (simplified as VG below) Krishna et al. [8] is a dataset for vision tasks and it has different kinds of annotations, including 18k object classes and 13k attribute classes annotations, their attributes are generated from the region caption. Thus, the attribute annotation in VG is “imperfect”, not only half of the object instances are unlabeled with attributes, but also another half of the object instances are only labeled with several conspicuous attributes. We evaluate several multi-attributes detect methods on VG (in Section 4), and the result shows that the multi-binary-classifier method cannot work well on “imperfect” attribute dataset. Because, the multi-binary-classifier method regard all the unlabeled attributes as negative samples to optimize the model, and more likely to predict negative result on the long-tail attribute.

In this paper, we aim to design a GRU-based attribute detection module, with the relationship between attributes taken into consideration, for the attribute detection to avoid the performance reduce in “imperfect” attribute dataset. We consider adopt the CNN for handle GRU for obtaining the attribution, CNN can robustly classify objects, even if they are placed in different directions, which is known as the property of invariance. More specifically, convolutional neural networks are able to stably cope with changes in image translation, image viewing angle, image size, or light conditions. Moreover, CNN can directly use image data as input, no need to manually preprocess the image and additional feature extraction and other complex operations, retaining more data information, by reducing the loss of data features, in the field of video processing cold have achieved very good results. GRU network has fewer network parameters, using the hidden state for information transfer. The convergence time is shorter, which can better meet the real-time needs of human-computer interaction and image processing. The points above has been illustrated in the revised manuscript. The GRU-based attribute detection module has strong processing capabilities for long-term correlation data, and better meet the real-time requirements by fewer network parameters. And we compare our method with other attribute detect methods in VG dataset and the result is shown in Section 4, which achieved the better results using the same datasheet and implementation detail by current methods.

2. Related works

2.1 Attributes detection

The focus of AI adoption is restricted to improving the efficiency of operations or the effectiveness of operations. It can be leveraged to improve the stakeholder experience as well. We have seen continuous growth in the adoption of AI within the IT industry. With the help of artificial intelligence, an organization can get ROI in real time. This means that organizations will see their efforts being paid off. This is one of the most important AI trends. As more data regulations come into play, trust in AI will be pivotal. Most researches in the field of multi-attribute detection are limited to the special object class. For instance, Sarker, Kamal, Jabreel, Rashwan, Banu, Radeva and Puig (2018) is focusing on the attributes of food, Li et al. [9] and Lin, Zheng, Zheng, Wu and Yang (2017) are focusing on the attributes of person, and Huang et al. [6] Mahbub et al. [16] Zhuang et al. [28] are focusing on the attributes of face. These methods have special design for the selected scenario, but their method may not work well in other scene.

Most of the above research were using multi-binary-classifier method (such as Li et al. [9] Lin et al. (2017) Mahbub et al. [16] Zhuang et al. [28]) to detect the attributes. Two of them (Li et al. [9] and Lin et al. (2017)), which have released the source code in github, were selected to compare with our method in Section 4.

We also select two methods (DVRL Liang et al. [12] and ML-GCN Chen et al. [3]), which can simply change the attribute class to adapt to the given scene, for comperation with our method.

The DVRL Liang et al. [12] do not limit to the attribute of a special object class, and use the correlation matrix (between object and attribute) to improve the performance. We also evaluate the above method in Section 4 for comparison.

In the implementation of DVRL Liang et al. [12], the guided graph is used to represent the attributes of each object, which are generated from the dataset before the training by the object-attribute correlation matrix. There are both advantages and disadvantages of this method, through most of the impossible attributes are filtered, several positive attributes also be filtered. And the performance will significant decrease when the object detection module makes mistakes, because they will use the wrong object-attribute correlation to filter attributes at the current condition.

The ML-GCN Chen et al. [3] use Graph Convolutional Networks (GCN) to take the relation between the detection target in to consideration in the multi-label detection task, they take object detection as an example and apply a multi-layers GCN on the objects correlation matrix to capture the relation between objects. The attribute detection is also a multi-label detection task and the above method should be capable, we evaluate this method on attribute detection in Section 4.

Our method is inspired by Yang et al. [24], which use a sequence model for article attribute detection. The author designs a LSTM-base model to detect the attributes from articles in Yang et al. [24]. We think the RNN-based method has potential on attribute detection in the vision field and design a GRU-based method to detect the attributes for each object instance. For comparison with DVRL and ML-GCN, we do not use object-attribute (or attributes) correlation matrix from the dataset explicitly.

2.2 Multi-label classification

The multi-attribute detection is not a simple multi-class classification problem, it is rather a multi-label classification problem, because more than one attributes can be detected from a single object instance and the number of the detected attribute is varieties. The multi-label classification method is an extension of multi-class classification, through a single label that accurately classifies an instance into one or more classes. In a multi-label problem, there is no limit to the number of classes that an instance can be assigned to. Formally, multi-label classification can be seen as a problem finding a model that maps the input x to the binary vector y (assigning a value of 0 or 1 to each element (tag) in y). Several transformation methods exist for multi-label classification. Baseline is a binary correlation approach whose core competency is to train individual binary classifiers for each label independently. In our proposed method, the GRU-based module is used to process the multi-label classification problem and we take the relation between attributes into consideration.

The multi-label classification solutions proposed inspired by deep learning techniques. Some of these solutions rely on autoencoders, which take into account the nature of unsupervised learning. With this approach, the underlying features of any given data can be efficiently extracted, resulting in a well-encoded reduced dataset. Some authors rely on convolutional neural networks (CNNs) to solve prediction problems such as image processing, sound, text, and video. CNNs use a mix of convolutional, pooling, and standard processing layers to capture abstract features that describe the problem. The role of the convolutional layer is to detect the local connections of features from the previous layer, while the pool layer is used to merge reasonably similar features into the higher layers. Meanwhile, other researchers are focusing on the specific application sector of federated learning, namely the data-driven cooperative localization and location data processingï¼Œin order to better understand the attributes during training processes. The Yin et al. [30] presented a centralized implementation, where interested readers can find more details about the measurements, configurations of the DNNs, as well as a diagram of the whole navigation system.

3. Variable-length attributes detection network

Figure 1 shows the whole diagram of our proposed method, which consists of three parts, including feature extraction, multi-attribute detection and the loss computation method. Two primary modules of our network are the feature extract module and GRU-based multi-attribute detection module. We also use a message passing struct from the object detection to the attribute detection branch to refine the attribute features and take the relationship between the object and attributes in to consideration.

Figure 1.

The whole structure of our model VADN. The DCNN generates shared convolutional features, the RPN predicts object regions. Our model can be divided into 3 modules, which are the feature extraction module (to extract the attribute appearance, color and space features), attribute feature refinement module, and the GRU-based multi-attribute detection module.

3.1 Feature extraction

In the proposed method, the appearance features for each object are obtained from the deep convolutional network model (DCNN) with ROI-pooling layer. Because the DCNN is pretrained for object detection, we cannot ensure the extracted features can satisfy the need of attribute detection. Moreover, the ROI-pooling layer will lose most of the space features, because it resized the input features to the fixed size. We consider that the color and space features are especially useful in the attribute detection, thus we design two special modules for color and space feature extraction (as shown in Fig. 1) to extend the extracted appearance features from the pretrained feature extractor DCNN.

3.1.1 Attribute appearance features

We directly use the pooled features from DCNN as the input features, and then use two continuous fully connected layers (FC_1 and FC_2) with ReLU unit to extract appearance feature vector. These two fully connected layers both have a size of 1024, and the extracted appearance feature vector is in length of 1024.

3.1.2 Color features

We first use two convolution layers, which have 8 and 32 convolution kernels respectively, to extract color features. Because the color features should be space independent, these two convolution layser both have 1*1 convolution kernels. And then we use Max-Pooling and Mean-Pooling to do the down-sample. After that, a ROI-pooling layer is used to extract color feature map for each object instance. Finally, we use a fully connected layer with ReLU unit to generate the color feature vector, which is in length of 128.

3.1.3 Space features

The space features consist of local space features and global space features. As shown in Fig. 2, we first generate a 6-tuple from the object ROI and use a fully connected layer (FC_space_local) to embed the 6-tuple to the local space features. We then use a fully connected layer (FC_space_global) to embed the local space features to the global space feature’s space. Because the number of object proposals is variable, the max function is used to mix the global space features from different object instance and generate the global space feature vector, finally.

The 6-tuple is defined as $(\frac{x_{i}^{o}}{w^{I}+1},\frac{y_{i}^{o}}{h^{I}+1},\frac{w_{i}^{o}}{w^{I}+1}% ,\frac{h_{i}^{o}}{h^{I}+1},\log\left({1+\frac{w_{i}^{o}}{h_{i}^{o}+1}}\right),% w_{i}^{o}{\ast}h_{i}^{o})$ , where $x_{i}^{o}$ , $y_{i}^{o}$ are the center position of the object box $i$ ,and $w_{i}^{o}$ , $h_{i}^{o}$ are width and height of object box $i$ . and $w^{I}$ , $h^{I}$ are width and height of the whole image, respectively.

Figure 2.

Space feature extraction module, which uses object ROIs to extract local and global space features.

3.1.4 Refinement process using object appearance features

The object and attribute appearance features are both extracted from the given image but in different branch, and these two types of features have different focus. We consider that there are relationships between the object and attributes and use the object features to refine the attribute features. During the feature refinement process, we do not directly concatenate these two types of appearance features together. Instead, a gate function is used to refine the attribute appearance features with the help of object appearance features. For each object instance, we use the object appearance features, which are extracted from the object detection branch, to refine the attribute appearance features. The features from “object to attribute” are simply defined as follows:

$\displaystyle\tilde{x}^{(0\to a)}_{i}=\sigma_{<a,0>}(x^{(a)}_{i},x^{(0)}_{i})x% ^{(0)}_{i}$ (1)

Where $\tilde{x}^{(0\to a)}_{i}$ denotes the gated features from object to attribute for object instance $i$ . $x_{i}^{(a)}$ and $x_{i}^{(o)}$ denote the attribute appearance features and object appearance features of object $i$ , respectively. $\sigma_{\langle{a,o}\rangle}$ is a gate function to control the weight of appearance object features $x_{i}^{(o)}$ . $\sigma_{\langle{a,o}\rangle}$ is defined as follows:

$\displaystyle\sigma_{\left\langle{a,o}\right\rangle}\left({x_{i}^{\left(a% \right)},x_{i}^{\left(o\right)}}\right)=\mathop{\sum}\limits_{g=1}^{G}\textit{% sigmoid}\left({w_{\left\langle{a,o}\right\rangle}^{\left({g}\right)}\cdot\left% [{x_{i}^{\left(a\right)},x_{i}^{\left(o\right)}}\right]}\right)$ (2)

where $G$ denotes the number of the gate templates for the input features, we use 128 empirically. Each $g$ of $w_{\left\langle{a,o}\right\rangle}^{\left(g\right)}$ indicates a special feature template. The sigmoid function is used to limit the value domain. The more the input features $\left[{x_{i}^{\left(a\right)},x_{i}^{\left(o\right)}}\right]$ match the template, the more the weight will be closer to 1. These templates are automatically learned during the training phase.

We then use gated features $\tilde{x}^{(0\to a)}_{i}$ to refine the attribute appearance features, which is defined as follows:

$\displaystyle x_{i,\textit{refined}}^{\left(a\right)}=x_{i,\textit{last}}^{% \left(a\right)}+W^{o\to a}\cdot\textit{ReLU}\left(\tilde{x}^{(0\to a)}_{i,% \textit{last}}\right)$ (3)

Where $W^{o\to a}$ is a fully connected layer, ReLU is a general Relu unit.

3.2 GRU-based variable-length attributes detection module

We generate the attribute word vector by Pennington et al. [18] before the training phase, and the attribute word vector will be used in the GRU-base multiattribute detection module. The precomputed word vector dictionary is computed from 6B token and 400K vocab by Pennington et al. [18], and the similarity of the two given attribute word vectors can represent the correlation of the two attributes, generally.

In our proposed method, the variable-length attributes detection module works on the following principle: At state $t$ , we predict an attribute $A_{t}$ with the attribute features and the past attribute results in the consideration. Then, we want our model more likely to detect other high related attributes at the next step $t+1$ . Therefore, we input the word vector of attribute $A_{t}$ and use the GRU unit to refine the attribute features. Because the relationship of attributes is implicated in the word vector, and we design a special strategy to arrange the target attribute sequence, our model can predict the high-related attributes at each step.

In order to satisfy such principle, we design a GRU based model as shown in Fig. 3. The GRU-based multi-attribute detection module consists of four parts. The FC_attris a fully connected layer, which transforms the concatenated input features to attribute feature space. The transformed attribute feature ( $h_{t}$ ) not only include the vision features, but also include the embedded attribute results in the past. The Encoder in 3 search the word vector by the given attribute in the precomputed word vector dictionary, and output the word vector $x_{t}$ to the GRU. The classifier predicts the most likely attribute $A_{t}$ at state $t$ using features $h_{t}$ . After the attribute $A_{t}$ was detected, we use the corresponding word vector of the attrib ute $A_{t}$ as $x_{t}$ to refine the attribute features $h_{t}$ in the $t$ state, then output the refined attribute features $h_{t+1}$ in the $t+1$ state to detect other attributes.

Figure 3.

GRU-based multi-attribute detection module. Attributes are detected at each step.

Figure 4.

Dimension reduction display of several example attribute word vector. The red circle is the start node, the connected nodes are labeled attributes for a given object instance, the edges are the unique way from the start node “red” and pass the other labeled attribute nodes by the greed method.

3.3 Attributes loss function

As we have mentioned in Section 3.2, we regard the target attributes for an object as a digraph (as shown in 3), and the edges are the relationship among these attributes. There are edges between all the pairs of the attribute nodes in the digraph. We regard the word vector similarity as the strength of the relationship. The stronger the relationship between the two attributes, the high value of the edge weight will be. Because the digraph is static, we can find a unique way from the given start attribute node and pass the other given nodes with the sum of the edge weight be biggest. For example, as we shown in 4, we assume that there is an object instance with four labeled attributes “wooden, large, brown, red”, we select “red” as the start node and use the greed method (select highest edge from the last node step by step) among these attribute nodes to find the unique way which pass themselves. Then, we select other nodes “wooden”, “large”, “brown” as start node and repeat the above steps, respectively. Finally, we have four unique ways of these four nodes, and the 4 target attribute sequences are generated.

In the training phase, we design a special strategy to select one target attribute sequence from the target attribute sequences and compute the loss, the strategy is described as below:

1.
There are no attribute labeled in the annotation. In this condition, we consider that these object’s attributes are high probability omitted by the annotator. Thus, we simply skip this object instance in the attribute loss computing process.
2.
The attribute annotation is not empty and all the labeled attributes are appeared in the predicted attribute sequence. In this condition, we consider that all the labeled attributes are detected, but several omitted attributes may still exist. We try to keep the predicted result in this condition and slightly punish the unlabeled attributes, which are detected, to $<\textit{unkonwn}>$ tag. Because all the target attributes are predicted, the predict order is doesn’t matter, we arrange the target sequence to match the predicted sequence (as shown in Fig. 5) to minimize the loss. For the unlabeled attributes in the predicted attribute sequence, the omitted attributes and the wrong attributes are coexisting. If the unlabeled attributes are highly by annotator’s omit and there are number of positive examples in the other training images, these attributes can beat the slight punish and stay in the predicted attribute sequence for these object instances. Otherwise, slight punish also can correct the prediction.

Figure 5.
Example of condition 2.

3.
The attribute annotation is not empty and the first predicted attribute is not appeared in the attribute labels. This condition is strong possibility in the first few epochs of the training phase. We randomly select one target attribute sequence from the target attribute sequences and compute the loss.
4.
The attribute annotation is not empty and the first predicted attribute is appeared in the attribute labels. This condition is a general state in the training phase. We select the target attribute sequence, which is starting with the first predicted attribute, from the target attribute sequences and compute the loss. When we determine the target attribute sequence, the target sequences are padded to fix-length sequence for more easily computing the loss from batch of the object instances, and the attribute loss is computed as the following:

In the condition 3 and condition 4:

$\displaystyle\textit{loss}_{\textit{attr}}=\frac{\sum_{i=1}^{A}C\left({\textit% {Seq}_{\textit{predict}}^{i},\textit{Seq}_{\textit{target}}^{i}}\right)\cdot% \textit{Mask}_{i}}{\sum_{i=1}^{A}\cdot\textit{Mask}_{i}}$ (4)

where $C\left(\ldots\right)$ is a general cross entropy loss function, $\textit{loss}_{\textit{attr}}$ denotes the loss of attributes detection, $A$ is the maximum number of attributes per object, $\textit{Seq}_{\textit{predict}}^{i}$ and $\textit{Seq}_{\textit{target}}^{i}$ denote the $i$ -th attribute in the predict sequence and the target sequence, respectively. The Mask is a mask matrix, which is defined as following:

$\displaystyle\textit{Mask}_{i}=\left\{\begin{array}[]{ll}1,&\textit{if the $i$% -- th attribute is labeled attribute}\\ 0,&\textit{if the $i$ -- th attribute is padding}\\ \end{array}\right.$ (5)

In the condition 2:

$\displaystyle\textit{loss}_{\textit{attr}}=\frac{\sum_{i=1}^{A}C\left({\textit% {Seq}_{\textit{predict}}^{i},\textit{Seq}_{\textit{target}}^{i}}\right)\cdot% \textit{Mask}_{i}}{\sum_{i=1}^{A}\cdot\textit{Mask}_{i}}+\alpha\textit{loss}_{% \textit{punish}}$ (6)

$\displaystyle\textit{loss}_{\textit{punish}}=\frac{\sum_{i=1}^{A}C\left({% \textit{Seq}_{\textit{predict}}^{i},\textit{Token}_{\left\langle{\textit{% unknown}}\right\rangle}}\right)\cdot\textit{Mask}_{i}^{\textit{punish}}}{\sum_% {i=1}^{A}\textit{Mask}_{i}^{\textit{punish}}}$ (7)

where $C\left(\ldots\right)$ is a general cross entropy loss function, $\textit{loss}_{\textit{attr}}$ denotes the loss of attributes detection, $\textit{loss}_{\textit{punish}}$ denotes the punish of the unlabeled attributes, $A$ is the maximum number of attributes per object, $\textit{Seq}_{\textit{predict}}^{i}$ and $\textit{Seq}_{\textit{target}}^{i}$ denote the $i$ -th attribute in the predict sequence and the target sequence, respectively. The $\textit{Token}_{\left\langle{\textit{unknown}}\right\rangle}$ denotes the $\left\langle\textit{unknown}\right\rangle$ tag. The Mask and the $\textit{Mask}^{\textit{punish}}$ are mask matrixes, which are defined as following:

$\displaystyle\textit{Mask}_{i}=\left\{{{\begin{array}[]{ll}1,&\text{if the $i$% -th attribute is labeled attribute}\\ 0,&\text{otherwise}\\ \end{array}}}\right.$ (8) $\displaystyle\textit{Mask}_{i}=\left\{{{\begin{array}[]{ll}1,&\text{if the $i$% -th attribute is}\left\langle\textit{unknown}\right\rangle\\ 0,&\text{otherwise}\\ \end{array}}}\right.$ (9)

4. Experiments

4.1 Dataset

In our experiments, we use VG (Visual Genome) Krishna et al. [8] as the primary dataset. There are 108249 images in VG dataset, and the number of attribute instances is 1,670,182. The attribute annotation in VG dataset are not standard in many cases. For instance, some attributes are merged as a single attribute (example: “black” and “white” are always labeled as “black and white”), several attributes are synonymous but their words are different (example: “grey” and “gray” has the same meaning but they are labeled as two different attributes), and some attributes are labeled as shott sentences (example: “playing tennis”).

Therefore, we use WordNet Miller (1998) and NLTK Bird and Loper [2] to normalize the attributes in VG dataset. Then, we random select 46164 images for training and 10000 images for testing. We selected the most 100 frequency attribute labels during the preprocess. After the preprocessing of attributes, we have 343,838 attribute instances in total, and 6.12 attributes per image 1. The training dataset has 280k attributes in total and 6.1 attributes per image. The test dataset has 61k attributes in total and 6.1 attributes per image. In other dedicated attributes detection dataset, such as CUB-200, they have a standard annotation principle during the annotation process and the attribute annotation is binary-vector. We can ensure all the labeled attributes are positive or negative samples, therefore nothing else need to do. CUB-200 bird dataset Wah et al. [23], which has 200 object classes and 313 attribute classes, consists of 11788 images with 5994 and 5794 for training and testing, respectively. It has 31.48 attributes per image on average.

CUB-200 bird dataset Wah et al. [23], which has 200 object classes and 313 attribute classes, consists of 11788 images with 5994 and 5794 for training and testing, respectively. It has 31.48 attributes per image on average and the attribute annotation is binary.

4.2 Implementation details

MSDN Li et al. [10] is a multi-tasks detection network, which can perform object detection, relationship detection and caption generation jointly. In our experiments, we use MSDN as our base model only for object detection, and the parameters in MSDN are fixed during the training phase. In order to compare our model with other method effectively, such as DVRL and MSDN, we select VGG16 as base feature extraction module. The parameters in the additional layers are normally initialized and fine-tuned during the training phase. The Adam is used to optimize our model during the training phase. The learning rate is starting with 0.001 and will be reduced by a factor of 0.2 after each epoch. Moreover, our model can be trained end-to-end.

To prove all of our designs are working well in the attribute detection task and no invalid module is existing, we do ablative studies in Section 4.4. sFour variant models are designed as below:

1.
The “VADN_base” is the base model in the ablative study, only the attribute appearance feature is used to detect the attribute.
2.
The “VADN_C” has an extra color feature extract module compared with the “VADN_base”.
3.
The “VADN_CS” has an extra space feature extract module compared with the “VADN_C”.
4.
The “VADN_CSR” has an extra feature refinement module compared with the “VADN_CS”.

We also design a variant model (VADN_freq) without our loss compute method as we describe in Section 3, and we simply regard the ascend order of the frequency for attributes as an alternative order to verify the effect of our loss computation method.

For comparison with the existing methods, we retrain four existing attributes detect models, which have released on git hub for attribute detection task, on VG dataset for comparison. To eliminate the data difference among these methods, all the models are using a same pre-trained RPN model for object region proposal. The Deep MAR Li et al. [9] and APR Lin et al. (2017) are designed for person attribute detection task, we regard the object as person and retrain these two models on VG dataset. The DVRL Liang et al. [12] is designed for attribute detection on image, and ML-GCN Chen et al. [3] is introduced for multilabel detection, we simply regard the attributes for the given object region as the target without any modify on the model structure.

In the above comparison, we only train the attribute detect branch, but the object detects branch and the shared convolution head are fixed. We further train our model VADN_CSR with our object detection model jointly to finetune the weights, and we call the fine-tuned model as our final model VADN_final. After the fine-tune process, our model has a slight improvement on both the object detect and the attribute detect tasks. Figure 6 gives the scene graph visualization results of our proposed method.
4.3 Evaluation metrics

Due to the imperfect attribute annotations in the VG dataset, the accuracy is not an effective metric for multi-attribute detection. The predicted attributes, which do not appear in the ground truth, may still be correct attributes. Considering that 66% of the object are not annotated with attributes in VG, the number of potential attributes can be huge. We use F1-score, which considers recall and precision rates both, as a primary metric to evaluate our model. Note that we ignore the object instances which have no attributes annotations. We also use the average attributes number per object (Attr per object) as a metric to show the effectiveness of our attribute features and loss methods.

The WmAP is used in FineTag Zakizadeh et al. [26] to evaluate their model on CUB200, the higher, the better.

4.4 Ablative analysis of feature type

To analyze the performance of each module, we perform an ablative analysis. We compare our four variant models for the multi-attribute detection task using recall, precision and F1-score metrics. The results are shown in Table 1.

Color features: As shown in Table 1, our model VADN_C has 1.41% and 1.38% improvement compared with model VADN_base using recall and precision rate metrics, respectively. The F1-score also has 1.4% improvement.

Space features: As shown in Table 1, our model VADN_CS has 0.1% and 1.29% improvement compared with model VADN_C on recall and precision rate, respectively. And the F1-score also has 0.69% improvement. The improvement of precision metric benefit from the space features.

Refinement process: As shown in Table 1, our model VADN_CSR achieves 0.5% improvement compared with VADN_CS model while using recall rate metrics. The F1- score also has 0.25% improvement. The improvement of recall metric benefit from the space features.

The refinement process from object features to attribute features is effective for attributes detection, we think the primary reason is that attribute features focus on the whole object region and object features are focusing on the foreground. The refinement process can filter several background features for attributes detection and the refined features are more suitable for attribute detection.

Table 1
Ablative analysis of the variant models. A, C and S are three modules to extract attribute appearance features, color features and space feature, respectively. The R is the refinement module, which uses the object appearance features to refine the attribute appearance features

Model	Model setting				Attribute task
	A	C	S	R	Recall rate	Precision rate	F1 score
VADN_base	$\surd$				32.97	32.93	32.95
VADN_C	$\surd$	$\surd$			34.38	34.31	34.35
VADN_CS	$\surd$	$\surd$	$\surd$		34.48	35.60	35.04
VADN_CSR	$\surd$	$\surd$	$\surd$	$\surd$	34.98	35.60	35.29

Figure 6.

(a) is the image with object and attribute labels. (b) is the THREE-semantic-level scene graph. There are many things on the table, such as food, bowl and glass. Because the plate is not detected, some relationships, which regard sandwich as object or subject, are unsuitable.

4.5 Comparison of arrange method

We compare our arrange method with a simple frequency order arrangement in the rearrange process which we discuss in Section 3.3. The VADN_freq is 7.22% lower than our rearrange method on the attribute detection task. Thus, our consideration, the difficulties of the detection for different attribute is not a simple ascend order of attribute frequency in the dataset, is reasonable. Our method can find a suitably order of the attributes for each object, and tend to predict more attributes for each object instance, which is suitable in “imperfect” attribute dataset. Because the unlabeled attributes are also may be appeared for the object instance.

Table 2
Evaluation results of the different arrange method

	Recall rate	Attr per object
VADN_freq	27.76	0.92
VADN	34.98	2.27

4.6 Comparison with existing works

4.6.1 Attributes detection task

The Deep MAR Li et al. [9] and APR Lin et al. (2017) are multi-binary-classifier method, which are widely used in attribute detection. But this method is not suitable for “imperfect” attribute dataset, because these method regard all the unlabeled attributes as negative samples in the training phase. And, our VADN has 8.51% and 18.1% improvement on F1-score compared to Deep MAR and APR, respectively.

The DVRL Liang et al. [12] use the object-attribute correlation matrix to improve the performance on attribute detection. The attribute detection in DVRL is depend on the object detection excessively, and object detect error will spread to the attribute detection. The attribute detect module in our method is partially based on the object features, and our model has 13.19% improvement on the on the F1-score compared to DVRL. Through the DVRL has a high recall performance, they predict more error attributes and have a lower comprehensive performance. The ML-GCN Chen et al. [3] is design for multi-label detection, but is not suitable for attributes detection. Because the correlation between attributes is complex in fact, and the attributes correlation matrix cannot represent the relation between attributes in the images completely. Moreover, the VG dataset is “imperfect”, thus the ML-GCN has low performance.

In another dataset CUB-200, our VADN also achieves 1.3% improvement compared to FineTag Zakizadeh et al. [26] in CUB-200 dataset.

Table 3
Evaluation results of existing methods on VG and CUB-200 datasets. The VGG16+rank and FineTag are both provided by Zakizadeh et al. [26]

	VG		CUB
	Recall rate	F1 SCORE	WmAP
VGG16+rank_momentum	$-$	$-$	44.0
VGG16+rank_Adam	$-$	$-$	50.0
FineTag_momentum	$-$	$-$	46.0
FineTag_Adam	$-$	$-$	52.0
APR Lin et al. (2017)	10.49	17.19	$-$
ML-GCN Chen et al. [3]	34.80	2.18	$-$
DeepMAR Li et al. [9]	30.03	26.78	$-$
DVRL Liang et al. [12]	37.27	22.10	$-$
VADN_CSR	34.98	35.29	53.3

4.6.2 Object and relationship detection tasks

The object and relationship detection are not the goal in our design, but we also find these two tasks have a slight improvement with the help of attribute detection. We integrate our model in to MSDN and fine-tune the whole model. As shown in Table 4, VADN has 0.21% and 0.10% improvement for the object detection task using Recall@64 and MAP metrics, respectively. And VADN has 0.19% and 0.13% improvement in relationship detection task using Recall@50 and Recall@100 metrics, respectively.

Table 4
Results of object and relationship detection. The current version of MSDN is our base model, and VADN is our final model

	Ojbect task		Relationship task
	Rec@64	MAP	Rec@50	Rec@100
MSDN Li et al. [10]	$-$	7.43	10.72	14.22
MSDN (current version)	46.68	16.29	11.71	14.09
VADN_final	46.80	16.39	11.73	14.09

5. Conclusion

This paper addresses the problems of multi-attribute detection for imperfect attribute dataset. We propose a special loss function to transform the multi-attribute detection problem to a sequence predict problem. Then, we design a GRU-based attributes detection module to solve the multi-attribute detection problem. Three different types of features are used for attribute detection and a refinement process is used to further improve the performance. We perform ablative studies and several comparisons to show the performance of all our design. Furthermore, we compare our model VADN with several existed methods to prove the performance of our method. In the further, firstly, we consider, by using a multi-agent reinforcement learning algorithm to select the optimal aggregation threshold across different relationships, to optimize the model from the fusion weights based on cross-validation to guarantee a reliable and robust prediction performance. Secondly, Flexible design of end-to-end networks and loss functions is necessary in order to optimize the combination of branch settings and attention mechanisms. Thirdly, we will combine our attribute detect model with other high semantic level tasks (such as question & answer and caption) to further improve the performance.

References

Antol

Agrawal

Mitchell

Batra

Lawrence Zitnick

Parikh

. Vqa: Visual question answering, in: Proceedings of the IEEE international conference on computer vision, 2015; pp 2425-2433.

Bird

Loper

, Nltk: the natural language toolkit, in: Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, Association for Computational Linguistics. 2004, p. 31.

Chen

Wei

Wang

Guo

. sMulti-label image recognition with graph convolutional networks, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019; pp. 5172-5181.

Dai

Zhang

Lin

, Detecting visual relationships with deep relational networks, in: Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, IEEE. 2017; pp. 3298-3308.

Farhadi

Endres

Hoiem

Forsyth

. Describing objects by their attributes, in: Computer Vision and Pattern Recognition, 2009. CVPR 2009 IEEE Conference on, IEEE. 2009; pp. 1778-1785.

Huang

Loy

Tang

. Deep imbalanced learning for face recognition and attribute prediction. IEEE Trans Pattern Anal Mach Intell. 1806.00194; 2018.

Johnson

Karpathy

Fei-Fei

. Densecap: Fully convolutional localization networks for dense captioning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016; pp. 4565-4574.

Krishna

Zhu

Groth

Johnson

Hata

Kravitz

Chen

Kalantidis

Shamma

, et al., Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision. 2017; 123: 32-73.

Chen

Huang

. Multi-attribute learning for pedestrian attribute recognition in surveillance scenarios, in: ACPR, 2015; pp. 111-115.

10.

Ouyang

Zhou

Wang

. Scene graph generation from objects, phrases and region captions, IEEE International Conference on Computer Vision, 2017.

11.

Liang

Zhang

Gan

Xing

. Recurrent topic-transition gan for visual paragraph generation. 2017 IEEE International Conference on Computer Vision (ICCV), 2017; pp. 3382-3391.

12.

Liang

Lee

Xing

. Deep variation-structured reinforcement learning for visual relationship and attribute detection, in: Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, IEEE. 2017; pp. 4408-4417.

13.

Liang

Shen

Feng

Lin

Yan

. Semantic object parsing with graph lstm, in: European Conference on Computer Vision, Springer. 2016; pp. 125-143.

14.

Chikontwe

Lee

. Deep Multi-Task Network for Learning Person Identity and Attributes, in IEEE Access, 2018; 6: 60801-60811.

15.

Long

Shelhamer

Darrell

. Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015; pp 3431-3440.

16.

Mahbub

Sarkar

Chellappa

. Segment-based methods for facial attribute detection from partial faces. IEEE Transactions on Affective Computing, 2018.

17.

Huang

Chen

Ouyang

Wan

Xue

. Image Captioning With End-to-End Attribute Detection and Subsequent Attributes Prediction, in IEEE Transactions on Image Processing, 2020; pp. 4013-4026.

18.

Pennington

Socher

Manning

. Glove: Global vectors for word representation, ins: Empirical Methods in Natural Language Processing (EMNLP), 2014; pp. 1532-1543.

19.

Redmon

Divvala

Girshick

Farhadi

. You only look once: Unified, real-time object detection, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

20.

Ren

Girshick

Sun

. Faster r-cnn: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems, 2015; pp. 91-99.

21.

Gong

Wang

Cheng

Wang

. Zero-Shot Classification Based on Multitask Mixed Attribute Relations and Attribute-Specific Features, in IEEE Transactions on Cognitive and Developmental Systems, 2020,

22.

Taherkhani

Nasrabadi

Dawson

. A Deep Face Identification Network Enhanced by Facial Attributes Prediction, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018; pp. 666-6667.

23.

Wah

Branson

Welinder

Perona

Belongie

. Multiclass recognition and part localization with humans in the loop, 2011 International Conference on Computer Vision, 2011; pp. 2524-2531.

24.

Yang

Sun

Wang

. SGM: sequence generation model for multi-label classification, in: Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20–26, 2018; pp. 3915-3926.

25.

Mei

Rui

. Multi-level attention networks for visual question answering, in: Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, IEEE. 2017; pp. 4187-4195.

26.

Zakizadeh

Sasdelli

Qian

Vazquez

. Finetag: Multiattribute classification at fine-grained level in images, 2018.

27.

Zhang

Kyaw

Chang

S-F

Chua

T-S

. Visual Translation Embedding Network for Visual Relation Detection, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017; pp. 3107-3115.

28.

Zhuang

Yan

Chen

Wang

. Multi-task Learning of Cascaded CNN for Facial Attribute Classification, 24th International Conference on Pattern Recognition (ICPR), 2018; pp. 2069-2074.

29.

Yin

Lin

Cui

. Wireless Traffic Prediction With Scalable Gaussian Process: Framework, Algorithms, and Verification, in: IEEE Journal on Selected Areas in Communications, pp. 1291-1306.

30.

Yin

Lin

Kong

. Sergios Theodoridis, Shuguang Cui, FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing, in: IEEE Open Journal of Signal Processing, 2020; pp. 187-215.

Variable-length sequence model for attribute detection in the image

Abstract

Keywords

1. Introduction

2. Related works

2.1 Attributes detection

2.2 Multi-label classification

3. Variable-length attributes detection network

3.1.1 Attribute appearance features

3.1.2 Color features

3.1.3 Space features

4.1 Dataset

4.2 Implementation details

4.4 Ablative analysis of feature type

Table 1 Ablative analysis of the variant models. A, C and S are three modules to extract attribute appearance features, color features and space feature, respectively. The R is the refinement module, which uses the object appearance features to refine the attribute appearance features

Table 2 Evaluation results of the different arrange method

4.6.1 Attributes detection task

Table 3 Evaluation results of existing methods on VG and CUB-200 datasets. The VGG16+rank and FineTag are both provided by Zakizadeh et al. [26]

Table 4 Results of object and relationship detection. The current version of MSDN is our base model, and VADN is our final model

References

Table 1
Ablative analysis of the variant models. A, C and S are three modules to extract attribute appearance features, color features and space feature, respectively. The R is the refinement module, which uses the object appearance features to refine the attribute appearance features

Table 2
Evaluation results of the different arrange method

Table 3
Evaluation results of existing methods on VG and CUB-200 datasets. The VGG16+rank and FineTag are both provided by Zakizadeh et al. [26]

Table 4
Results of object and relationship detection. The current version of MSDN is our base model, and VADN is our final model