Transferring learning from multi-person tracking to person re-identification

Abstract

Learning to discriminate, whether two person-images correspond to the same person or not, is a daunting challenge when only two images per person are available. This task is called single-shot person re-identification (re-id) and it assumes that each one of the two available images was captured from a different camera view entailing variations in pose, resolution, scale, illumination and background. Addressing this task through supervised training of a deep convolutional neural network is susceptible to model overfitting due to the critical lack of enough labelled data. This paper proposes to exploit the transference of learning previously acquired from a multi-object-tracking (MOT) domain. In this context, a unique deep triplet architecture has been trained on both domains. Six different levels of transfer learning have been implemented and evaluated, proving that the transference of leaning from a different domain remarkably increases the re-id performance. Experimental results validate accuracy and robustness of the proposed method as comparable to other state-of-the-art techniques. These results also confirm that, despite the data problem, deep learning is also applicable to the single-shot re-id task.

Keywords

Transfer learning deep learning person re-identification multi-object tracking pair-wise binary classification

1. Introduction

Person re-identification (re-id) consists of matching people across non-overlapping camera views, at different locations and time. It has become one of the most studied tasks in video surveillance since many other applications, like tracking or behaviour analysis, rely on the person re-id performance [1].

Even though many research efforts from the computer vision field have focused on solving the re-id problem [2], it remains unsolved [3]. This is because, in unconstrained scenarios, a person’s appearance often changes dramatically across camera views due to changes in body pose, view angle, occlusion and illumination conditions. These changes cause intra-class variations, that means quite different representations of the same identity. Moreover, the presence of different people with a similar appearance introduces inter-class ambiguities, i.e. similar representations corresponding to different people. The intra-class ambiguities tend to distance the representations of a person in the feature space. Conversely, inter-class ambiguities close the features corresponding to different people. Therefore, the clustering of the features according to the identity that they are rendering is a hard task, turning the recognition of the identities into a daunting challenge.

Recently, the use of deep re-id models has been boosted by the success of Convolutional Neural Networks (CNNs) in various vision problems, like image classification [4], objects detection [5], face identification [6], transportation [7, 8, 9], structures analysis [10, 11, 12, 13, 14, 15, 16], medical diagnosing [17, 18], or even in computational modelling [19, 20], combinatorial optimization [21], and big data management [22]. Most of the networks used in person re-id treat this task as a pairwise binary classification to discriminate between matched and mismatched pairs of images and they are trained with a supervised learning framework, like the work presented in [23].

However, person re-identification poses a unique challenge to deep supervised learning, since a deep model with millions of parameters must be learnt from a small training set. Two types of re-id datasets can be found: multi-shot recognition [24], where a tracklet of every individual (i.e. small sequence of images) is available for each camera view; and single shot recognition, where only one image per person and per view is used. Figure 1 shows examples of matched pairs (in each column) from the single-shot PRID2011 dataset [25]. Every pair is formed by one image captured from each one of the two different cameras views, a and b. This paper is focused on the single-shot case, where the overfitting of the model is particularly acute due to the small size of the available labelled data.

Figure 1.

Examples of matched pairs belonging to the single shot PRID2011 dataset.

The objective of the presented work is to address the re-id data problem in deep models learning. Given the requirement of a large amount of labelled data to learn a non-overfitted model, able to generalise the solution with unknown samples, and the small size of single-shot re-id datasets, the transference of learning from a different domain has been adopted.

In contrast to the use of other re-id datasets or classification datasets, this paper proposes the use of a Multi-object tracking (MOT) dataset to pre-train a re-id model, which has been subsequently fine-tuned with the re-id target dataset.

With this approach, the re-id network architecture does not require any modification to transfer learning from people tracking to re-identification task. Instead of that, we propose the adaptation of the MOT dataset to feed a re-id network. Therefore, a new source dataset has been created by extracting people images from MOT sequences.

The use of the proposed cross-sequence dataset provides certain benefits to the re-id model learning. First, the unbalanced nature of the re-id data, where there are only two instances of a certain individual against the high number of different people, is attenuated. In the new dataset, many detections of the same person are extracted, making easier the convergence of the model learning process.

In addition, samples from different MOT sequences have been compared in the training. This setting helps to avoid the dependence on the characteristics of a certain camera view, and consequently, to avoid the negative transfer of learning.

Therefore, this paper addresses the single-shot reidentification problem by transferring the similarity degree of appearance learnt from MOT datasets, and its main contributions are: i) the creation of a new source dataset to pre-train a person re-id model from MOT sequences, by designing and using a data generation tool, which has been publicly delivered;1 ii) the implementation and evaluation of different stages of transfer learning, according to different values for the number of layers that are initialized with the pre-trained weights, and for the number of layers that are fine-tuned on the target re-id dataset. This prior analysis provides experienced knowledge about the best configuration for transferring learning from MOT to re-id domain, which is applicable beyond the presented experiments. iii) Finally, the last contribution is the learning of a degree of appearance similarity measurement applicable to both, tracking and re-id tasks through a unique deep network architecture.

The pair-wise binary classification performance of the pre-trained model has been tested on the MOT17 dataset. Furthermore, the effects of the different stages of transfer learning on the final re-id capacity have been evaluated over two of the most challenging re-id datasets: PRID20111 [25], and VIPeR [26], proving the effectiveness of the proposed approach.

The rest of the paper is organized as follows: Section 2 presents the existing related work. The used learning model is introduced in Section 3 and Section 4 describes the different conducted stages of transfer learning. Section 5 evaluates the learning process evolution and presents the experimental results, and some concluding remarks are given in Section 6.

2. Related work

Traditionally, re-identification problem has been faced through two different types of approaches: first, those which enhance the design of distinctive feature representations, like graph representations [27] and spatial co-occurrence representations [28]. And second, the methods to optimally combine and quantify visual features, such as boosting with AdaBoost [26], Ranking Support Vector Machines (Rank-SVM) [29], Probabilistic Relative Distance Comparison (PRDC) [30], or Metric Learning algorithms, such as Linear Discriminant Analysis (LDA) [31] and Logistic Discriminant Metric Learning (LDML) [32].

These mentioned methods address the learning of a global weighting that reflects the stability of each feature component across two cameras, so they can be grouped under the paradigm of Global Feature Importance (GFI) methods. On contrary, in [33], a Prototype-Sensitive Feature Importance based method is proposed to adaptively weight features according to different clusters of population.

On the other hand, a new approach has been recently adopted. This, which has been inspired by the success of deep Convolutional Neural Networks (CNNs) in image classification [4], is based on the learning of Deep re-id models.

In general, two types of CNN models are commonly employed in computer vision. The first type is the classification model, which is used in image classification [4] and object detection [34]. The second is the Siamese model using image pairs [35] or triplets [36] as input. The first two works in re-id to use deep learning [37, 38], employed a siamese neural network [39] to determine if a pair of images belong to the same identity. Yi et al. [38] add an additional cost function in the network, while Li et al. [37] use a finer body partitioning. Siamese Networks consist of two deep CNNs branches, sharing parameters and joined in the last layer, where the loss function performs a pairwise verification. Later, Ahmed et al. [40] improved the siamese model by computing the cross-input neighbourhood difference features, which compares the features from one input image to features in neighbouring locations of the other image.

Another variation of the Siamese network is the one known as Triplet model, presented in [36], where face re-identification is addressed. This model has been widely used to automatically find salient high-level representations from raw images, like in the work presented in [41]. The work presented in [42] demonstrated how triplet model helps to alleviate the unbalance nature of the re-id datasets, by allowing multiple triplet combinations of re-id samples.

In the Triplet model, each training input is a set of three samples, two are rendering the same person and the third one, a different identity. Therefore, this model allows the comparison between a matched and a mismatched pair, from each triplet input, so the objective function can maximize the relative distance between them. For that reason, this has been the learning architecture used in this work to train VGG16 networks [43]. Hence, each branch of the proposed Triplet model is a VGG16 net. Other triplet approaches that chose the VGG16 model to learnt representative features are the presented in [44, 23] to solve face and person re-identification problems, respectively. This last work also adds LSTM networks, since they adopted a multi-shot re-id approach, in contrast to the single shot re-id that is presented in this work. In [45] a combination of LSTM and VGG networks are used solve people tracking.

However, addressing the re-id problem by means of Deep learning involves an exceptional challenge, because of the lack of training data. Single-shot re-id datasets, such as PRID2011 [25] or VIPeR [26], provide only two images for each identity so currently most CNN-based re-id methods focus on the Siamese and Triplet model, treating the re-id task as a pairwise binary classification.

The requirement of a large amount of labelled data to learn a non-overfitted model, able to generalise the solution with unknown samples, severely limits its scalability in real applications. To overcome this limitation, some works employ unlabelled data from video-surveillance people detection [46, 47]. Nevertheless, without labelled matching pairs across camera views, existing unsupervised models are not capable of learning what makes a person recognisable under severe appearance changes. Therefore, the matching performance of unsupervised approaches is generally lower than that for supervised methods.

Given the inefficiency of unsupervised methods in re-id and the insufficiency of labelled data to apply supervised ones, transferring the models learned from a larger auxiliary dataset becomes necessary. The literature presents transfer learning, a.k.a. domain adaptation [48], as a widely applied tool in deep re-id model learning, where the target task is short of labelled data. The most common deep transfer learning strategy is fine-tuning [49]: firstly, a base network is trained on a large source data, then the weights corresponding to the first n layers are copied in the target network, and the remaining layers are randomly initialised; and secondly, fine-tuning is performed over these layers or over all layers.

In theory, domain adaptation can be performed by any unsupervised deep learning method, such as, auto-encoder [50] and dictionary learning [51] which can be integrated as the later layers of a CNN network [52]. The method presented in [53] transfers a pre-trained model to solve re-id on an unlabelled target dataset. In addition, soft-label self-training based deep unsupervised learning has become popular recently [54]. In [55], a novel co-training based unsupervised domain adaptation method is proposed, in [56] a semi-supervised approach is presented, and in [57], both supervised and unsupervised settings are used to perform a two-stepped finetuning strategy.

On the other hand, to keep the advantages of supervised models and to cope with the lack of labelled training data, some recent works use auxiliary source datasets from a different domain, taking a multi-task joint training approach. These methods go beyond existing re-id datasets and consider much larger sources, like classification datasets. An example of vast classification dataset is ImageNet dataset [58], which contains millions of images of thousands of object categories. This has been shown useful as source dataset for model pre-training in re-id. In [55], the ImageNet dataset is employed in a two-stepped fine-tuning strategy, to solve a re-id model.

However, transferring knowledge from a classification dataset to a re-id dataset has a number of drawbacks, due to the differences in the tasks for which they were meant, and the dissimilarities in their input data. In [55], several model architecture modifications, and different loss functions are proposed, to combine the learning on two different domains. Indeed, the inputs of a re-id model are person detection images, which have very different aspect ratios and much lower resolutions than classification images. Consequently, the architectures used to train models with them are quite different. That implies the modification and adaptation of the traditional re-id architectures to be able to receive the transferred knowledge from the classification models.

Other methods aim to minimise the discrepancy between the marginal [59, 60] or joint [61] distributions of the source and target datasets, by blurring the domain boundary [59], with a cross-domain loss. However, this method is unsuitable when the source and target domains have completely different tasks, like in [55]. A systematic study is presented in [49] which examines how transferable features of different layers are between different domains. This work concludes that the generalisation ability decreases when the discrepancy between the source and target tasks increases.

For that reason, instead of adapting the re-id model network to a completely different domain dataset, in this work, a new auxiliary source dataset is created from a different domain but keeping the necessary characteristics to feed a re-id model. A multi-object tracking dataset has been chosen, concretely, the MOT17 dataset from the MOT challenge. As most of the datasets from the people tracking domain, the MOT17 provides the localization of the people detections and their identities at every frame of a bunch of video sequences.

In that way, the resolution and aspect ratio of the created auxiliary dataset is quite similar to those in re-id datasets, so we keep the advantages of using an auxiliary re-id dataset, but with the possibility of getting a larger training dataset. Moreover, a pairwise binary classification can be adopted in both the pre-training step, with the new dataset and the fine-tuning stage, with the re-id target dataset. Hence the same network is used as source and target model.

In order to use a unique architecture, other works use different re-id datasets as source and target datasets. The cross-dataset transfer learning has been recently adopted for re-id to provide transferable discriminative information from other re-id labelled data to a given target dataset. Among the existing cross-dataset transfer learning works, the literature presents [62], where an SVM multi-kernel learning transfer strategy is adopted, and [63, 64], where multi-task metric learning models are employed. In [65], a single model is learnt across multiple re-id datasets before the fine-tuning in each one of them.

The size of the training data, even when it is formed by a combination of re-id datasets, is relatively small, which tend to overfit a camera-pair-specific model. In consequence, that model cannot be directly transferred to solve re-id in a target re-id dataset containing people who never appeared in the source datasets, which were captured from a new camera pair.

Besides that, the use of existing re-id labelled datasets, each captured from camera pairs with different viewing conditions, to pre-train a model, allows learning the view-invariant features of a person’s appearance. However, the differences between the source and target domains, due to the drastically different camera viewing conditions could result in a negative transfer [53].

Therefore, the negative transfer of the pre-learnt view-to-view transitions should be avoided. For that reason, contrary to the cross-dataset approach, this work deals with the lack of data by means of creating a new vast dataset from multi-object tracking sequences. In addition, people detections from different sequences are compared in the pre-training of the model. In that way this is fed with input data captured from different cameras views, avoiding the learning, and consequently the negative transfer of specific camera-to-camera transformations.

3. Re-identification model

Considering single-shot re-identification task as an isolated module, its objective is to identify the person represented by an image from one view (probe image) among all the images from the other view (gallery images).

This is achieved by calculating the distances between the probe image and all the gallery images. Then, the gallery image presenting the smallest distance is selected as the correct match.

The distances are not computed directly over the raw images but over a representative feature array. Therefore, it is necessary to learn an embedding $F(I)$ , from an image $I$ into a feature space, such that the distances between samples corresponding to the same person are small and, between different people are larger.

This feature embedding has been modelled by a deep convolutional neural network, whose architecture is described in Section 3.1. To train the weights of this neural network, a Triplet model has been employed, as it is described in Section 3.2.

The objective of this model is to perform person re-identification, so re-id domain is considered as the target domain. However, learning previously acquired in a different domain, called source or auxiliary domain can be transferred to initialize the weights of the model.

The presented learning model has been used for both, its pre-training on a source dataset, and its fine-tuning on a target dataset, allowing the use of a unique architecture to transfer learning between two different data domains.

3.1 Model architecture

The transformation of every image into its corresponding point in the feature space is performed by a deep convolutional neural network, DCNN. Concretely, the 16-layered network presented as the D version of a set of Very Deep Convolutional Neural networks in [43] and hereafter called VGG16. The selection of this architecture has been inspired by its extensive use in both domains, multi-object tracking [45] and re-identification [23], with demonstrated good results. In addition, this architecture fits perfectly for the proof of concept of the proposed approach, conducted in this work, and its configuration is easily implemented with Caffe libraries [66].

VGG16 presents thirteen convolution layers and three fully connected (FC) layers. Moreover, this network presents a SoftMax layer as the final layer, which has been removed in this work. Besides that, the input size has been modified in order to adapt it to the re-identification training samples dimensions. Therefore, the input of the proposed DCNN is a person RGB image of 128 $\times$ 64 pixels and its output is a point of the feature space, represented by a 1000-dimensional array. The layers specifications for the proposed VGG16-based embedding are listed in Table 1. The dimension of the inputs and outputs of every layer are determined by the size of the network input, and the successive performed operations (convolutions and pooling), whose parameters were defined in [43]. In addition, all hidden layers are provided of the rectification (ReLU) non-linearity [4].

Table 1
Structure of the used VGG16-based model. The input and output sizes are described in #rows $\times$ #cols $\times$ #filters; the kernel, in #rows $\times$ #cols $\times$ #filters, stride, and #outputs for FC layers

Layer	Input size	Output size	kernel
Conv-1-1	128 $\times$ 64 $\times$ 3	128 $\times$ 64 $\times$ 64	3 $\times$ 3 $\times$ 3
Conv-1-2	128 $\times$ 64 $\times$ 64	128 $\times$ 64 $\times$ 64	3 $\times$ 3 $\times$ 64
Pool-1	128 $\times$ 64 $\times$ 64	64 $\times$ 32 $\times$ 64	2 $\times$ 2 $\times$ 64, 2
Conv-2-1	64 $\times$ 32 $\times$ 64	64 $\times$ 32 $\times$ 128	3 $\times$ 3 $\times$ 64
Conv-2-2	64 $\times$ 32 $\times$ 128	64 $\times$ 32 $\times$ 128	3 $\times$ 3 $\times$ 128
Pool-2	64 $\times$ 32 $\times$ 128	32 $\times$ 16 $\times$ 128	2 $\times$ 2 $\times$ 128, 2
Conv-3-1	32 $\times$ 16 $\times$ 128	32 $\times$ 16 $\times$ 256	3 $\times$ 3 $\times$ 128
Conv-3-2	32 $\times$ 16 $\times$ 256	32 $\times$ 16 $\times$ 256	3 $\times$ 3 $\times$ 256
Conv-3-3	32 $\times$ 16 $\times$ 256	32 $\times$ 16 $\times$ 256	3 $\times$ 3 $\times$ 256
Pool-3	32 $\times$ 16 $\times$ 256	16 $\times$ 8 $\times$ 256	2 $\times$ 2 $\times$ 256, 2
Conv-4-1	16 $\times$ 8 $\times$ 256	16 $\times$ 8 $\times$ 512	3 $\times$ 3 $\times$ 256
Conv-4-2	16 $\times$ 8 $\times$ 512	16 $\times$ 8 $\times$ 512	3 $\times$ 3 $\times$ 512
Conv-4-3	16 $\times$ 8 $\times$ 512	16 $\times$ 8 $\times$ 512	3 $\times$ 3 $\times$ 512
Pool-4	16 $\times$ 8 $\times$ 512	8 $\times$ 4 $\times$ 512	2 $\times$ 2 $\times$ 512, 2
Conv-5-1	8 $\times$ 4 $\times$ 512	8 $\times$ 4 $\times$ 512	3 $\times$ 3 $\times$ 512
Conv-5-2	8 $\times$ 4 $\times$ 512	8 $\times$ 4 $\times$ 512	3 $\times$ 3 $\times$ 512
Conv-5-3	8 $\times$ 4 $\times$ 512	8 $\times$ 4 $\times$ 512	3 $\times$ 3 $\times$ 512
Pool-5	8 $\times$ 4 $\times$ 512	4 $\times$ 2 $\times$ 512	2 $\times$ 2 $\times$ 512, 2
FC-6	4 $\times$ 2 $\times$ 512	1 $\times$ 1 $\times$ 4096	4096
FC-7	1 $\times$ 1 $\times$ 4096	1 $\times$ 1 $\times$ 4096	4096
FC-8	1 $\times$ 1 $\times$ 4096	1 $\times$ 1 $\times$ 1000	1000

3.2 Learning model

To learn the weights of the previously described deep neural network, a Triplet model approach is proposed since it has been demonstrated to alleviate the unbalanced nature of the re-id datasets [42]. Indeed, the number of instances of a query identity is very limited, specifically when compared with the vast quantity of potentially available different people representations.

According to the triplet model, the network is triplicated in three branches, which are forced to share the same parameters. That means that the learnt weights are identical in the three branches, so only one feature descriptor is learnt, but it is computed for each one of the three different input images, as Fig. 2 shows.

The first branch receives an anchor sample, $I_{i}^{a}$ , the second one takes an image, $I_{i}^{p}$ , rendering the same person than the anchor image, and the input of the third branch is a different person’s image, $I_{i}^{n}$ . Therefore, every sample, $I_{i}$ , of the training dataset is constituted by a triplet of person images, $I_{i}=\langle I_{i}^{a},I_{i}^{p},I_{i}^{n}\rangle$ . $I_{i}^{a}$ and $I_{i}^{p}$ present the same identification number and they form a matched or positive pair of samples. The identification number of $I_{i}^{n}$ is different, so the pair formed by this image and the anchor image, $I_{i}^{a}$ , is a mismatched one, also called negative pair.

Figure 2.

Triplet learning architecture.

The goal is to learn a transformation from the image to the feature space, such it leads the representations for the same person near and far away from different person representations. These constraints are imposed by the triplet loss function.

The feature representation for an image, $F_{W}\left(I_{i}\right)$ is given by the output of the deep convolutional neural network (VGG16), which maps an input image to the feature space. Such embedding depends on the learnt values for the network weights, $W$ .

The similarities between the input images, $I_{i}^{a},I_{i}^{p},$ $I_{i}^{n}$ , are measured by computing the squared Euclidean distance between their corresponding features arrays, $F_{W}\left(I_{i}^{a}\right)$ , $F_{W}\left(I_{i}^{p}\right)$ , $F_{W}\left(I_{i}^{n}\right)$ . The triplet objective formulation, presented in [36], establishes a relative distance relationship. It requires that the distance of the negative pair be larger than that of the positive one by a predefined margin, $\lambda$ , as it is defined by Eq. (3.2).

$\displaystyle{\left\|F_{W}\left(I_{i}^{a}\right)-F_{W}\left(I_{i}^{p}\right)% \right\|}^{2}+\lambda<$ $\displaystyle\left\|F_{W}\left(I_{i}^{a}\right)-F_{W}\left(I_{i}^{n}\right)% \right\|^{2},\forall{I}_{i}\in I$ (1)

This equation ensures that the anchor sample is closer to all samples of the same person, in the feature space, than it is to any sample of other people since $I$ is the set of all the triplets in the training set.

However, in the one-shot person re-identification challenge, there is only one positive sample for every anchor image. So, the available data is not enough to adopt an individual-meant approach, which clusters the samples of the same person close from each other and distant to another person identity cluster.

On contrary, the proposed method treats all the possible positive pairs as a set rendering the similarity class. In the same way, negative pairs represent the dissimilarity class. The discrimination between similarity and dissimilarity is learnt by means of comparing every positive pair with all the negative pairs generated from the same anchor, taking advantage of the triplet model architecture for that.

The huge amount of possible triplet combinations makes impossible the used of all the data in every iteration, due to the limitations of the computational memory resources. This has dealt to implement a mini-batch-based learning algorithm approach. Therefore, eventually, the loss function to minimise, $f_{L}$ , is defined by Eq. (2), where $B$ ( $B=$ 64) is the number of elements in a batch of triplets, $I^{t}=\left\{I_{i}\right\}$ .

Batch gradient descent, where the batch is formed by all the training samples, provides more accuracy in the parameters updating than Stochastic gradient descent ( $B=$ 1). Besides, with the vectorized computation performed by the employed deep learning framework (Caffe), the time taken by Batch gradient descent to perform an update has decreased until acceptable values. For that, in the performed training processes, the chosen batch size was that as large as the available memory resources make possible.

The value of $\lambda$ has been experimentally set to 0.4. Not remarkable differences have been found on the performance of the algorithm when the margin value is slightly changed. However, much higher values make the learning objective too demanding. This results in oscillating values of the loss function, and consequently, the model convergence is slower.

$\displaystyle f_{L}\left(W,I^{t}\right)=$ (2) $\displaystyle\sum\limits_{i=1}^{B}\left[\max\left({\begin{array}[]{l}\lambda+% \left\|F_{W}\left(I_{i}^{a}\right)-F_{W}\left(I_{i}^{p}\right)\right\|^{2}\\ -\left\|F_{W}\left(I_{i}^{a}\right)-F_{W}\left(I_{i}^{n}\right)\right\|^{2},0% \\ \end{array}}\right)\right]$

4. Transfer learning

The re-id performance of the proposed model is measured over some re-id datasets, so these are known as target datasets. Because of the insufficiency of labelled data provided by the target re-id datasets to avoid the overfitting of the trained neural model, transferring the parameters learned on a larger auxiliary dataset becomes necessary.

This auxiliary dataset, also called source dataset, is used to pre-train the neural model. In that way, the learning acquired by the whole model or by some of its layers could be transferred to solve the re-id task with improved performance.

The employed source and target datasets, and the levels of learning transference from one to the other domain are described in the following sections.

4.1 Target and source datasets

The following two datasets have been used as target datasets: PRID2011 (Person Re-ID) dataset2 [25] and VIPeR (Viewpoint-Invariant Pedestrian Re-identi- fication) dataset3 [26]. These are two of the most widely used datasets for evaluating re-identification approaches.

Both are composed of two sets (A and B) of person images captured from two static non-overlapping camera views with remarkable differences in camera characteristics, illumination, person’s poses, and background. The person images have a resolution of 64 $\times$ 128 pixels. The main difference between these datasets is that in PRID2011 all the images from the set A were captured from the same camera view, and all the samples from set B were acquired from a second camera view different from the first one. However, in VIPeR, the images were captured from many different views, even inside the same set, but maintaining the assumption that the representation of a person in set A was captured from different camera view than that from which its representation in set B was captured. In addition, most of the examples contain a viewpoint change of 90 degrees.

Figure 3.

Triplet samples from PRID2011 dataset.

The number of images and different individuals presented by each dataset (in their single-shot version) are exposed below:

Figure 4.

Triplet samples generated from MOT17 dataset.

•

PRID2011: Set A contains 385 different images, and set B, 749 images. 200 of the captured individuals are rendered in both sets, and 90 of them were randomly extracted to be used to generate the training set and 10 for the cross-validation set. The test set has been formed by following the procedure described in [25], i.e., the images of set A for the 100 remaining individuals with representation in both sets have been used as a probe set. The gallery set has been formed by 649 images belonging to set B (all images of set B except the 100 corresponding to the training and cross-validation individuals).

•

VIPeR: It presents 632 pedestrians, each one with representation in both sets, A and B. For evaluation, the procedure described in [26] has been followed. The set of 632 pairs has been randomly split into two sets of 316 pairs each. One of them has been divided in a proportion of 90% -10% to create the training and cross-validation sets, respectively; and the other set has been used for testing the algorithm. Therefore, the gallery set is formed by 316 images from set A, and the probe set by their matching images from set B.

For both datasets, those individuals selected as training samples, have been properly combined to form a huge number of training triplets, and in the same way, for the cross-validation set. This has been performed by a data generation tool,4 which takes the anchor samples from one of the views (A) and the positive and negative samples from the other one (B), as Fig. 3 shows through some triplet examples extracted from PRID2011. In order to generate a vast number of triplets, every available positive pair has been grouped with every possible negative sample.

Before training and testing over the described re-id datasets, the model has been pre-trained using an auxiliary dataset. This auxiliary dataset, also called source dataset, has been created by extracting people detections from the MOT17 dataset.5 This dataset includes seven variated and labelled real-world surveillance sequences, meant to train and test multi-person tracking algorithms. The MOT17 dataset belongs to the MOT challenge and contains the same set of sequences as MOT16 [67], but with extended ground truth.

A data generation tool6 has has been employed to extract and combine people to create the training and cross-validation sets of triplets. From each frame, the appearing people have been extracted thanks to their bounding boxes, which are defined in the dataset ground truth, as well as, their identification number. Subsequently, triplets have been generated from the extracted people images, taking as the anchor and the positive images pairs of samples from the same individual. These samples have not why to be consecutive in time, that means that a certain time step is allowed between the frames from which they come. Then, for each pair, the negative sample is added to form a triplet. The negative sample is an image from a different individual and can be taken even from a different sequence of the MOT17 dataset, as Fig. 4 shows.

The person images have been resized to the same resolution than re-id datasets samples, 64 $\times$ 128, in order to be able to use the same neural architecture to train with the source and the target datasets.

4.2 Levels of transfer learning

Fine-tuning [49] has been the adopted transfer learning strategy in this work. This is composed of two training processes: firstly, the neural model (presented in Section 3) is trained on the source dataset, created from tracking sequences. Secondly, the model is fine-tuned over the target dataset. To do that, the pre-trained weights corresponding to the first $l_{T}$ layers have been transferred, and the remaining weights have been initialized using Xavier method [68]. Then, the learning algorithm is conducted over the target dataset, so that, only the weights of the last $l_{U}$ layers are updated thorough the second training.

Therefore, $l_{T}$ is the number of pre-trained layers transferred to the final model and $l_{U}$ is the number of layers whose weights are trained or updated by the target dataset. Considering a model of depth $L$ , i.e., $L$ is the total number of layers of the model, it can be deduced that: the weights of the first $L-l_{U}$ layers are learnt using the source dataset in the first training; the weights of the last $L-l_{T}$ layers are trained in the second training with the target dataset; and the $l_{t}+l_{U}-L$ central layers are pre-trained in the first training process and fine-tuned in the second one. These deductions are represented in Fig. 5, where the contribution of the source dataset to the parameters learning is represented in blue, and yellow is used for the target dataset.

Depending on the values given to $l_{T}$ and $l_{U}$ , six different levels of transfer learning have been designed and tested in this work, as Table 2 exposes.

Table 2
Set-up values for $l_{T}$ and $l_{U}$ , for every level of transfer learning

Level	$l_{T}$	$l_{U}$	$L-l_{U}$	$l_{t}+l_{U}-L$	$L-l_{T}$
0	0	16	0	0	16
1	7	16	0	7	9
2	7	9	7	0	9
3	16	16	0	16	0
4	16	9	7	9	0
5	16	0	16	0	0

Figure 5.

Contribution of the source (blue) and target (yellow) datasets on the learning of the different layers weights for an example model, and for given values of $l_{T}$ and $l_{U}$ .

Level 0. Level 0.

No layer is pre-trained in the source dataset ( $l_{T}=$ 0). So, the model is totally and uniquely learnt over the target dataset, where it is also tested. In this level, there is no transferred learning.

Level 1.

Seven of the layers pre-trained on the source dataset (0 $<l_{T}<L$ ) are tranferred and fine-tuned on the target dataset. So, in the second training the initial values of the first seven layers comes from the first training.

Level 2.

Seven of the layers pre-trained on the source dataset (0 $<l_{T}<L$ ) are transferred to the final model, but not fine-tuned on the target dataset. So, in the final model, the final values of the first seven layers come from the first training.

Level 3.

All the layers pre-trained on the source dataset ( $l_{T}=L$ ) are tranferred and fine-tuned on the target dataset. So, in the second training, the initial values of the layers come from the first training.

Level 4.

All the layers pre-trained on the source dataset ( $l_{T}=L$ ) are tranferred to the final model, but only nine are fine-tuned on the target dataset. In the final model, the final values for the first seven layers come from the first training. The rest of the layers also takes the initial values from the first training but they are fine-tuned in the second training.

Level 5.

All the layers are trained on the source dataset ( $l_{T}=L$ ) and none of them is fine-tuned ( $l_{U}=$ 0). So, the model is totally trained on the source dataset but tested on the target one. In this level the complete model is tranferred from the source domain to the target one.

In general, the intuition behind fine-tuning is that low-level features are learnt from the source dataset. This dataset is formed by person images from MOT sequences, so many of the learnt features can be transferred to solve the task of re-identifying people. Then, by fine-tuning the parameters and learning the final layers of the model on the re-id target dataset, high-level descriptors are learnt. These descriptors automatically acquire the capacities of discriminating people and recognising an identity in images from different views, through their training on the target dataset. That is because, contrary to the source dataset, the re-id target dataset contains samples of images captured from different camera views.

5. Experimental results

The transfer learning method basically consists of a first stage of pre-training on the source dataset and a second stage of fine-tuning in the target dataset. Instead of only evaluating the re-id performance of the final model, a prior test of the pre-trained model has been also conducted. Therefore, both stages of transfer learning have been evaluated.

The following Subsections 5.1 and 5.2 describe the evaluation of the pre-trained and final model respectively. Both present the employed metrics and datasets, as well as the conducted experiments and the obtained results.

5.1 Pre-trained model evaluation

In this work, a pair-wise binary classification model has been pre-trained over samples that were extracted from a MOT dataset. The obtained model is meant to initialize the weights of a re-identification network. Therefore, a prior test of the performance of this model to discriminate between positive and negative pairs of person images is essential to evaluate whether it is suitable to be used as the initializer. For that reason, this model has been evaluated by measuring its performance to discriminate between positive and negative pairs of samples.

A test set has been generated by extracting pairs of samples from the MOT17 sequences [67], as it has been explained in Section 4.1. For that, different individuals from those appearing in the training and cross-validation sets have been used. In order to provide a fair evaluation, the test set has been formed by 20000 positive pairs and the same number of negatives ones, presenting a completely balanced proportion of samples from each class.

The performance of the pre-trained model to properly classify the pairs samples of the test set has been rendered by a Relative Operating Characteristic (ROC) curve [69], shown by Fig. 6. This curve plots the True Positive Rate (TPR), also called Sensitivity or Recall, against the False Positive Rate (FPR), also known as fall-out rate, at various discrimination threshold, th, settings.

Figure 6.

ROC curve of the pre-trained model.

The TPR and FPR metrics are defined by Eqs (3) and (4), respectively, where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives and FN is the number of false negatives. The obtained values for these metrics are listed in Table 3.

$\displaystyle\textit{TPR}=\frac{\textit{TP}}{\textit{TP}+\textit{FN}}$ (3) $\displaystyle\textit{FPR}=\frac{\textit{FP}}{\textit{FP}+\textit{TN}}$ (4)

The classification test proves the high capacity of the proposed re-id architecture to learn to discriminate between positive pairs of images, belonging to the same person, and negative ones, corresponding to different people.

The distance between the features of each image of a pair measures the degree of dissimilarity between them. Therefore, the chosen threshold, th, divides the distance space in two ranges of values corresponding to each class. The objective was to learn features that make the distance between two images of the same person lower than a certain threshold, and higher for different people images, in order to provide a high TPR and low FPR.

Table 3

Metrics scores resulting from the binary classification performed by the model pretrained on the MOT dataset, for different values of th

th	TN	FN	TP	FP	TPR	FPR
0	20000	20000	0	0	0	0
0.05	19997	14727	3	5273	0.26365	0.00015
0.1	19990	11599	10	8401	0.42005	0.0005
0.15	19984	9516	16	10484	0.5242	0.0008
0.2	19970	7879	30	12121	0.60605	0.0015
0.25	19936	6542	64	13458	0.6729	0.0032
0.3	19878	5428	122	14572	0.7286	0.0061
0.35	19765	4540	235	15460	0.773	0.01175
0.4	19600	3849	400	16151	0.80755	0.02
0.45	19441	3370	559	16630	0.8315	0.02795
0.5	19264	2968	736	17032	0.8516	0.0368
0.55	19030	2646	970	17354	0.8677	0.0485
0.6	18785	2340	1215	17660	0.883	0.06075
0.65	18499	2082	1501	17918	0.8959	0.07505
0.7	18166	1861	1834	18139	0.90695	0.0917
0.75	17727	1612	2273	18388	0.9194	0.11365
0.8	17093	1387	2907	18613	0.93065	0.14535
0.85	16344	1166	3656	18834	0.9417	0.1828
0.9	15356	863	4644	19137	0.95685	0.2322
0.95	13751	430	6249	19570	0.9785	0.31245
1	0	0	20000	20000	1	1

The results show that choosing the value 0.7 for th, the model keeps the TPR higher than 90% and the FPR lower than 10%, which are considerably good values for a challenging task as re-identification, even in a MOT sequence. In addition, these results prove the potential applicability of this model not only in re-identification task but also for a visual-based association of identities in a tracking algorithm.

5.2 Fine-tuned model evaluation

In the second stage of transfer learning, the weights of a certain number of layers of the pre-train model are transferred to the network and then some of its layers weights are fine-tuned. In this work, six different levels of transfer learning (defined in Section 4.2) have been conducted over two different re-id datasets, PRID2011 and VIPeR dataset. Therefore, twelve different final models have been obtained and evaluated. The procedures exposed in Section 4.1. have been applied to select test data.

The performance of the learnt models has been evaluated by computing their Cumulative Matching Characteristic (CMC) curve [70], which is a standard re-id performance measurement.

To obtain the CMC curve, first, every image from the probe set is coupled with every image from the gallery set and the distance metric (squared Euclidean distance) between them is computed. The matches presenting the lowest values of the distance metric are considered as the top matches since two images belonging to the same person should be rendered close in the feature space and further from different people representations. The distance metrics obtained by comparing a probe image with all the gallery set images are ranked. This process is repeated for each one of the probe images. Then the CMC curve renders the expectation of finding the correct match within the top $r$ matches, for different values of $r$ , called ranks. Therefore, the CMC curve is always increasing since the correct matches found within the top $r$ matches will be also found within the top $r+1$ matches.

The time required to compute the proposed image descriptor using a Nvidia Titan Xp graphic process unit is 11.34 ms. This time has been measured as the average time taken to compute the output of the trained network for 65.000 different samples. The obtained computation time makes possible the use of the proposed descriptor in a real-time surveillance system.

The CMC scores obtained for the models learnt over the PRID2011 dataset by applying different transfer learning levels are listed in Table 4, and represented in Fig. 7, in ease of exposition. Analogously, Table 5 and Fig. 8 present the CMC scores for the models obtained with VIPeR dataset.

Table 4
CMC scores (in [%]) for different levels of transfer learning, tested over PRID2011 dataset

	Rank, $r$
	1	5	10	20	50	100
Level 0	2	7	10	20	30	52
Level 1	2	8	18	27	46	65
Level 2	2	8	11	17	34	59
Level 3	4	14	20	33	58	77
Level 4	2	12	23	29	47	75
Level 5	2	6	7	14	29	42

Figure 7.

CMC curves for different levels of transfer learning, tested over PRID2011 dataset.

Table 5

CMC scores (in [%]) for different levels of transfer learning, tested over VIPeR dataset

	Rank, $r$
	1	5	10	20	50	100
Level 0	6	16	26	39	69	88
Level 1	10	32	48	64	80	93
Level 2	11	31	48	67	83	95
Level 3	11	35	54	69	90	96
Level 4	10	33	50	70	90	95
Level 5	5	17	25	38	58	75

Figure 8.

CMC curves for different levels of transfer learning, tested over VIPeR dataset.

In general, both graphics show that the extreme levels 0 and 5 present the worst results and the intermediate levels 3 and 4, the best scores.

In level 0, the model has been trained and tested in the target dataset. Because of the small number of samples on the target dataset (especially the number of positive samples), this model suffers from overfitting and it is not able to generalise the learnt solution to the unknown samples from the test set. Figure 9 shows the learning curve for the level 0 training on PRID2011 and VIPeR datasets.

Figure 9.

Learning curves for level 0 training with PRID2011 (a) and VIPeR dataset (b).

The curves plot the loss value, $f_{L}$ , thorough the learning iterations, it. Although the proposed algorithm has been trained only using the training set, the loss value, for both the training and the cross-validation sets, has been represented, in blue and orange respectively. The training loss strongly decreases, so the model is converging, but the cross-validation loss increases since the model do not generalise with the unknown samples from the cross-validation set. These differences between both loss values indicate that the model is overfitted.

On contrary, the problem in level 5 is that the model has been trained on an auxiliary dataset, where, despite presenting a huge number of samples, they were not captured from two different camera views. In consequence, the model is not able to learn a feature robust against the intra-class variations.

Table 6

Accuracy metric values comparison

		Accuracy, ac (in [%])
PRID2011 dataset	Level 0	80.34
	Level 3	85.93
VIPeR dataset	Level 0	87.84
	Level 3	92.28

On the other hand, in levels 3 and 4, all the weights pre-trained in the source dataset have been transferred to the model. Then, in level 3 all the layers have been fine-tuned with the target dataset and, in level 4, only the final $l_{U}$ layers. Hence, firstly the model has been trained to extract low-level features from many samples of people, provided by MOT sequences, and secondly, high-level representations have been learnt by fine-tuning with the target dataset. The target dataset allows the model to learn features able to deal with the intra-class variations and inter-class ambiguities, which is inherent to the re-id task and re-id datasets.

In addition, the model overfitting is reduced by the pre-training in a larger auxiliary dataset, as Fig. 10 shows with the reduction of the cross-validation loss.

Figure 10.

Learning curves for the fine-tuning of level 4 with PRID2011 (a) and VIPeR dataset (b).

The intuition under this approach is that the most representative features of a person are automatically learnt on the MOT dataset. From the set of learnt descriptors, the low-level ones, which are learnt in the earlier layers of the network model, are kept. These low-level features have been trained on a larger and more variate dataset, so they not overfit the re-id training data. Then, the most high-level representations, coded in the further layers, are fine-tuned on a re-id target dataset to make them more discriminative.

Furthermore, prediction accuracy have been also estimated during the training process. This metric is the percentage of well-classified triplets over the total, considering as well-classified those triplets where the negative pair distance is larger than the positive pair distance by a threshold, $\lambda$ . The accuracy, $A c$ , for a triplet $I_{i}=\langle I_{i}^{a},I_{i}^{p},I_{i}^{n}\rangle$ , is calculated with Eq. (5). Table 6 presents the final average accuracy percentage values measured over the test samples. This value has been obtained for the models learnt with levels 0 and 3 of transfer learning, over PRID2011 and VIPeR dataset. The table shows the accuracy increment produced by the transference of learning.

$\displaystyle Ac(I_{i}^{t})=\begin{cases}1&\text{if∼{}}\|F_{W}\left(I_{i}^{a}% \right)-F_{W}\left(I_{i}^{p}\right)\|^{2}+\\ &\lambda\leqslant\|F_{W}\left(I_{i}^{a}\right)-F_{W}\left(I_{i}^{n}\right)\|^{% 2}\\ 0&\text{otherwise}\\ \end{cases}$ (5)

The models that have been obtained by transferring the MOT dataset learning clearly outperform the models directly learnt on the re-id dataset. This result can be taken as proof of concept to verify that transfer learning can be exploited to solve the re-id data problem, which is the main objective of this work.

Although the proposed method does not directly address the inter-class variations and intra-class ambiguities problems, it implicitly alleviates them, through the pretraining of the model in a larger and more variate dataset. The higher scores of CMC demonstrate how the proposed transference of learning from MOT domain makes the re-id model capable of successfully managing a larger number of inter-class variation cases and intra-class ambiguities.

Concretely the best performance is achieved when all the network layers are pre-trained in the source dataset and fine-tunned in the target one. This is what we have call level 3 of transfer learning.

For that reason, the models obtained by applying level 3 of transfer learning have been compared with other state-of-the-art methods, and their CMC scores are listed in Tables 7 and 8, for PRID2011 and VIPeR dataset, respectively.

Table 7

Re-id methods comparison on PRID2011 dataset, ‘–’ indicates no result was reported

	Rank, $r$
	1	10	20	50	100
Proposed method	4	20	33	58	77
PSFI $+$ PRDC [33]	3	16	24	39	–
PRDC [30]	3	15	23	38	–
PSFI $+$ rankSVM [33]	4	13	20	32	–
RankSVM [29]	4	13	19	32	–
LDA [31]	4	14	21	35	48
Euclidean [72]	3	10	14	28	45
GFI [71]	4	10	17	32	–
LDML [32]	2	6	11	19	32
PSFI [33]	1	4	7	14	–

Table 8

Re-id methods comparison on VIPeR dataset, ‘–’ indicates no result was reported

	Rank, $r$
	1	10	20	50	100
Proposed method	11	54	69	90	96
PSFI $+$ PRDC [33]	16	51	66	–	–
PRDC [30]	16	51	66	–	–
PSFI $+$ rankSVM [33]	16	51	66	–	–
RankSVM [29]	15	50	65	–	–
LDA [31]	7	25	37	61	79
Euclidean [72]	7	24	34	55	73
GFI [71]	9	27	34	–	–
LDML [32]	6	24	35	54	72
PSFI [33]	10	31	43	–	–

Instead of automatically find high-level features by using deep learning algorithms, the compared methods are mainly based on the computation of hand-crafted low-level features and the learning of a metric distance to compare them.

Linear Discriminant Analysis (LDA) [31], and Logistic Discriminant Metric Learning (LDML) [32], are discriminant methods. They learn a metric distance over a set of previously computed features, on contrary to the proposed approach where the whole re-id model is jointly trained, resulting in a notable improvement.

In [33], a Prototype-Sensitive Feature Importance based method is proposed to adaptively weight features according to different clusters of population. On contrary, [71] present a Global Feature Importance (GFI) approach, addressing the learning of a global weighting, i.e. a vector of generic weights and invariant to the population. Other examples of GFI methods are Ranking Support Vector Machines (Rank-SVM) [29], and Probabilistic Relative Distance Comparison (PRDC) [30]. No population discrimination has been made in this work, and the general weighting of the features to create a global person descriptor have been implicitly and automatically performed by the proposed deep neural model learning. This proposed approach outperforms the previously cited methods and even the combination of some of them (PSFI $+$ PRCD and PSFI $+$ RankSVM).

In [72], the Euclidean distance is directly applied to compare two person representations. This paper also proposes to use the Euclidean distance, with the difference that the compared descriptors have been learnt by a deep re-id neural model, which produces a remarkable improvement in the performance.

In general, the proposed re-id model presents better results than the other compared methods, which do not use deep learning models. Therefore, this paper proves that, as long as the lack of re-id data is faced by a proper strategy like transfer learning, a deep learning approach is applicable to solve the challenging task of single-shot re-id.

6. Conclusions

The main goal of this paper was to solve a problem derived from the lack of data when addressing the single-shot re-identification task through the learning of a deep triplet neural network. With that objective, this paper proposes the transference of learning from the Multi-Object Tracking (MOT) domain to the Re-Identification (re-id) one.

In order to avoid the overfitting of the proposed neural model, a huge amount of triplet samples have been generated from a set of MOT sequences. Therefore, the MOT domain has been adapted to feed a re-id neural model, and not the other way around.

Different levels of transfer learning have been designed and analysed, and the results have proved that the transfer of learning from a larger and different domain dataset provides a remarkable improvement in the re-identification performance of the proposed deep model.

The method has been evaluated over two of the most challenging and commonly used re-id datasets: PRID20111 [25], and VIPeR [26], proving the effectiveness of the proposed approach, which outperforms some of the most used metric learning methods.

Besides proving the potential application of deep and transfer learning to solve single-shot re-identifi- cation, this paper proposed a generalized solution to learn a degree of appearance similarity metric. The conducted evaluation presents this metric as a potential tool for comparing people images not only in the context of re-identification but also in people tracking applications. In people tracking, an identity association process based on visual appearance can be provided by our degree of appearance similarity measurement. This measurement is the result of comparing two images with our modelled deep features, and it can be used as the cost of matching a certain detection with previous captures of a certain tracked identity.

Footnotes

The data generation tool has been implemented as a set of c $++$ functions, which are publicly available under http://github.com/ magomezs/dataset_factory/tree/master/data_factory_from_mot.

Publicly available under: https://www.tugraz.at/institute/icg/ research/team-bischof/lrs/downloads/PRID2011/.

Publicly available under: https://vision.soe.ucsc.edu/node/178.

The data generation tool has been implemented as a set of c $++$ functions, which are publicly available under http://github.com/ magomezs/dataset_factory/tree/master/data_factory_from_reid.

Publicly available under: https://motchallenge.net/.

The data generation tool has been implemented as a set of c $++$ functions, which are publicly available under http://github.com/ magomezs/dataset_factory/tree/master/data_factory_from_mot.

Acknowledgments

This work was supported by the Spanish Government through the CICYT projects (TRA2015-63708-R and TRA2016-78886-C3-1-R), and Ministerio de Educación, Cultura y Deporte para la Formación de Profesorado Universitario (FPU14/02143), Ayudas a la movilidad para estancias breves y traslados temporales (2018) of Programa Estatal de promoción del talento y su empleabilidad, and Comunidad de Madrid through SEGVAUTO-TRIES (S2013/MIT-2713). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPUs used for this research.

References

Vezzani

Baltieri

Cucchiara

. People reidentification in surveillance forensics: A survey. ACM Computing Surveys (CSUR). 2013; 46(2): 29.

Gong

Cristani

Loy

Hospedales

. The re-identification challenge. Person Re-Identification. 2014; 1-20.

Zheng

Yang

Hauptmann

. Person reidentification: Past, present and future. arXiv Preprint ArXiv: 1610.02984. 2016.

Krizhevsky

Sutskever

Hinton

. Imagenet classification with deep convolutional neural networks. in: Advances in Neural Information Processing Systems. 2012; 1097-1105.

Molina-Cabello

Luque-Baena

López-Rubio

Thurnhofer-Hemsi

. Vehicle type detection by ensembles of convolutional neural networks operating on super-resolved images. Integrated Computer-Aided Engineering. 2018; 25(4): 321-333.

Wang

Bai

. Regional parallel structure based CNN for thermal infrared face identification. Integrated Computer-Aided Engineering. 2018; 25(3): 247-260.

Hashemi

Abdelghany

. End-to-end deep learning methodology for real-time traffic network management. Computer-Aided Civil and Infrastructure Engineering. 2018; 33(10): 849-863.

Nabian

Meidani

. Deep learning for accelerated reliability analysis of transportation networks. Computer-Aided Civil and Infrastructure Engineering. 2018; 33(6): 459-480.

Xue

. A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects. Computer-Aided Civil and Infrastructure Engineering. 2018; 33(8): 638-654.

10.

Gao

Mosalam

. Deep transfer learning for image-based structural damage recognition. Computer-Aided Civil and Infrastructure Engineering. 2018; 33(9): 748-768.

11.

Lin

Nie

. Structural damage detection with automatic feature-extraction through deep learning. Computer-Aided Civil and Infrastructure Engineering. 2017; 32(12): 1025-1046.

12.

Rafiei

Khushefati

Demirboga

Adeli

. Supervised deep restricted boltzmann machine for estimation of concrete compressive strength. ACI Materials Journal. 2017; 114(2): 237-244.

13.

Rafiei

Adeli

. Novel machine learning based algorithm to detect damage in highrise building structures. The Structural Design of Tall and Special Buildings. 2017; 26(18): DOI: 10.1002/tal.1400.

14.

Rafiei

Adeli

. Novel unsupervised deep learning model for global and local health condition assessment of structures. Engineering Structures. 2018; 156(1): 598-607.

15.

Yang

Luo

Huang

Yang

. Automatic pixel-level crack detection and measurement using fully convolutional network. Computer-Aided Civil and Infrastructure Engineering. 2018; 33(12).

16.

Zhang

Wang

KCP

Yang

Dai

Peng

Fei

Liu

Chen

. Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network. Computer-Aided Civil and Infrastructure Engineering. 2017; 32(10): 805-819.

17.

Antoniades

Spyrou

Martin-Lopez

Valentin

Alarcon

Sanei

Took

. Deep neural architectures for mapping scalp to intracranial EEG. International Journal of Neural Systems. 2018; 28(8).

18.

Morabito

Campolo

Mammone

Versaci

Franceschetti

Tagliavini

Sofia

Fatuzzo

Gambardella

Labate

Mumoli

Tripodi

Gasparini

Cianci

Sueri

Ferlazzo

Aguglia

. Deep learning representation from electroencephalography of early-stage creutzfeld-jakob disease and features for differentiation from rapidly progressive dementia. International Journal of Neural Systems. 2017; 27(2): 1650039.

19.

Pan

Paun

Zhang

Neri

. Spiking neural p systems with communication on request. International Journal of Neural Systems. 2017; 27(8): 1750042.

20.

Bible

Paun

Pan

Neri

. Simplified and yet turing universal spiking neural P systems with communication on request. International Journal of Neural Systems. 2018; 28(8): 1850013.

21.

Zhang

Rong

Neri

Perez-Jimenez

. An optimization spiking neural p system for approximately solving combinatorial optimization problems. International Journal of Neural Systems. 2014; 24(5): 1440006.

22.

Torres

Galicia

Troncoso

Martínez-Álvarez

. A scalable approach based on deep learning for big data time series forecasting. Integrated Computer-Aided Engineering. 2018; 25(4): 335-348.

23.

Liu

Feng

Jiang

Yan

. End-to-end comparative attention networks for person re-identification. arXiv Preprint ArXiv: 1606.04404, 2016.

24.

Chan-Lang

Pham

Achard

. Bidirectional sparse representations for multi-shot person re-identification. in: 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). 2016; 263-270.

25.

Hirzer

Beleznai

Roth

Bischof

. Person re-identification by descriptive and discriminative classification. in: Scandinavian Conference on Image Analysis. Springer. 2011; 91-102.

26.

Gray

Tao

. Viewpoint invariant pedestrian recognition with an ensemble of localized features. in: European Conference on Computer Vision (ECCV). Springer. 2008; 262-275.

27.

Gheissari

Sebastian

Hartley

. Person reidentification using spatiotemporal appearance. in: IEEE Conference on Computer Vision and Pattern Recognition. 2006.

28.

Wang

Doretto

Sebastian

Rittscher

. Shape and appearance context modeling. in: IEEE International Conference on Computer Vision. 2007.

29.

Prosser

Zheng

Gong

Xiang

. Person re-identification by support vector ranking. in: British Machine Vision Conference. 2010; 2: 6.

30.

Zheng

Gong

Xiang

. Person reidentification by probabilistic relative distance comparison. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2011; 649-656.

31.

Fisher

. The use of multiple measurements in taxonomic problems. Annals of Eugenics. 1936; 7(2): 179-188.

32.

Guillaumin

Verbeek

Schmid

. Is that you? metric Learning Approaches for Face Identification, IEEE 12th International Conference on Computer Vision. 2009; 498-505.

33.

Liu

Gong

Loy

Lin

. Evaluating feature importance for re-identification. in: Person Re-identification, Springe. 2014; 203-228.

34.

Girshick

Donahue

Darrell

Malik

. Rich feature hierarchies for accurate object detection and semantic segmentation. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014; 580-587.

35.

Varior

Haloi

Wang

. Gated siamese convolutional neural network architecture for human reidentification. in: European Conference on Computer Vision (ECCV). Springer. 2016; 791-808.

36.

Schroff

Kalenichenko

Philbin

. Facenet: A unified embedding for face recognition and clustering. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015; 815-823.

37.

Zhao

Xiao

Wang

. Deepreid: Deep filter pairing neural network for person re-identification. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014; 152-159.

38.

Lei

Liao

et al. Deep metric learning for person re-identification. in: ICPR. 2014; 34-39.

39.

Bromley

Guyon

LeCun

Säckinger

Shah

. Signature verification using a siamese time delay neural network. in: Advances in Neural Information Processing Systems. 1994; 737-744.

40.

Ahmed

Jones

Marks

. An improved deep learning architecture for person re-identification. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015; 3908-3916.

41.

Cheng

Gong

Zhou

Wang

Zheng

. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016; 1335-1344.

42.

Gómez-Silva

Armingol

de la Escalera

. Balancing people re-identification data for deep parts similarity learning. Journal of Imaging Science and Technology. 2018.

43.

Simonyan

Zisserman

. Very deep convolutional networks for large-scale image recognition. in: ICLR. 2015.

44.

Zhuang

Lin

Shen

Reid

. Fast training of triplet-based deep binary embedding networks. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016; 5955-5964.

45.

Sadeghian

Alahi

Savarese

. Tracking the untrackable: Learning to track multiple cues with long-term dependencies. Proceedings of the IEEE International Conference on Computer Vision. 2017.

46.

Farenzena

Bazzani

Perina

Cristani

Murino

. Person re-identification by symmetry-driven accumulation of local features. in: IEEE Conference on Computer Vision and Pattern Recognition. 2010.

47.

Wang

Gong

Xiang

. Unsupervised learning of generative topic saliency for person reidentification. in: Proc BMVC. 2014.

48.

Pan

Yang

. A survey on transfer learning. IEEE TKDE. 2010; 22(10): 1345-1359.

49.

Yosinski

Clune

Bengio

Lipson

. How transferable are features in deep neural networks? in: NIPS. 2014; 3320-3328.

50.

Hinton

Salakhutdinov

. Reducing the dimensionality of data with neural networks. Science. 2006; 313(5786): 504-507.

51.

Kenneth

Joseph

Bhaskar

Kjersti

Te-Won

Terrence

. Dictionary learning algorithms for sparse representation. Neural Computing. 2003; 15(2).

52.

Wang

Lin

Zuo

Zhang

. Dictionary pair classifier driven convolutional neural networks for object detection. in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.

53.

Peng

Xiang

Wang

Pontil

Gong

Huang

Tian

. Unsupervised cross-dataset transfer learning for person re-identification. in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016; 1306-1315.

54.

Huang

Loy

Tang

. Unsupervised learning of discriminative attributes and visual representations. in: CVPR. 2016.

55.

Geng

Wang

Xiang

Tian

. Deep transfer learning for person re-identification. arXiv Preprint ArXiv: 1611.05244. 2016.

56.

Liu

Song

Tao

Zhou

Chen

. Semi-supervised coupled dictionary learning for person re-identification. in: Proc CVPR. 2014.

57.

Chen

Wang

Shi

Yan

Geng

Tian

Xiang

. Deep transfer learning for person re-identification. 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM). 2018; 1-5.

58.

Deng

Dong

Socher

Fei-Fei

. Imagenet: A large-scale hierarchical image database. in: CVPR IEEE. 2009; 248-255.

59.

Ganin

Lempitsky

. Unsupervised domain adaptation by backpropagation. in: ICML. 2015.

60.

Zhang

Chang

Wang

. Deep transfer network: Unsupervised domain adaptation. CoRR, Abs/1503.00591. 2015.

61.

Long

Wang

Jordan

. Deep transfer learning with joint adaptation networks. CoRR. abs/1605.06636; 2016.

62.

Layne

Hospedales

Gong

. Domain transfer for person re-identification. in: ARTEMIS. 2013.

63.

Yang

Tao

. Person re-identification over camera networks using multi-task distance metric learning. IEEE TIP. 2014; 23(8): 3656-3670.

64.

Wang

Zheng

Zhang

. Crossscenario transfer person re-identification. 2015.

65.

Xiao

Ouyang

Wang

. Learning deep feature representations with domain guided dropout for person re-identification. in: CVPR. 2016.

66.

Jia

Shelhamer

Donahue

Karayev

Long

Girshick

Guadarrama

Darrell

. Caffe: Convolutional architecture for fast feature embedding. in: Proceedings of the 22nd ACM International Conference on Multimedia. 2014; 675-678.

67.

Milan

Leal-Taixé

Reid

Roth

Schindler

KMO

. T16: A benchmark for multi-object tracking. arXiv Preprint ArXiv: 1603.00831. 2016.

68.

Glorot

Bengio

. Understanding the difficulty of training deep feedforward neural networks. in: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 2010; 249-256.

69.

Hanley

McNeil

. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143(1): 29-36.

70.

Moon

Phillips

. Computational and performance aspects of pca-based face-recognition algorithms. Perception. 2001; 30(3): 303-321.

71.

Loy

Xiang

Gong

. Time-delayed correlation analysis for multi-camera activity undestanding. International Journal Computer Vision. 2010; 90(1): 106-129.

72.

Hirzer

Roth

Köstinger

Bischof

. Relaxed pairwise learned metric for person reidentification. Computer Vision-ECCV. 2012; 780-793.

Transferring learning from multi-person tracking to person re-identification

Abstract

Keywords

1. Introduction

3. Re-identification model

3.1 Model architecture

Table 1 Structure of the used VGG16-based model. The input and output sizes are described in #rows × #cols × #filters; the kernel, in #rows × #cols × #filters, stride, and #outputs for FC layers

4.1 Target and source datasets

Table 2 Set-up values for l T and l U , for every level of transfer learning

5.1 Pre-trained model evaluation

Table 4 CMC scores (in [%]) for different levels of transfer learning, tested over PRID2011 dataset

Footnotes

Acknowledgments

References

Table 1
Structure of the used VGG16-based model. The input and output sizes are described in #rows $\times$ #cols $\times$ #filters; the kernel, in #rows $\times$ #cols $\times$ #filters, stride, and #outputs for FC layers

Table 2
Set-up values for $l_{T}$ and $l_{U}$ , for every level of transfer learning

Table 4
CMC scores (in [%]) for different levels of transfer learning, tested over PRID2011 dataset