Image retargeting quality assessment: A survey

Abstract

With the increasing variety of display devices, image retargeting has become an indispensable technology for adjusting the aspect ratio of images to adapt to different display terminals. Since the retargeting operation would cause geometric distortion and content loss of the image, the image retargeting quality assessment (IRQA) is necessary to guide the retargeting algorithm’s optimization, selection, and design. Our paper mainly works for systematically reviewing the state-of-the-art technologies in IRQA. And then, this paper further discusses image registration algorithms for matching the original image and the retargeted image. Next, we investigate the feature measurement methods for image retargeting quality evaluation. To facilitate the quantitative assessment of the IRQA methods, this paper gives a list of publicly open datasets and the performance of the mainstream methods. Finally, some promising research directions towards IRQA are pointed out. From this survey, engineers from the industry may find skills to improve their image retargeting systems, and researchers from academia may find ideas to conduct some innovative work.

Keywords

Registration algorithm image retargeting quality assessment

1 Introduction

In recent years, due to the diversity of display devices and the versatility of image media sources, images and videos need to adapt to display terminals with different resolutions and aspect ratios [1, 2]. Image retargeting technology adjusts the size of input images while retaining important areas [3, 4] of display devices with different resolutions and aspect ratios. Since the composition and characteristics of each image are different, there is no retargeting method that can achieve satisfactory retargeting results in various types of photos [5, 6]. In addition, although objective image quality assessment methods have made significant progress, they are limited to two images of the same sizes. Due to some information loss or geometric changes, the size and resolution of the retargeted image are different from those of the original image, so it is necessary to evaluate their difference through registration or similarity measurement. Therefore, it is crucial to design an image Retargeting Quality assessment (IRQA) index consistent with a subjective assessment to quantify the effects of different retargeted operators [7] and optimize the image retargeted technology to obtain high-quality retargeted images.

According to the development of IRQA, this paper summarizes the key technologies, as shown in Fig. 1. Early IRQA indicators usually use Edge Histogram (EH) [8] and color layout (CL) [9] to measure the difference between the retargeted image and the original one. Taking the image as a whole, the spatial edge similarity and color distribution similarity between the original image and the retargeted image are calculated. Bidirectional Similarity (BDS) [10] is used to calculate the bidirectional mapping between the patches of the original image and the patches of the retargeted image as a measure. Bidirectional Warping (BDW) [11] is similar to BDS, and the mapping in BDW adopts asymmetric dynamic time warping, which minimizes the warping cost and keeps the patches in order. Still, they think that each patch is equally essential, which quickly leads to the loss of important information of retargeted images [12]. Earth-Mover’s Distance (EMD) [13] can capture the structural attributes of images, which is more consistent with human subjective perception than the above indicators. These methods mainly detect the quality of the retargeted image by calculating the similarity between the original image and the retargeted image. Since the retention of important contents is not considered [14], the consistency between these indicators and the subjective perception is low. To align the original image with the retargeted image and compare the quality of images with different sizes, Scale-invariant feature transform-flow (SIFT flow) [15] uses SIFT descriptor [16] to match the original image with the retargeted image and improves the consistency with subjective assessment. By using SIFT Flow method, most IRQA methods first apply the registration algorithm to establish the pixel correspondence between the original image and the retargeted image, which solves the problem that cannot be directly evaluated because of the different sizes and resolutions. Then features from the original and retargeted images are extracted to calculate their similarity (mainly considering geometric distortion and content loss). Finally, these features are fused to get the predicted IRQA score. With the success of deep learning in image processing, high-level semantic features have become another vital measure feature in IRQA. Preserving the semantic information of the image is the primary task of image retargeting, and IRQA without precise semantic component evaluation is incomplete[17, 18].

Fig. 1

Timeline of representative IRQA methods.

1.1 Related works

In the last decades, there are a few reviews on IRQA. Most of them focus on the classical image quality assessment measures. In 2012, Ma et al. summarized the simple mathematical distance metrics on IRQA and evaluated these metrics on their proposed dataset [19]. In 2020, Karimi et al. described subjective and objective image quality assessment metrics, and then, based on low-level and high-level features, IRQA methods are classified and their performance indicators are listed [20]. However, the involved IRQA algorithms in literature [20] are limited. In 2022, Asheghi [21] listed the existing IRQA methods from subjective and objective perspectives while summarizing image retargeting methods. The objective IRQA methods are only listed by year and not classified. Also, since there are different aspect ratios between the original and the retargeted image, image registration is generally used as the preprocess operation for the task of IRQA. However, the existing IRQA surveys do not analyze and discuss the registration algorithms. Besides, they are described without emerging classical topics. Considering the limitations of the above reviews, the image registration algorithms are introduced in detail in our paper, which can promote the understanding of the later researchers on the development of IRQA and we summarize the general order of existing IRQA algorithms, as shown in Fig. 2, which is roughly divided into four stages: (1) registering the original image with the retargeted image; (2) extracting features and measuring similarity; (3) fusing the features in stage (2); and (4) predicting the quality of the retargeted image. Based on the implementation algorithms of (1) and (2), the existing IRQA algorithms are comprehensively summarized.

Fig. 2

General method of IRQA.

The main contributions of this work are as follows:

The existing IRQA algorithms are summarized into three parts: registering the pixels of the original image with the retargeted image; extracting features for feature measurement; Fusing multiple features.

In this paper, the registration algorithms are summed up for the IRQA for the first time. The feature measurement methods are analyzed from several aspects: aesthetic perception, high-level semantic feature based on deep learning, etc.

We thoroughly compare the performance of existing IRQA methods, shedding light on potential directions for future research from registration, aesthetics, and symmetry based on deep learning.

The organization of this paper is as follows: Section 2 summarizes the IRQA methods involved in this paper briefly. Section 3 introduces the registration algorithm used by IRQA. Section 4 describes the method of feature measurement involved in IRQA and the common-used feature fusion methods; Section 5 lists the datasets and evaluation indexes of image retargeting and compares the performance of state-of-the-art IRQA methods. The conclusion and the future development are discussed inSection 6.

2 Literature review

According to publication years, we summary the state-of-the-art IRQA algorithms in Table 1. In addition, we further classify and describe the literatures in Table 1 according to feature measurement inTable 3. As shown in Table 1, we analyze the advantages and disadvantages of the referenced papers, and provide a general overview of these IRQAalgorithms.

Table 1
A summary of representative literatures in the field of IRQA

Method Year Advantages Disadvantages

SSIM [25] 2014 The SSIM map can represent the strength of the structural information of each spatial location. Cropping is selected frequently in multi-operator retargeting.

GLS [35] 2014 Considering three dominant distortion types for image retargeting for the first time. Relying on a large multi-class training dataset.

PGD+SLR [26] 2014 Proposing a novel metric to quantify the information loss of a retargeted image. Having limitations for high-texture images.

HDPM [36] 2015 Having high correlation with MOS. Lacking accurate measurement of detail distortion.

Liu [40] 2015 Having lower computational complexity and high efficiency. Having low correlation with subjective assessment.

Zhang [37] 2015 Formulating the problem in terms of regions instead of pixels correlates well with human subjective tests. Using simple weighting for each feature, losing quality information.

ARS [24] 2016 Having accurate registration results. Having poor performance for symmetry features.

Ma [32] 2016 Describing image perceptual quality using ranking learning. Paired data is required in the training and testing phases, and the amount of data is large.

BNSSD [47] 2017 Measuring edge distortion accurately. Having high time complexity.

Shigwan [31] 2017 Outperforming SIFT in registration accuracy and execution time. Having less consistent with subjective assessments.

Karim [83] 2017 Introducing the concept of measuring the homogeneity of distribution of deformities throughout the image. Having low scores on symmetry features.

Zhang [82] 2017 Combining both fidelity measures and inconsistency detection to determine the final retargeting quality. Lacking semantic measure.

Liang [45] 2017 Satisfying image aesthetics, and maintaining symmetry. The speed of the evaluation is slow.

Fu [51] 2018 Compensating for high-level features that cannot be captured by handcrafted features. Limited by existing image retargeting database.

Jiang [34] 2018 Getting rid of the dependence on the registration algorithm. Having high time complexity.

Karimi [94] 2018 Versatility and simplicity in measuring different types of distortion. high time complexity.

Li [43] 2019 Designing different measurement values according to foreground target detection results Having relatively little semantic analysis.

Zhang [29] 2019 Using SURF registration to achieve faster speed. Having poor performance of symmetry features.

Jiang [50] 2020 Simulating the human visual system’s perception of retargeted images. Leading to overfitting during training because of limited number of samples.

Ranking [12] 2020 Comparing the results of retargeted images with different original images. Having poor performance in texture features.

TRASIM [49] 2021 Improving consistency with subjective assessments with Bidirectional measurement. Depending on SIFT-flow highly to build rewarping.

RN-IRQA [6] 2021 Reducing the negative impact of inaccurate registration on image fidelity measurements. Global structural features are not considered.

INSEM [52] 2021 Having in-depth study of semantic attributes. Having poor performance on images without significant instances.

Yao [59] 2021 Assessing the degree of distortion of visually important content. Depending on the accuracy of backward registration.

RR-EMD [39] 2021 Using EMD metrics to guide how to preserve the local and global information of the reference image in the retargeted image. Only using part of the reference information, so the correlation is relatively low.

LGGD+ [38] 2021 Having good compatibility and improving existing IRQA methods. Symmetry features are not considered.

Method	Year	Advantages	Disadvantages
SSIM [25]	2014	The SSIM map can represent the strength of the structural information of each spatial location.	Cropping is selected frequently in multi-operator retargeting.
GLS [35]	2014	Considering three dominant distortion types for image retargeting for the first time.	Relying on a large multi-class training dataset.
PGD+SLR [26]	2014	Proposing a novel metric to quantify the information loss of a retargeted image.	Having limitations for high-texture images.
HDPM [36]	2015	Having high correlation with MOS.	Lacking accurate measurement of detail distortion.
Liu [40]	2015	Having lower computational complexity and high efficiency.	Having low correlation with subjective assessment.
Zhang [37]	2015	Formulating the problem in terms of regions instead of pixels correlates well with human subjective tests.	Using simple weighting for each feature, losing quality information.
ARS [24]	2016	Having accurate registration results.	Having poor performance for symmetry features.
Ma [32]	2016	Describing image perceptual quality using ranking learning.	Paired data is required in the training and testing phases, and the amount of data is large.
BNSSD [47]	2017	Measuring edge distortion accurately.	Having high time complexity.
Shigwan [31]	2017	Outperforming SIFT in registration accuracy and execution time.	Having less consistent with subjective assessments.
Karim [83]	2017	Introducing the concept of measuring the homogeneity of distribution of deformities throughout the image.	Having low scores on symmetry features.
Zhang [82]	2017	Combining both fidelity measures and inconsistency detection to determine the final retargeting quality.	Lacking semantic measure.
Liang [45]	2017	Satisfying image aesthetics, and maintaining symmetry.	The speed of the evaluation is slow.
Fu [51]	2018	Compensating for high-level features that cannot be captured by handcrafted features.	Limited by existing image retargeting database.
Jiang [34]	2018	Getting rid of the dependence on the registration algorithm.	Having high time complexity.
Karimi [94]	2018	Versatility and simplicity in measuring different types of distortion.	high time complexity.
Li [43]	2019	Designing different measurement values according to foreground target detection results	Having relatively little semantic analysis.
Zhang [29]	2019	Using SURF registration to achieve faster speed.	Having poor performance of symmetry features.
Jiang [50]	2020	Simulating the human visual system’s perception of retargeted images.	Leading to overfitting during training because of limited number of samples.
Ranking [12]	2020	Comparing the results of retargeted images with different original images.	Having poor performance in texture features.
TRASIM [49]	2021	Improving consistency with subjective assessments with Bidirectional measurement.	Depending on SIFT-flow highly to build rewarping.
RN-IRQA [6]	2021	Reducing the negative impact of inaccurate registration on image fidelity measurements.	Global structural features are not considered.
INSEM [52]	2021	Having in-depth study of semantic attributes.	Having poor performance on images without significant instances.
Yao [59]	2021	Assessing the degree of distortion of visually important content.	Depending on the accuracy of backward registration.
RR-EMD [39]	2021	Using EMD metrics to guide how to preserve the local and global information of the reference image in the retargeted image.	Only using part of the reference information, so the correlation is relatively low.
LGGD+ [38]	2021	Having good compatibility and improving existing IRQA methods.	Symmetry features are not considered.

Table 3

IRQA methods based on common-used feature measures

Framework	Methods	Features
Structural distortion and information loss.	GLS [35]	Measure graph structure similarity, image block similarity, and salient information loss.
	HDPM [36]	Calculate block similarity based on SSIM, measure feature loss based on SIFT, and graph structure distortion based on gray level co-occurrence matrix.
	Region [37]	Propose a region-based framework to evaluate the shape distortion and information loss.
	LGGD+ [38]	Consider local and global geometric distortions.
	RR-EMD [39]	Extract SIFT features and visual saliency features to calculate local and global structure similarity.
Reduce the corresponding matching error.	PGD+SLR [26]	Measure the pixel confidence level of a patch using LCM to suppress the false distortion detection caused by SIFT flow mismatch.
	SD+VCC [40]	Measure visual distortion and visual content change in the frequency domain (without matching) to reduce the corresponding matching error in space.
	Shigwan [31]	Measure the pixel confidence level of a patch using LCM to suppress the error-dense correspondence generated by the DSP algorithm.
	RN-IRQA [6]	Propose RCM to measure the registration accuracy of the original image and the retargeted image.
Improvement of ARS’s proposal and use.	ARS [24]	Establish block correspondence based on backward registration between the original and retargeted images, then measure the local block changes to evaluate the visual quality.
	MLF [41]	Combine low-level ARS, middle-level edge group similarity, and high-level face block similarity.
	Guo [42]	Redefine the length and width of the retargeted block in the local distortion measurement, and the quality of each ARS block is improved.
	Li [43]	Use different salience extraction methods to improve the calculation of the weight of each ARS block.
	MSDA [44]	Introduce a penalty factor for visual distortion to improve the quality of ARS block; Use VAF to calculate saliency map as the weight of local blocks.
	RN-IRQA [6]	Introduce penalty factor for geometric distortion to improve the quality of the ARS block when the whole local block is removed.
Add aesthetic measures.	Liang [45]	The aesthetic metric is the sum of the rule of thirds and the visual balance score, and the smaller the value, the better the retargeted result.
	Yan [46]	Extract eleven aesthetic features to train a neural network, and an aesthetic score ranging from 1 to 10 is predicted for each input image.
	Ranking [12]	Use the rule of thirds and the visual balance score to calculate the aesthetic score and assign a weight of 0.5, respectively.
Bidirectional metric	Region [37]	Evaluate information loss in the foreground area and background area bidirectionally.
	SD+VCC [40]	Use BDS to measure local visual content changes.
	BNSSD [47]	Measure salient information loss bidirectionally.
	BIMS [48]	Measure similarity of importance map bidirectionally.
	TRASIM [49]	Measure bidirectional geometric distortion and bidirectional information loss.
Involving deep learning.	Jiang [50]	Use a segmented stacked AutoEnCoder to fuse features.
	Fu [51]	Calculate texture change and semantic change to measure depth features.
	Li [43]	Extract semantic features by CNN to measure high-level semantic similarity in prospect measurement.
	INSEM [52]	Extract instance-level semantic features and assign different weights according to semantic categories.

3 Registration algorithm for image retargeting quality evaluation

Early quality assessment methods mainly use the distance between two images for measurement without considering the preservation of important contents or structural distortion [19]. Therefore, the objective evaluation results are not well consistent with subjective scores. Since the excellent performance of SIFT in registering, the IRQA index uses SIFT [16] to establish the pixel correspondence between the original image and the retargeted image. Then the extracted features are used to measure the similarity directly. Fast robust features (SURF) [22], Deformable spatial pyramid (DSP) [23], backward registration [24], and bidirectional registration are also common-used registering methods, and their principles, advantages, and disadvantages are summarized in Table 2, which describes how IRQA uses each algorithm to establish pixel correspondence.

Table 2
Comparative analysis of different registration algorithms. “P”, “A”, “D” indicates “Principles”, “Advantages”, “Disadvantages”, respectively

Algorithms Abbreviation Specific descriptions

SIFT P Detect feature points; describe the feature points by the vectors with orientations and scale information; match feature points.

A Good stability and invariance.

D Low real-time performance; low detection ratio on the objects with smooth edge.

SURF P Establish Hessian matrix to extract local feature points; reduce the dimension of the descriptor by using Haar wavelet to describe feature points; match feature point.

A Faster and more robust than SIFT.

D Unable to accurately detect feature points of objects with smooth edges.

SIFT Flow P Perform dense sampling and pixel-level SIFT feature matching on pairwise images.

A Robust matching between different parts of the object’s appearance/scene.

D Only pixel-level accuracy can be achieved.

DSP P Regularize the matching consistency from multiple spatial ranges of the whole image, coarse grid unit, and single pixel.

A Balance geometric regularization and accurate positioning; allow hierarchical optimization; apply at multiple scales.

D Perform well on multi-scale matching while time-consuming.

Backward registration P Establish MRF by the re-sampling grid model, then express backward registration as the labeling problem of the retargeted image pixels with minimum energy.

A High matching degree in smooth and non-textured areas.

D Affect the accuracy of quality assessment with inaccurate correspondence.

Bidirectional registration P Match the original image with the retargeted image bidirectionally based on SIFT flow/SIFT.

A The diversity and integrity of feature measurement are realized.

D High computational complexity.

Algorithms	Abbreviation	Specific descriptions
SIFT	P	Detect feature points; describe the feature points by the vectors with orientations and scale information; match feature points.
	A	Good stability and invariance.
	D	Low real-time performance; low detection ratio on the objects with smooth edge.
SURF	P	Establish Hessian matrix to extract local feature points; reduce the dimension of the descriptor by using Haar wavelet to describe feature points; match feature point.
	A	Faster and more robust than SIFT.
	D	Unable to accurately detect feature points of objects with smooth edges.
SIFT Flow	P	Perform dense sampling and pixel-level SIFT feature matching on pairwise images.
	A	Robust matching between different parts of the object’s appearance/scene.
	D	Only pixel-level accuracy can be achieved.
DSP	P	Regularize the matching consistency from multiple spatial ranges of the whole image, coarse grid unit, and single pixel.
	A	Balance geometric regularization and accurate positioning; allow hierarchical optimization; apply at multiple scales.
	D	Perform well on multi-scale matching while time-consuming.
Backward registration	P	Establish MRF by the re-sampling grid model, then express backward registration as the labeling problem of the retargeted image pixels with minimum energy.
	A	High matching degree in smooth and non-textured areas.
	D	Affect the accuracy of quality assessment with inaccurate correspondence.
Bidirectional registration	P	Match the original image with the retargeted image bidirectionally based on SIFT flow/SIFT.
	A	The diversity and integrity of feature measurement are realized.
	D	High computational complexity.

3.1 SIFT and SIFT flow algorithms

SIFT is a local operator describing local gradient information, which is a discrete feature representation, including feature point extraction, feature point description, and feature point matching. The SIFT descriptor extracts the local structure of the image and encodes the context information. Then a discrete and discontinuous stream estimation algorithm matches the SIFT feature at the pixel level. In 2014, Fang et al. used SIFT Flow algorithm to establish the pixel correspondence between the original image and the retargeted image [25]. Then they measured the retention of structural information in the retargeted image by the structural similarity index measure (SSIM).

To measure the quality of two images with different sizes, Hsu et al. [26] also used SIFT flow algorithm establishing dense correspondence estimation to measure the perceived geometric distortion and information loss. A new measurement method was proposed to quantify the information loss of retargeted images. As shown in Fig. 3, based on SIFT Flow mapping, the saliency map of the retargeted image is obtained by warping the saliency map of the original image [27]. The saliency loss ratio (SLR) is measured by the ratio of the sum of the saliency values before and after the retargeting operation, as shown in Equation (1).

Fig. 3

Information loss estimation [26].

$d_{SLR} = 1 - \sum_{p} | E_{r} (p) | / - \sum_{p} | E_{o} (p) |$ (1) where E_r (p) denotes the warped pixel-wise saliency map based on the SIFT flow vector at the pixel p, similarly, E_o (p) denotes the saliency map of the original image.

3.2 SURF algorithm

Speeded Up Robust Features (SURF) are an interest point detector and descriptor with invariant scale and rotation [22] which generates feature points and extracts features by constructing a Hession matrix (Gaussian convolution process corresponding to SIFT). In feature point descriptor generation, Haar wavelet response is used to allocate the directions of SURF feature points. Four directions are formed by the sum of the horizontal direction, the sum of the vertical direction, the sum of absolute values of the horizontal direction, and the sum of absolute values of the vertical direction. In comparison, SIFT counts eight gradient directions in each region. The generation time of the feature point descriptor of SURF is reduced by half, which dramatically improves the feature extraction and description speed and generates stable edge points. Zhang et al. [28] proposed an intermediate domain optimization registration method based on SURF, which can establish a robust match. Then they used the SURF features extracted by the SURF algorithm to calculate the Average Local Similarity (ALS) of all image blocks [29]. According to the unmatched SURF features, the retargeted image’s Content Loss Degree (CLD) is calculated, too, see Equations (2) and (3). $ALS = \sum_{i = 1}^{f_{num}} {SSIM}_{i} / (f_{num} + c)$ (2) $CLD = (n - f_{num}) / - n$ (3) where f_num indicates the number of SURF features matched in the original image and the retargeted image; n is the number of features extracted from the original image; SSIM_i means structural similarity; Avoiding the denominator in Equation (2) be 0, let c = 1.

3.3 DSP algorithm

The state-of-the-art matching methods (SIFT Flow and PatchMatch [30]) are generally performed at the pixel level. In [23], using the DSP algorithm, a pyramid model is proposed, which can regularize the matching consistency in multiple spatial ranges (from the whole image to the coarse grid unit and then to each pixel) simultaneously. The accuracy and running time have been greatly improved. To measure the perceived geometric distortion and information loss, Shigwan et al. [31] used the DSP algorithm to replace SIFT flow algorithm [26] and establish a fast and dense correspondence. The predicted quality score is more consistent with subjective evaluation.

3.4 Backward registration algorithm

Zhang et al. [24] showed that the geometric change estimation efficiently clarifies the relationship between the original and retargeted images. They proposed a backward registration algorithm that maps the retargeted image to the original image to reveal the geometric change. The registration accuracy is further improved than before. This paper interprets image retargeting as resampling grid generation and forward sampling. As shown in Fig. 4(a), the red nodes are the regular retargeted image grid, while the gray nodes are the regular original image grid and the blue nodes are the resampling grid in different shapes. The backward registration is formulated as a problem that reveals the resampling grid, which is a reverse problem compared to the forward resampling, as shown in Fig. 4(b). The red node p is the observed pixel in the retargeted image. The blue node l is the unknown resampling location for pixel p, and the subscripts {L, T, B, R} denote the four-connected neighborhood. It mainly estimates the resampling location for each pixel from the retargeted image. The backward registration problem is expressed as the marking problem of retargeted image pixels with minimum energy [24], and the computation process is summarized as Equation (4).

Fig. 4

Forward resampling and backward registration [24].

$E = E_{d} + λ E_{s}$ (4) where E_d is the data item, and E_s is the smooth item.

To get accurate backward registration results, this paper develops a hybrid feature descriptor in data item by using the CIE-Lab intensity component, dense SIFT descriptor, and the relative position component, which is different from using the SIFT descriptor for matching before. $f_{p} = [\begin{matrix} {Lab}_{1 \times 3} & {SIFT}_{1 \times 128} & {Pos}_{1 \times 2} \end{matrix}]$ (5) where Lab denotes the pixel intensity in CIE-Lab color space, SIFT is the histogram of the gradient information in the local neighborhood, Pos is the relative positions of the pixels normalized into the range of [–1, 1].

3.5 Bidirectional registration algorithm

A bidirectional registration algorithm was proposed in [10] to improve the accuracy of backward registration. This algorithm generally uses SIFT flow registration from two directions: forward matching from the original image to the retargeted image and backward matching from the retargeted image to the original image, which can be regarded as a supplement to backward registration. As shown in Fig. 5, the size of the visual image of forward matching is the same as the retargeted image, and the size of the visual image of backward matching is the same as the original image. Pixel matching is established in both forward and backward directions.

Fig. 5

Bidirectional registration.

3.6 Image retargeting quality evaluation without registration algorithm

No reference image quality evaluation method does not involve the registration algorithm. In 2016, at the first time, Ma et al. [32] proposed a no-reference quality assessment method for IRQA based on pairwise rank learning, which uses GIST [33] (a kind of scene feature descriptor, which can capture the most sensitive information for the evaluation of the perceived quality of the retargeted image) extracted from each image to do pairwise rank learning, and a ranking model is obtained. Based on the learning model’s ranking results and subjective quality scores, the image quality scores are obtained by the exponential curve fitting ranking function.

When using the registration algorithm, the researchers found that incorrect matching easily occurred when pixel correspondence was established between the overlapped texture and the smooth region of the image [34]. To solve this problem, Jiang et al. [34] tried not to establish the dense correspondence between images but measure the similarity between the original image and the retargeted image from a global perspective. Two overcomplete dictionaries are learned from an image to represent its corresponding distortion-sensitive features. The dictionaries of the retargeted image and the original image are connected to obtain a joint overcomplete dictionary and perform sparse representation. Then, similarity measurement is carried out. However, this method is less efficient than SIFT Flow and EH methods because learning features from over-complete dictionaries is very time-consuming, accounting for about 70% of the total time.

4 Feature measurement method for image retargeting quality evaluation

In the first stage of image registration, the pixel correspondence between the original and the retargeted image is established, which is the crucial step of IRQA and solves the problem that the original and the retargeted image are different in size. Then the features of the two images are measured. Also, the feature measurement method consistent with subjective perception can improve the consistency between IRQA and subjective evaluation [53]. Different feature measurement methods are classified and summarized in Table 3.

4.1 Structural distortion and information loss

Many experiments show that shape distortion and content information loss can improve metric performance [19]. Therefore, most IRQA methods use structural distortion and information loss as measure indicators. The following IRQA methods mainly consider these two measurement indicators. Zhang et al. [34] proposed a novel objective index that accounts for three major determining factors for humans visual perception on retargeted images: Global structural distortion (G), local, regional distortion (L), and significant information loss (S). Figure 6 shows the visualization results of three dominant distortion types for image retargeting. They are measured respectively by graph structural similarity, image block similarity, and information integrity. Finally, the machine learning model fuses these features to predict the quality score. The quality index is called GLS.

Fig. 6

Three dominant distortion types [35].

To limit visual artifacts in the retargeted image, preserve the critical content of the original image, and preserve internal structures of the original image, Hybrid Distortion Pooled Model (HDPM) [36] is proposed to take into account some common types of mixed distortion, such as image local distortion, content information loss, and image structure distortion. Firstly, SIFT descriptors are extracted from the original image and the retargeted image, then matched, and the local distortion degree of the retargeted image is measured by calculating SSIM between two matching blocks; then, the loss of content information is evaluated by using unmatched SIFT descriptors, and the degree of distortion of image structure is quantified by using Gray-level co-occurrence matrix (GCLM) [54] of two images; finally, these three distortion types are linearly fused to predict the final quality. Because some distortions are difficult to evaluate by pixels, a framework based on region evaluation is proposed to measure the shape distortion and information loss [38]. Firstly, this paper extracts the original image’s saliency map and a group of region segments used to determine the region information. Then, the SIFT Flow method establishes the corresponding relationship between the original and the retargeted image area. And the measures of local distortion, global distortion, and information loss of the retargeted image are calculated. Finally, the quality index is the weighted sum of these three indexes.

Since the perception of a human visual system depends on the edge of the image, the geometric distortion caused by image retargeting often distorts the edge, which seriously affects the retargeted image quality. Peng et al. [38] proposed Local and Global Geometric Distortions (LGGD) metric, which measures both local and global geometric distortions using a sketch token-based local edge descriptor (ST-LED) to represent geometric-aware features. This index named LGGD+has good compatibility and can be fused with existing IRQA indexes (including geometric and non-geometric distortion measures). It can be found from Tables 5 and 6 that the correlation between LGGD+and human subjective perception is not very high. However, LGGD is an excellent supplement to the existing IRQA. Designing a quality calculation model that covers all quality-related aspects isn’t easy. The proposed LGGD, which is also a breakthrough, measures geometric distortion more accurately and improves IRQA performance to a higher level.

In IRQA, full reference images are not always available, but reference images can be described with partial information. Wei et al. [39] proposed a reduced reference IRQA method to calculate the local structural similarity and global salience similarity between the original image and the retargeted image by extracting SIFT features and visual saliency features. Since pixels at different positions have different weights, weighted EMD is proposed to calculate the similarity between features. Finally, local EMD and global EMD are fused to obtain the final quality score. Unlike the IRQA method based on a full reference image, the reduced reference evaluation method only uses part of the reference image information. Usually, its correlation index is relatively low. This paper aims to achieve a breakthrough in reduced reference quality assessment of retargeted images.

4.2 Reducing the corresponding matching error

The first step of IRQA is to establish pixel correspondence between the original image and the retargeted image. However, the overlapping part of the texture and smooth area in the image is prone to wrong matching when establishing the corresponding pixel, which leads to inaccurate feature similarity measurement. Given this problem, researchers have done a lot of improvement work.

Hsu et al.[26] simultaneously considered perceptual geometric distortion(PGD) and information loss. Firstly, they measured the local Geometric Distortion Map (GDM) of the retargeted image using the local variance of the SIFT vector field of the image and warped the retargeted image back to the original image, and evaluated the prediction residual between the distorted image and the original image based on SIFT flow. It can be used as a Local Confidence Map (LCM) to suppress errors caused by a mismatch of SIFT streams. Then the visual saliency map (VSM) is used to determine the weight value of each block in GDM. Finally, PGD is calculated by combining GDM, VSM, and LCM. In addition, the information loss is measured by the saliency loss rate (SLR), which is the ratio of the saliency value of the retargeted image to the original image. To alleviate the measurement error in the spatial domain, a learning-based spatial-frequency domain metric (LSFM) [40] is proposed, which measures shape distortion and visual content change (local content change and global content change) in the spatial domain and frequency domain, respectively. It reduces the matching error in the spatial domain by measuring shape distortion and visual content change in the frequency domain without matching. Finally, machine learning is used to fuse different quality estimators. Similar to the confidence measurement used in the literature [26] to suppress error distortion detection caused by mismatch, Niu et al. proposed a metric named RN-IRQA [6]. The retargeted image is reconstructed by backward registration. The similarity map between the retargeted image and the reconstructed image is calculated to measure the accuracy of backward registration, which is recorded as registration confidence measure (RCM). For RCM, the blacker pixels, the lower the accuracy of backward registration. Then, it is fused with similarity measure to obtain reliable image fidelity measurement. This paper also puts forward the strategy of noticeability-based pooling (NBQ), which reflects the different sensitivities of different image regions to distortion. Compared with the previous Importance Weighted Pooling (IWP) strategy, it makes excellent progress.

4.3 Aspect ratio similarity

The proposed backward registration [24] reveals the geometric changes in image retargeting. As shown in Fig. 7, the geometric changes establish the block correspondence between the original and retargeted images. An Aspect Ratio Similarity (ARS) metric is designed to evaluate the local block changes and then evaluate the quality of the retargeted image. After that, most of the literature used ARS as a part of the measurement features. Zhang et al. [41] further proposed a measurement method based on Multiple-Level Feature (MLF), which used the backward registration algorithm to estimate the pixel correspondence between the original image and the retargeted image, then generated multi-level features according to the corresponding relationship, including low-level ARS [24], Edge Group Similarity [55] and high-level Face Block Similarity [56]. MLF simulates different attribute changes in image retargeting and solves the deformation problem of common high-level semantics, such as the inability to capture a face and structural deformation effectively. Guo et al. [42] mainly considered global distortion and local distortion. For measuring the deformation of the whole structure, elastic registration based on the B-line function and foreground retention rate are proposed. Improved ARS (IARS) and concentrated deletion ratio are proposed to measure local content loss. Because the retargeting algorithms are different, the retargeted blocks are irregular polygons. It is unreasonable to regard the longest edge as width and height in [24]. In this paper, for better measuring the change in local aspect ratio, the length and width are redefined: the maximum length and width of a block is divided by four and multiplied by the number of blocks divided by the retargeted image N as the length and width of the retargeted block; Finally, the weighted linear fusion method is used to fuse the four features and evaluate the quality of the retargeted image.

Fig. 7

ARS frame diagram [24].

Wu et al. [44] proposed a method based on multi-scale distortion-aware (MSDA) feature, which further improved the calculation of ARS [24]. Since different distortions appear on different image scales, this paper also obtains ARS features of the image from blocks with multiple scales (8*8 and 16*16). But for images with complex foreground objects, symmetrical structures, and textures, the consistency with subjective perception is not better than MLF. The proposed ARS has made different improvements to ARS proposed in [42]. This paper points out that for each image block before and after retargeting, the original ARS [24]. Calculation Equation (6) has limitations. The visual distortion problem will be ignored when a block in the original image does not match the retargeted image. $S_{ar} = \frac{2 r_{w} r_{h} + C}{r_{w}^{2} + r_{h}^{2} + C} e^{- a (r_{m} - 1)^{2}}$ (6) where C is a small constant, avoiding denominator to become 0. α is a parameter greater than 0, which balances information loss and visual distortion by adjusting the penalty degree of information loss. r_w and r_h are change ratios of the width and height of the retargeting block, and r_m is the absolute block size change which is used to measure visual distortion, r_m = (r_w + r_h)/ - 2. $(2 r_{w} r_{h} + C) / - (r_{w}^{2} + r_{h}^{2} + C)$ is the loss of vision measurement. exp (- a (r_m - 1) ²) indicates the loss of measurement information. $S_{iar} = {\begin{matrix} \frac{2 r_{w} r_{h}}{r_{w}^{2} + r_{h}^{2}} \exp (- a (r_{m} - 1)^{2}) r_{w} \neq 0 | | r_{h} \neq 0 \\ λ e^{- a (r_{m} - 1)^{2}} r_{w} = r_{h} = 0 \end{matrix}$ (7)

Therefore, λ is introduced as the penalty factor of visual distortion, and the improved equation is Equation (7).

To extract the accurate saliency map as the weight for fusing the improved ARS block scores, Zhang et al. [57] proposed a Visual Attention Fusion Framework (VAF) to calculate the saliency map. This framework integrates the salient map generated by the DCTS [27] algorithm and BSCA [58] algorithm, and the features of face and lines are enhanced. The low-level, medium-level, and high-level image features are considered, improving accuracy compared with the previous saliency detection methods. In 2019, Li et al. [43] used backward registration to establish pixel correspondence, and then different measurement methods were used depending on whether there are foreground objects. For images with significant foreground, foreground measure (including high-level semantic similarity feature and low-level size ratio feature) and global measure (including ARS feature and edge group similarity feature) are used; otherwise, the only global measure is performed. Niu et al. [6] also improved the quality of the ARS block [24], the improved equation is as follows [5]. $S_{IAR} (k) = {\begin{matrix} \frac{2 r_{w} (k) r_{h} (k)}{r_{w}^{2} (k) + r_{h}^{2} (k)} r_{w} (k), r_{h} (k) \neq 0 \\ λ r_{w} (k) = r_{h} (k) = 0 \end{matrix}$ (8) where k represents the index of the local block in the original image. r_w (k) and r_h (k) are change ratios of the width and height, respectively, the penalty factor λ is introduced to calculate the geometrical distortion when the whole local block is removed.

According to the improved ARS, area similarity and block-level shape similarity are used to measure the features to evaluate information loss and the geometric changes of each retargeted block. RCM and similarity measurement are fused to obtain block-level image fidelity measurement.

Huma beings are sensitive to the shape distortion of important regions of retargeted images; however, the existing IRQA algorithms seldom consider such situations, leading to the mismatch between subjective perception and objective evaluation results. In 2021, Yao et al. [59] proposed the ARS metric to measure the saliency region between the original and retargeted images. A new algorithm is used to extract the salient region of the retargeted image, and the calculation process is shown in Fig. 8. Backward registration mapping is established by using the SIFT Flow algorithm. L represents the position information of important areas of the original image, and R represents the position information of backward registration. Then, the intersection C of L and R is used to form the set of an important region of the retargeted image. In this paper, the width and height of the important region of the original image and retargeted image are marked as SO_W, SO_H, SR_W, SR_H, respectively; the change rates of width and height between the original image and the retargeted image are expressed as SO_RW = SR_W/ - SO_W, S_RH = SR_H/SO_H, respectively; the average value of the change rate is denoted as S_RM = (S_RH + S_RW)/ - 2. Therefore, the calculation equation of ARS of the important region is as Equation (9). $SARS = (\frac{2 \times S_{RW} \times S_{RH} + K}{S_{RW}^{2} + S_{RH}^{2} + K}) \times (e^{- a (S_{RM} - 1)^{2}})$ (9)

Fig. 8

Generation of salient regions of retargeted images [59].

In conclusion, ARS plays a vital role in IRQA.

4.4 Aesthetic measures

Users always favor beautiful photos since they provide users with a more pleasant feeling, so aesthetic image analysis has attracted more and more attention in computer vision [60 –62]. As part of feature measurement, aesthetics can improve the consistency between IRQA and subjective evaluation, so it has attracted the attention of many researchers.

Unlike the previous image retargeting method, which mainly considers structural distortion and content loss, the flowing methods think the feature measurements from several aspects. Liang et al. [45] put forward five key factors: 1) the retention of significant regions; 2) analysis of the influence of artifacts; 3) retaining the global structure; 4) observing aesthetic rules; 5) maintaining symmetry. The aesthetic rules of this paper mainly refer to the rule of thirds and visual balance, and each of them is assigned 50% weight during the calculation. Finally, the five measurement results are linearly fused to predict the aesthetic score. Yan et al. [46] proposed an open framework for IRQA, which uses the saliency weighting, CW-SSIM model with SIFT guidance [63], and aesthetic evaluation model, and combines them with eight existing retargeting evaluation methods (seven early distance methods and SSIM indicators) to assess the quality of image retargeting, and the aesthetic evaluation model is shown in Fig. 9. And 1320 images from the Aesthetic Visual Analysis (AVA) [64] dataset are selected as the training dataset, and 11 aesthetic features are extracted from each image as input to train neural networks to predict the aesthetic scores of subjective images. In [45], the features provided by these new and old models are used to train radial basis function networks [65, 66] to learn and predict the retargeted image quality. In 2019, Liu et al. [12] used General regression neural network (GRNN) [67] to simulate the combination of nine known objective quality evaluation indexes. They used nine-dimensional vectors to express the quality of the retargeted image, as shown in Equation (10).

Fig. 9

Aesthetic evaluation model [46].

$v (R_{ij}) = (Q_{1} (I_{i}, R_{ij}), Q_{2} (I_{i}, R_{ij}), . . ., Q_{9} (I_{i}, R_{ij}))$ (10) where v represents a vector, I_i represents the ith original image, and R_ij represents the jth retargeted image. Q_i means the nine evaluation indexes selected.

These indicators can be roughly divided into four categories: (1) retention of global structure; (2) reservation of significant areas; (3) the influence of visual distortion and introduced artifacts; (4) aesthetic measurement. The calculation method of aesthetic measurement is the same as [44]. The calculated scores retain the scores of the same original images and provide a reference for comparing the retargeting results of different original images.

4.5 Bi-directional measurement

Bi-directional measurement mainly refers to assessing quality in image retargeting through some features in a bi-directional way. In the early stage, the loss of relevant content was especially taken into account by bi-directional measurement. Zhang et al. [37] used bidirectional measurement to evaluate regional information loss for foreground region and background region information loss. Liu et al. [40] used the measurement method of BDS to measure the change of local visual content. Chen et al. [47] first incorporated the measurement of image Natural Scene Statistics (NSS) into IRQA. Then the significant global structure distortion and bidirectional significant information loss are measured, and a bi-directional natural salient scene distortion model (BNSSD) is proposed. Finally, the above three measures are input into the support vector regression model to predict the scores. Among them, the forward saliency information loss measure calculates how much saliency information of the original image is retained in the retargeted image. Backward salient information loss measure calculates the salient information of the original image correctly recovered from the retargeted image. In 2018, Oliveira et al. [48] proposed bi-directional Importance Map Similarity (BIMS). The critical step of the method is to measure features, such as bi-directional quality score, retargeting ratio feature, and keypoint matching feature, in a bi-directional way. The bi-directional feature is helpful to estimate the relative position and analyze how some information is retained after retargeting. When the related content is lost, or visual distortion occurs, accurate feature similarity measurement can be made by analyzing these positions. Bidirectional measures of geometric distortion also began to develop. Shao et al. [49] proposed a transform-Aware Similarity (TRASIM) measurement method, which includes:(1) bidirectional geometric distortion (BDGD) measurement, which used salient values as weights to calculate the forward and backward geometric distortions; (2) bidirectional information loss (BDIL) which calculate the changes of salient regions by forward and backward retargeting; (3) global salient structure distortion (GSSD). The TRASIM framework is shown in Fig. 10, which is different from the previous IRQA method. A similarity transformation is established by bi-directional re-warping to simulate different types of retargeting operators, and geometric distortion and content loss are measured based on similarity transformation to determine the quality of the retargeted image.

Fig. 10

TRASIM frame diagram [49].

4.6 Image retargeting quality evaluation method based on deep learning

Some evaluation methods use simple regression algorithms to fuse features, which cannot simulate the perception process of the retargeted images in the human visual system. So Jiang et al. [50] trained a segmented stacked AutoEnCoder based on geometric shape and content matching [68, 69] to fuse features and output two scores. Finally, the final image retargeting quality score is obtained by combining the two scores through a weight mechanism. In recent years, IRQA based on deep learning mainly used neural networks to extract semantic features as high-level measuring features.

Fu et al. [51] extracted manual features by establishing the pixel correspondence between the original and retargeted images to measure structural distortion and information loss. Then the pre-trained VGG16 network [70] was used to construct encoders to extract deep-learned features for measuring texture similarity and semantic similarity. As shown in Fig. 11, texture similarity is calculated by convolution1-1 of VGG16, and semantic features are measured by the output of the second fully connected layer. Finally, manual features and deep-learned features are combined to evaluate the quality of retargeted images. In [43], foreground measurement (including high-level semantic similarity feature and low-level size ratio feature) and global measure (including improved ARS feature and edge group similarity feature) are used as extracted features for images with foreground objects. Fu’s method directly reshapes the image to the input size required by the network when extracting semantic features, which will completely lose the original aspect ratio information. An adaptive input method is designed in [43] to meet the requirement that the input size of the input image is the same as that of the network without changing the original image’s aspect ratio, which further improves the accuracy of semantic measurement. Compared with the MLF evaluation method, although ARS and edge group features are considered in MLF, it only finds a face in semantic composition. In this paper, the semantic analysis of foreground objects is carried out, so its effect is relatively good.

Fig. 11

Deep feature map extracted by vgg16 [51].

The literature mentioned above all regard semantic features as additional features. However, in [52], semantic features are the primary measurement features, and the instance is used as the basic semantic unit. When comparing the original image with the retargeted image, the original image can be segmented using the Mask R-CNN model [71]. Still, this model cannot be used for instance segmentation of retargeted images. The backward registration strategy establishes an instance of pixel correspondence to reconstruct the retargeted image. Then, the semantic features at the instance level (shape distortion, size similarity, information loss, and position movement) are extracted. An adaptive pooling method based on semantics is proposed to integrate these semantic features. Finally, the global feature ARS is integrated into the instance features, which further improves the consistency with subjective evaluation.

4.7 Fusion method of the above features

Effective image quality prediction depends not only on selecting features but also on the mechanism of fusing all features into a single quality score. However, the contribution of each feature to the final quality score may be different. Some common-used basic pooling methods include simple summation, multiplication, and linear combination. Still, these methods have some shortcomings in that they assume each feature’s relative importance, which lacks a convincing basis [35]. In recent years, support vector regression (SVR) algorithms including linear kernel SVR (l-SVR), polynomial kernel SVR (poly-SVR), radial basis function kernel (RBF-SVR) [72] are often used to train prediction functions to fuse different features and have achieved good results; In addition, the fusion operation in Ambiguous D-means fusion clustering [73] can fuse the best features to generate a satisfactory result image, which may be used to fuse the above features and improve the quality of images.

5 IRQA dataset, evaluation index, and performance comparison

5.1 Dataset

The commonly used image retargeting quality evaluation dataset is RetargetMe [7], CUHK [19], and NRID [26]. Table 4 briefly describes these three datasets from the aspects of the number of original images, retargeting operators, the number of retargeted images, and image attributes.

(1) RetargetMe dataset.

It is the first dataset for IRQA, which includes 37 original images. Each image has one or more of six attributes, line/edge, face/person, foreground object, texture, geometric structure, and symmetry. The length and width of each original image are reduced by 25% or 50%, and 8 different retargeting operators are used respectively, including: 1) Seam Carving (SEAM) [74]; 2) Nonhomogeneous Warping (WARP) [75]; 3) Scale and Stretch (SCST) [76]; 4) Multi Operator (MOP) [11]; 5) Uniform scaling (SCAL); 6) ShiftMap, SHIF) [77]; 7) Manual Cropping (CROP); and 8) Streaming Video (STVI) [78]. This dataset contains a total of 269 retargeted images. Subjective research adopts the method of paired comparison [79] by showing participants two different retargeted images and allowing the participants to choose the image with the best visual effect, then vote for it. The total number of votes obtained is taken as the subjective score of the retargeted image.

(2) CUHK dataset.

CUHK dataset contains 57 original images, and each original image adopts three different retargeting operations. Like the RetargetMe dataset, the length and width are scaled by 25% or 50%. The retargeting operators used by each image may be different because they are randomly selected from 10 representative retargeting operators. The retargeting operators come from 8 operators in RetargetMe and two other operators (Optimized seam cutting and scale (SCSL) [80] and Energy-Based Deformation, ENER) [81]). The dataset consisted of 171 retargeted images using a subjective evaluation method: participants rate each image on a five-point scale of “very bad”, “bad”, “fair”, “good” and “very good”, and use the above scores to calculate the Mean Opinion Score (MOS) of each retargeted image.

Table 4
Common datasets for image retargeting.

Dataset RetargetMe NRID CUHK

Original images 37 35 57

Retargeting operator 8 5 10

Retargeted images. 296 175 171

Line/edge 25 15 –

Face/person 15 15 –

Foreground object 18 16 –

Texture 6 12 –

Symmetry 6 6 –

Geometry 16 8 –

Scaling ratio 0.5;0.75 0.75 0.5;0.75

Subjective score Pair-wise Pair-wise MOS

Dataset	RetargetMe	NRID	CUHK
Original images	37	35	57
Retargeting operator	8	5	10
Retargeted images.	296	175	171
Line/edge	25	15	–
Face/person	15	15	–
Foreground object	18	16	–
Texture	6	12	–
Symmetry	6	6	–
Geometry	16	8	–
Scaling ratio	0.5;0.75	0.75	0.5;0.75
Subjective score	Pair-wise	Pair-wise	MOS

(3) NRID dataset.

NRID dataset contains 35 original images, and each image has five retargeted images. The retargeting operators include the SAME, WARP, MULTI, SCAL, and SIFT, containing 175 retargeted images. Each image also has six attributes mentioned by RetargetMe. The subjective evaluation method is the same as that used in RetargetMe. Because there are few experimental results in this dataset, the performance comparison of IRQA in NRID is not listed in this paper.

5.2 Commonly-used evaluation indicators

(1) Kendall rank correlation coefficient (KRCC).

KRCC is used to measure the correlation between predicted values and MOS. n_c represents the number of image pairs consistent with subjective ranking, n_d is the number of image pairs inconsistent with subjective ranking, and n represents the ranking sequence length. $KRCC = (n_{c} - n_{d}) / - (0.5 (n - 1))$ (11)

(2) Pearson linear correlation coefficient (PLCC).

PLCC is used to describe the linear correlation between predicted values and MOS. X, Y are the predicted value and the ground truth, respectively. cov (X, Y), is the covariance, and σ_X and σ_Y are the corresponding variances. PLCC is easily affected when there is a maximum or minimum value in the predicted value. $PLCC (X, Y) = cov (X, Y) / - (σ_{X} σ_{Y})$ (12)

(3) Spearman rank-order correlation coefficient (SRCC).

SRCC is used to measure the monotonic relationship between two variables. r_W and r_Yrepresent the rank values of X and Y, respectively. For example, the value with the highest score is rated as level 1, the value with the lowest score is rated as level 9, and the other values are rated as an integer between 2 and 9. SRCC mainly evaluates the hierarchical correlation of two sets of data, which can alleviate the problem that PLCC is susceptible to extreme values. $SRCC (X, Y) = cov (r_{X}, r_{Y}) / - (σ_{r_{X}} σ_{r_{Y}})$ (13)

(4) Root Mean Squared Error (RMSE).

RSME estimates the difference between the predicted value and the true value MOS. m is the number of values. $RMSE = \sqrt{\frac{1}{m} \sum_{i = 1}^{m} (X_{i} - Y_{i})^{2}}$ (14)

(5) Outward ratio (OR)

OR represents the ratio of the number of outliers to the total number of objective evaluation scores. The abnormal value belongs to values outside the range [MOS - 2σ, MOS + 2σ], σ indicates the standard deviation of the objective score. $OR = N_{false} / - N_{total}$ (15)

The larger PLCC and SRCC, the better the correlation between subjective and objective scores among the above evaluation indexes. The smaller RMSE and OR, the closer the subjective score is to the objective score.

5.3 Performance comparison

The performances of IRQA methods under the registration algorithms listed in Table 1 are compared, respectively, as shown in Tables 5 and 6.

Table 5
Performance comparison of IRQA model in RetargetMe dataset

Category Metric. Average KRCC per attribute. Overall KRCC

Line Face Foreground Texture Geometric Symmetry Mean Std p-val

Edge People Object Structure

SIFT/SIFT flow IR-SSIM [25] 0.309 0.452 0.377 0.321 0.313 0.333 0.363 0.271 1e-3

PGD + SLR [26] 0.431 0.390 0.389 0.286 0.438 0.523 0.415 0.296 6e-10

GLS [35] 0.382

BNSSD [47] 0.4480 0.5518 0.4943 0.4234 0.4665 0.4708 0.4734 0.2565 —

Karim [83] 0.453 0.589 0.564 0.494 0.431 0.380 0.494 0.261 1e-10

Zhang [82] 0.632 0.605 0.597 0.588 0.591 0.651 0.625 0.231 2e-21

Fu [51] 0.497 0.472 0.468 0.393 0.545 0.631 0.494 0.243 1.6e-14

TRASIM [49] 0.465 0.600 0.532 0.476 0.522 0.524 0.504 0.237 4e-15

Ranking [12] 0.591 0.619 0.611 0.445 0.607 0.476 0.575 — 1e-17

SURF LGI [28] 0.425 0.268 2.6e-09

Zhang [29] 0.674 0.680 0.602 0.522 0.621 0.497 0.599 0.076 —

Backward ARS [24] 0.463 0.519 0.444 0.330 0.505 0.464 0.452 0.283 1e-11

Registration MLF [41] 0.486 0.605 0.544 0.384 0.536 0.536 0.512 0.251 1e-14

Jiang [50] 0.466 0.512 0.452 0.434 0.515 0.443 0.476 0.243 —

Guo [42] 0.486 0.600 0.567 0.393 0.536 0.464 0.512 0.238 —

Li [43] 0.521 0.598 0.619 0.473 0.554 0.548 0.552 0.221 —

MSDA [44] 0.511 0.633 0.539 0.455 0.571 0.535 0.533 0.265 9e-21

RN-IRQA [6] 0.540 0.629 0.568 0.500 0.612 0.607 0.568 0.243 2e-25

INSEM [52] 0.586 0.562 0.552 0.607 0.594 0.469 0.537 0.188 —

LGGD+ [38] 0.525 0.571 0.524 0.473 0.538 0.476 0.517 0.244 —

w/o Registration Ma [32] 0.477 — —

Jiang [33] 0.413 0.282 —

Category	Metric.	Average KRCC per attribute.	Overall KRCC
SIFT/SIFT flow	IR-SSIM [25]	0.309	0.452	0.377	0.321	0.313	0.333	0.363	0.271	1e-3
	PGD + SLR [26]	0.431	0.390	0.389	0.286	0.438	0.523	0.415	0.296	6e-10
	GLS [35]							0.382
	BNSSD [47]	0.4480	0.5518	0.4943	0.4234	0.4665	0.4708	0.4734	0.2565	—
	Karim [83]	0.453	0.589	0.564	0.494	0.431	0.380	0.494	0.261	1e-10
	Zhang [82]	0.632	0.605	0.597	0.588	0.591	0.651	0.625	0.231	2e-21
	Fu [51]	0.497	0.472	0.468	0.393	0.545	0.631	0.494	0.243	1.6e-14
	TRASIM [49]	0.465	0.600	0.532	0.476	0.522	0.524	0.504	0.237	4e-15
	Ranking [12]	0.591	0.619	0.611	0.445	0.607	0.476	0.575	—	1e-17
SURF	LGI [28]							0.425	0.268	2.6e-09
	Zhang [29]	0.674	0.680	0.602	0.522	0.621	0.497	0.599	0.076	—
Backward	ARS [24]	0.463	0.519	0.444	0.330	0.505	0.464	0.452	0.283	1e-11
Registration	MLF [41]	0.486	0.605	0.544	0.384	0.536	0.536	0.512	0.251	1e-14
	Jiang [50]	0.466	0.512	0.452	0.434	0.515	0.443	0.476	0.243	—
	Guo [42]	0.486	0.600	0.567	0.393	0.536	0.464	0.512	0.238	—
	Li [43]	0.521	0.598	0.619	0.473	0.554	0.548	0.552	0.221	—
	MSDA [44]	0.511	0.633	0.539	0.455	0.571	0.535	0.533	0.265	9e-21
	RN-IRQA [6]	0.540	0.629	0.568	0.500	0.612	0.607	0.568	0.243	2e-25
	INSEM [52]	0.586	0.562	0.552	0.607	0.594	0.469	0.537	0.188	—
	LGGD+ [38]	0.525	0.571	0.524	0.473	0.538	0.476	0.517	0.244	—
w/o Registration	Ma [32]							0.477	—	—
	Jiang [33]							0.413	0.282	—

Table 6

Performance comparison of IRQA model in CUHK dataset

Category	Metric	PLCC	SRCC	RMAE	OR
SIFT/SIFT flow	GLS [35]	0.4622	0.476	10.932	0.1345
	BNSSD [47]	0.7528	0.7514	—	—
	Zhang [82]	0.7974	0.7984	8.378	0.0643
	TRASIM [49]	0.7586	0.7216	8.609	0.0103
SURF	Zhang [29]	0.7528	0.7754	—	0.1331
Backward Registration	ARS [24]	0.6835	0.6693	9.855	0.0702
	MLF [41]	0.7577	0.7383	8.525	0.0294
	Guo [42]	0.6844	0.6588	9.8502	0.0409
	Jiang [50]	0.7012	0.6732	8.364	0.0574
	Li [43]	0.7745	0.7706	8.5421	0.0702
	MSDA [44]	0.7713	0.7717	8.593	0.035
	RN-IRQA [6]	0.7925	0.7619	8.234	0.0409
	INSEM [52]	0.798	0.748	7.905	0.023
	LGGD+ [38]	0.738	0.724	9.477	—
w/o Registration	Ma [32]	0.3107	0.2973	—	—
	Jiang [34]	0.644	0.616	10.763	—

In Tables 5 and 6, red fonts have the best representation performance, followed by bold fonts, and yellow fonts and green fonts rank third and fourth, respectively. Table 5 lists the performance comparison of different IRQA methods in the RetargetMe dataset. The mean and standard deviation of KRCC of each IRQA measure and the average KRCC of an attribute are given. It can be found that IRQA methods with good performance are concentrated in the backward registration algorithm. SURF algorithm has relatively poor performance in symmetric structure but has good performance in other aspects. Zhang et al. [82] achieve the best performance in symmetric structure, probably because of inconsistency detection at the area/block level. It can avoid the shape distortion caused by over-compressing or stretching the scale distortion of foreground objects and important areas. And in [52], semantic information is the primary measurement feature, so it achieves the best performance in textures. In literature [43], it detects whether there are foreground objects, so it has the best score on the performance of foreground objects. Table 6 lists the performance comparison of IRQA on the CUHK dataset. Similar to Table 5, IRQA methods with the highest performance rank mainly focus on the backward registration algorithm. The performances of the proposed evaluation methods in [6 , 82] are outstanding. As there are no experimental results on these two datasets and corresponding indicators in [31], and the source code of the paper [31] has not been published, our paper does not list its performance comparison.

Table 9

Descriptions of abbreviation and notation used in this paper

Abbreviations	Descriptions
and notations
SIFT	Scale-invariant feature transform
DSP	Deformable spatial pyramid
SURF	Speeded up robust features
SSIM	Structural similarity index measure
MOS	Mean Opinion Score
SVR	Support vector regression
Lc	The continuous resampling space corresponding to the original image
L	Resampling grid
Cov	Covariance, a measure of the overall error of two variables
σ	Variance, which describes the degree of dispersion of a variable

5.4 Cross-dataset comparison

To verify the generalization ability of IRQA, cross-dataset comparison (cross-dataset comparison) was put forward in 2018 and later, that is, training parameters on one dataset and testing on another dataset. In these experiments, a virtual DMOS is set for the quality of each retargeted image in RetargetMe and NRID datasets, which can correspond to CUHK. Thereby it ensures that the three datasets have the same score standard. Literatures [6 , 52] have carried out such experiments, and some comparative results are shown in Tables 7 and 8. The bold font on the right side is the score of the same training dataset and testing dataset, which shows that the cross-dataset comparison result is second only to those for the same training dataset and testing dataset, which proves that the proposed evaluation index has good generalization ability.

Table 7
Cross databases performance in [46]

Dataset Index

CUHK (train) KRCC(mean) 0.496/0.512

RetargetMe (test) KRCC (std) 0.256/0.251

CUHK (train) PLCC 0.730/0.7577

Testing (test) SRCC 0.713/0.7383

RMSE 9.230/8.525

∼ OR 0.047/0.0294

Dataset	Index
CUHK (train)	KRCC(mean)	0.496/0.512
RetargetMe (test)	KRCC (std)	0.256/0.251
CUHK (train)	PLCC	0.730/0.7577
Testing (test)	SRCC	0.713/0.7383
	RMSE	9.230/8.525
∼	OR	0.047/0.0294

Table 8

Cross databases performance in [69]

Train	Test	PLCC	SRCC
RetargetMe	CUHK	0.452	0.224
	NRID	0.412	0.243
CUHK	RetargetMe	0.465	0.312
	NRID	0.532	0.314
NRID	RetargetMe	0.422	0.246
	CUHK	0.463	0.247

6 Conclusions and future

IRQA is important in image retargeting, mainly applied in two aspects. Firstly, it can evaluate the performance of different image retargeting algorithms; secondly, it is widely used to optimize the multi-operator operation of image retargeting, determining which operator to use in each iteration to obtain the best-retargeted image. This paper mainly introduces the common-used registration algorithms. Then, according to different feature measurement methods, IRQA is classified and summarized. The datasets and evaluation indicators are outlined, and the performance of IRQA using different registration algorithms is compared. Although some achievements have been made in image retargeting quality assessment, it still faces the following limitations and challenges.

Recently, most IRQA methods first establish the dense correspondence between the original image and the retargeted image. However, some areas of the image face the problem of inaccurate registration, which will lead to the inconsistency between objective and subjective evaluations. With the continuous development of deep learning, we hope to use deep learning to establish the correspondence between the original image and the retargeted image to further solve inaccurate image registration. Registration methods based on deep understanding have gradually matured in other fields, such as remote sensing images [84] and image segmentation [85]. Like [84] said, using CNN to generate robust multi-scale feature descriptors to establish matching has obtained better results than SIFT matching. In the future, we will try these methods in IRQA.

Integrating semantic features in quality assessment is increasingly influential [86]. At present, low-level and intermediate-level features are considered more in image quality evaluation methods, and high-level semantic features are often considered additional features. Deep learning [87] is a powerful hierarchical feature representation tool for image classification and target recognition. In the future, we hope to accurately detect the semantic features of retargeted images and use the extracted semantic features to train the network for quality prediction.

IRQA often needs to evaluate the results from multiple angles and early adding aesthetic and symmetry evaluation to quality evaluation has achieved good results. However, the definition of symmetry detection is unclear, and its accuracy is not high, limiting its application. In recent years, the symmetry of image retargeting has also been studied [88, 89]. Aesthetic evaluation [90, 91] with deep learning has also been made progress. If symmetry and aesthetic assessments can be integrated with IRQA, the consistency between objective and subjective evaluations will improve.

In recent years, fuzzy set theory has been widely used because it is closer to the thinking process. Inspired by the fusion operation in the ambiguous D-means fusion clustering algorithm [73], we can use the fuzzy threshold method to convert the image to the fuzzy feature plane through the membership function, assigning different fusion weights to different measurement features. This method will further improve the quality of IRQA.

With the constant maturity of the IRQA algorithm, the quality evaluation of stereoscopic image retargeting [92, 93] will be further developed.

References

Zhou

, Chen

and Li

, Weakly supervised reinforced multi-operator image retargeting, IEEE Transactions on Circuits and Systems for Video Technology 31(1) (2021), 126–139.

Ahmadi

, Karimi

and Samavi

, Context-aware saliency detection for image retargeting using convolutional neural networks, Multimedia Tools and Applications 80(8) (2021), 11917–11941.

Danon

, Arar

, Cohen-Or

et al., Image resizing by reconstruction from deep features, Computational Visual Media 7(4) (2021), 453–466.

Kajiura

, Kosugi

, Wang

et al., Self-play reinforcement learning for fast image retargeting, Proceedings of the 28th ACM International Conference on Multimedia 2020, 1755–1763.

Mei

, Guo

, Sun

et al., Deep supervised image retargeting, 2021 IEEE International Conference on Multimedia and Expo (ICME) 2021, 1–6.

Niu

, Zhang

, Wu

et al., Image retargeting quality assessment based on registration confidence measure and noticeability-based pooling, IEEE Transactions on Circuits and Systems for Video Technology 31(3) (2021), 972–985.

Rubinstein

, Gutierrez

, Sorkine

et al., A comparative study of image retargeting, ACMSIGGRAPH Asia 2010 Papers 2010, 1–10.

Kasutani

, Yamada

The mpeg-7 color layout descriptor: a compact image feature description for highspeed image/video segment retrieval, Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205) 2001, 674–677.

Manjunath

B.S.

, Ohm

J.R.

, Vasudevan

V.V.

et al., Color and texture descriptors, IEEE Transactions on Circuits and Systems for Video Technology 11(6) (2001), 703–715.

10.

Simakov

, Caspi

, Shechtman

et al., Summarizing visual data using bidirectional similarity, 2008 IEEE Conference on Computer Vision and Pattern Recognition 2008, 1–8.

11.

Rubinstein

, Shamir

and Avidan

, Multi-operator media retargeting, ACM Transactions on graphics (TOG) 28(3) (2009), 1–11.

12.

Liu

Y.J.

, Han

, Ye

et al., Ranking-preserving cross-source learning for image retargeting quality assessment, IEEE Transactions on Pattern Analysis and Machine Intelligence 42(7) (2020), 1798–1805.

13.

Pele

, Werman

Fast and robust earth mover’s distances, 2009 IEEE 12th international conference on computer Vision 2009, 460–467.

14.

Lin

, Zhou

, Chen

DeepIR: a deep semantics driven framework for image retargeting, 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) 2019, 4–59.

15.

Liu

, Yuen

and Torralba

, Sift flow: dense correspondence across scenes and its applications, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(5) (2010), 978–994.

16.

Lowe

Object recognition from local scale-invariant features, Proceedings of the seventh IEEE international conference on computer Vision 1999. 1150–1157.

17.

Yan

, Niu

, Bare

et al., Semantic segmentation guided pixel fusion for image retargeting, IEEE Transactions on Multimedia 22(3) (2020), 676–687.

18.

Chen

L.C.

, Zhu

, Papandreou

et al., Encoder-decoder with atrous separable convolution for semantic image segmentation, Proceedings of the European conference on computer vision (ECCV) 2018, 801–818.

19.

, Lin

, Deng

et al., Image retargeting quality assessment: a study of subjective scores and objective metrics, IEEE Journal of Selected Topics in Signal Processing 6(6) (2012), 626–639.

20.

Karimi

, Entezami

Quality assessment for retargeted images: a review, 2020 International Conference on Machine Vision and Image Processing (MVIP) 2020, 1–6.

21.

Asheghi

, Salehpour

, Khiavi

A.M.

et al., A comprehensive review on content-aware image retargeting: From classical to state-of-the-art methods, , Signal Processing 195 (2022), 1–19.

22.

Bay

, Tuytelaars

, Gool

L.V.

Surf: speeded up robust features, European conference on computerVision. Springer 2006, 404–417.

23.

Kim

, Liu

, Sha

et al., Deformable spatial pyramid matching for fast dense correspondences, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2013, 2307–2314.

24.

Zhang

, Fang

, Lin

et al., Backward registration-based aspect ratio similarity for image retargeting quality assessment, IEEE Transactions on Image Processing 25(9) (2016), 4286–4297.

25.

Fang

, Zeng

, Wang

et al., Objective quality assessment for image retargeting based on structural similarity, IEEE Journal on Emerging and Selected Topics in Circuits and Systems 4(1) (2014), 95–105.

26.

Hsu

C.C.

, Lin

C.W.

, Fang

et al., Objective quality assessmentfor image retargeting based on perceptual geometric distortion andinformation loss, IEEE Journal of Selected Topics in SignalProcessing 8(3) (2014), 377–389.

27.

Fang

, Chen

, Lin

et al., Saliency detection in the compressed domain for adaptive image retargeting, IEEE Transactions on Image Processing 21(9) (2012), 3888–3901.

28.

Zhang

, Sander

P.V.

, Bermak

, Registration based retargeted image quality assessment, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2017, 1258–1262.

29.

Zhang

, Yu

, Guo

et al., Content-aware retargeted imagequality assessment, Information 10(3) (2019), 111.

30.

Barnes

, Shechtman

, Goldman

D.B.

et al., The generalized patchmatch correspondence algorithm, European Conference on Computer Vision 2010, 29–43.

31.

Shigwan

, Birajdar

Full reference image quality assessment of perceptual distortion based on image retargeting, Proceedings of the International Conference on Communication and Signal Processing 2016 (ICCASP 2016) 2017.

32.

, Xu

, Zhang

et al., No-reference retargeted image quality assessment based on pairwise rank learning, IEEE Transactions on Multimedia 18(11) (2016), 2228–2237.

33.

Oliva

and Torralba

, Modeling the shape of the scene: a holistic representation of the spatial envelope, International Journal of Computer Vision 42(3) (2001), 145–175.

34.

Jiang

, Shao

, Lin

et al., Learning sparse representation for objective image retargeting quality assessment, IEEE Transactions on Cybernetics 48(4) (2018), 1276–1289.

35.

Zhang

, Kuo

C.C.J.

An objective quality of experience (QoE) assessment index for retargeted images, Proceedings of the 22nd ACM international conference on Multimedia 2014, 257–266.

36.

Lin

J.X.

, Zhu

L.L.

, Chen

Z.B.

et al., Objective quality assessment for image retargeting based on hybrid distortion pooled model, 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX) 2015, 1–6.

37.

Zhang

, Ngan

K.N.

Region-based image retargeting quality assessment, 2015 IEEE International Conference on Image Processing (ICIP) 2015, 1757–1761.

38.

Peng

, Jiang

, Shao

et al., LGGD+: image retargeting quality assessment by measuring local and global geometric distortions, IEEE Transactions on Circuits and Systems for Video Technology 2021.

39.

Wei

, Zhao

and Peng

, Reduced reference quality assessment for image retargeting by earth mover’s distance, Applied Sciences 11(20) (2021), 9776.

40.

Liu

, Lin

, Chen

et al., Image retargeting quality assessment based on support vector regression, , Signal Processing: Image Communication 39 (2015), 444–456.

41.

Zhang

, Lin

, Li

et al., Multiple-level feature-based measure for retargeted image quality, IEEE Transactions on Image Processing 27(1) (2018), 451–463.

42.

Guo

, Hao

and Yu

, Image retargeting quality assessment based on content deformation measurement, , Signal Processing: Image Communication 67 (2018), 171–181.

43.

, Guo

and Jin

, A content-aware image retargeting quality assessment method using foreground and global measurement, , IEEE Access 7 (2019), 91912–91923.

44.

, Zhang

and Niu

, Retargeted image quality assessment based on multi-scale distortion-aware feature, Journal of Beijing University of Aeronautics and Astronautics 45(12) (2019), 2487.

45.

Liang

, Liu

Y.J.

and Gutierrez

, Objective quality prediction of image retargeting algorithms, IEEE Transactions on Visualization and Computer Graphics 23(2) (2017), 1099–1110.

46.

Yan

, Bare

, Li

et al., Learning quality assessment of retargeted images, , Signal Processing: Image Communication 56 (2017), 12–19.

47.

Chen

, Lin

, Liao

et al., Full reference quality assessment for image retargeting based on natural scene statistics modeling and bi-directional saliency similarity, IEEE Transactions on Image Processing 26(11) (2017), 5138–5148.

48.

Oliveira

S.A.F.

, Alves

S.S.A.

, Gomes

J.P.P.

et al., A bi-directional evaluation-based approach for image retargeting quality assessment, Computer Vision and Image Understanding 168 (2018), 172–181.

49.

Shao

, Fu

, Jiang

et al., Transformation-aware similarity measurement for image retargeting quality assessment via bidirectional rewarping, IEEE Transactions on Systems 51(5) (2021), 3053–3067.

50.

Jiang

, Yang

, Meng

et al., A deep evaluator for image retargeting quality by geometrical and contextual interaction, IEEE Transactions on Cybernetics 50(1) (2020), 87–99.

51.

, Shao

, Jiang

et al., Quality assessment of retargeted images using hand-crafted and deep-learned features, IEEE Access 6 (2018), 12008–12018.

52.

, Li

, Wu

et al., Quality evaluation for image retargeting with instance semantics, IEEE Transactions on Multimedia 23 (2021), 2757–2769.

53.

Zhai

and Min

, Perceptual image quality assessment: a survey, Science China Information Sciences 63(11) (2020), 1–52.

54.

Haralick

R.M.

, K Shanmugam and I H Dinstein, Textural features for image classification, , IEEE Transactions on Systems, Man, and Cybernetics 6 (1973), 610–621.

55.

Dollár

, Zitnick

C.L.

Structured forests for fast edge detection, Proceedings of the IEEE International Conference on Computer Vision 2013, 1841–1848.

56.

Megvii

Face++research toolkit (2013).

57.

Zhang

, Niu

, Lin

et al., Visual attention fusion framework for image retargeting quality assessment, , IOP Conference Series: Earth and Environmental Science 234 (2019), 012064.

58.

Qin

, Lu

, Xu

et al., Saliency detection via cellular automata, Proceedings of the IEEE conference on computer vision and pattern Recognition 2015, 110–119.

59.

Yao

, Tang

, Lu

et al., Image retargeting quality assessment based on aspect ratio similarity of salient region, 2021 International Conference on Computer Communication and Artificial Intelligence (CCAI) 2021, 47–51.

60.

Wang

, Wang

, Yamasaki

Image aesthetics prediction using multiple patches preserving the original aspect ratio of contents, arXiv 2020.

61.

Kao

, He

and Huang

, Deep aesthetic quality assessment with semantic information, IEEE Transactions on Image Processing 26(3) (2017), 1482–1495.

62.

, Lin

, Shen

et al., Deep multi-patch aggregation network for image style, aesthetics, and quality estimation, Proceedings of the IEEE international conference on computer Vision 2015, 990–998.

63.

Sampat

M.P.

, Z.Wang, S. Gupta, et al., Complex wavelet structural similarity: a new image similarity index, IEEE transactions on image Processing 18(11) (2009), 2385–2401.

64.

Murray

, Marchesotti

, Perronnin

AVA: a largescale database for aesthetic visual analysis, 2012 IEEE conference on computer vision and pattern Recognition 2012, 2408–2415.

65.

Broomhead

D.S.

, Lowe

Radial basis functions, multi-variable functional interpolation and adaptive networks, Royal Signals and Radar Establishment Malvern (United Kingdom), 1988.

66.

Schwenker

, Kestler

H.A.

and Palm

, Three learning phases for radial-basis-function networks, Neural Networks 14(4–5) (2001), 439–458.

67.

Specht

D.F.

, A general regression neural network, IEEE transactions on neural Networks 2(6) (1991), 568–576.

68.

Larochelle

, Erhan

, Courville

et al., An empirical evaluation of deep architectures on problems with many factors of variation, Proceedings of the 24th international conference on Machine Learning 2007, 473–480.

69.

Bengio

, Lamblin

, Popovici

et al., Greedy layer-wise training of deep networks, , Advances in Neural Information Processing Systems 19 (2006).

70.

Simonyan

, Zisserman.

Very deep convolutional networks for large-scale image recognition, arXiv (2014).

71.

, Gkioxari

, Dollár

et al., Mask r-cnn, Proceedings of the IEEE international conference on computer Vision 2017, 2961–2969.

72.

Wang

, Shao

, Jiang

et al., Combining retargeting quality and depth perception measures for quality evaluation of retargeted stereopairs, IEEE Transactions on Multimedia, 2021.

73.

Singh

and Bose

S.S.

, Ambiguous D-means fusion clustering algorithm based on ambiguous set theory: Special application in clustering of CT scan images of COVID-19, Knowledge-Based Systems 231 (2021), 1–26.

74.

Avidan

, Shamir

Seam carving for content-aware image resizing, ACM SIGGRAPH 2007 Papers 2007.

75.

Wolf

, Guttmann

, Cohen-Or

Cohen-Or, Non-homogeneous content-driven video-retargeting, 2007 IEEE 11th international conference on computer Vision 2007, 1–6.

76.

Wang

Y.S.

, Tai

C.L.

, Sorkine

et al., Optimized scaleand-stretch for image resizing, ACMSIGGRAPH Asia 2008 Papers 2008, 1–8.

77.

Pritch

, Kav-Venaki

, Peleg

Shift-map image editing, 2009 IEEE 12th international conference on computer Vision 2009, 151–158.

78.

Krähenbühl

, Lang

, Hornung

et al., A system for retargeting of streaming video, ACMSIGGRAPH Asia 2009 Papers 2009, 1–10.

79.

Bradley

R.A.

and Terry

M.E.

, Rank analysis of incomplete block designs: i. the method of paired comparisons, Biometrika 39(3/4) (1952), 324–345.

80.

Dong

, Zhou

, Paul

J.C.

et al., Optimized image resizing using seam carving and scaling, ACM Transactions on Graphics (TOG) 28(5) (2009), 1–10.

81.

Karni

, Freedman

, Gotsman

Energy-based image deformation, Computer Graphics Forum 2009, 1257–1268.

82.

Zhang

, Ngan

K.N.

, Ma

et al., Objective quality assessment of image retargeting by incorporating fidelity measures and inconsistency detection, IEEE Transactions on Image Processing 26(12) (2017), 5980–5993.

83.

Karimi

, Samavi

, Karimi

et al., Quality assessment of retargeted images by salient region deformity analysis, Journal of Visual Communication and Image Representation 43 (2017), 108–118.

84.

Yang

, Dan

and Yang

, Multi-temporal remote sensing image registration using deep convolutional features, , IEEE Access 6 (2018), 38544–38555.

85.

, Niethammer

DeepAtlas: joint semisupervised learning of image registration and segmentation, International Conference on Medical Image Computing and Computer-Assisted Intervention 2019, 420–429.

86.

Liu

, Wang

, Zhou

et al., Semantic-aware data quality assessment for image big data, Future Generation Computer Systems 102 (2020), 53–65.

87.

Wang

, Ma

, Zhang

et al., Deep learning for smart manufacturing: methods and applications, , Journal of Manufacturing Systems 48 (2018), 144–156.

88.

Patel

, Nagar

and Raman

, Reflection symmetry aware image retargeting, , Pattern Recognition Letters 125 (2019), 179–186.

89.

Nagar

and Raman

, Detecting approximate reflection symmetry in a point set using optimization on manifold, IEEE Transactions on Signal Processing 67(6) (2019), 1582–1595.

90.

Liu

, Zhang

, Yan

et al., Scene-oriented aesthetic image assessment, International Forum on Digital TV and Wireless Multimedia Communications 2020, 359–370.

91.

Sheng

, Dong

, Chai

et al., Revisiting image aesthetic assessment via self-supervised feature learning, Proceedings of the AAAI Conference on Artificial Intelligence 34(04) (2020), 5709–5716.

92.

, Shao

, Jiang

et al., Subjective and objective quality assessment for stereoscopic image retargeting, , IEEE Transactions on Multimedia 23 (2020), 2100–2113.

93.

Zhou

, Chen

and Li

, Hierarchical visual comfort assessment for stereoscopic image retargeting, , Signal Processing: Image Communication 95 (2021), 116236.

94.

Karimi

, Samavi

, Karimi

et al., Pyramidal modeling of geometric distortions for retargeted image quality evaluation, Multimedia Tools and Applications 77(11) (2018), 13799–13820.

Image retargeting quality assessment: A survey

Abstract

Keywords

1 Introduction

3.4 Backward registration algorithm

4 Feature measurement method for image retargeting quality evaluation

4.1 Structural distortion and information loss

4.3 Aspect ratio similarity

5 IRQA dataset, evaluation index, and performance comparison

5.1 Dataset

Table 7 Cross databases performance in [46] Dataset Index CUHK (train) KRCC(mean) 0.496/0.512 RetargetMe (test) KRCC (std) 0.256/0.251 CUHK (train) PLCC 0.730/0.7577 Testing (test) SRCC 0.713/0.7383 RMSE 9.230/8.525 ∼ OR 0.047/0.0294

References

Table 7
Cross databases performance in [46]

Dataset Index

CUHK (train) KRCC(mean) 0.496/0.512

RetargetMe (test) KRCC (std) 0.256/0.251

CUHK (train) PLCC 0.730/0.7577

Testing (test) SRCC 0.713/0.7383

RMSE 9.230/8.525

∼ OR 0.047/0.0294