Mining technology hot spots in the 3D printing industry for technology strategic planning based on MRCAI

Abstract

3D printing is the important part of the emerging industry, and the accurate prediction of technology hot spots (THS) in the 3D printing industry is crucial for the strategic technology planning. The patents of the THS are always in the minority and have outlier characteristics, so the existing single and rigid models cannot accurately and robustly predict the THS. In order to make up for the shortcomings of the existing research, this study proposes a model for robust composite attraction indicator (MRCAI), which avoids the impact of outlier patents on prediction accuracy depending on not only extracting the patent attraction indicators (AIs) but also constructing the robust composite attraction indicator (CAI) according to the rough consensus of predicted results of CAIs with high generalization. Specifically, firstly, this study selects the patent AIs from the four dimensions of the attraction: technology group attraction, state attraction, enterprise attraction and inventor attraction. Secondly, in order to completely describe the attraction features of patent, AIs are directly and indirectly integrated into CAIs. Thirdly, we reduce the influence of outlier patents on prediction accuracy from two aspects: on the one hand, we initially select the CAIs with good generalization performance based on the prediction error fluctuation range. On the other hand, we build the robust CAIs by calculating the consensus of CAIs with high generalization performance based on the rough set. Fourthly, the 3D printing industry technology attention matrix is constructed to map the effective technology strategic planning based on predicted patent backward citation count by MRCAI in the short, medium and long term. Finally, the experimental results on 3D printing patent data show that MRCAI can effectively improve the efficiency in dealing with samples with outlier patents and has strong flexibility and robustness in predicting the THS in 3D printing industry.

Keywords

Technology hot spots outlier samples robust CAI 3D printing technology attention matrix

1 Introduction

Science, Nature, and other technology journals show that 3D printing, the combination of advanced materials and manufacturing, is one of the Emerging Industries [1]. In the digital production wave, 3D printing technology plays an increasingly important role, which can help to realize the next industrial revolution [2]. The 3D printing industry global output value has maintained a growth rate of 12% and the market size will reach $10.8 billion in 2020 [3]. Most countries have raised the technological development of 3D printing industry to the national strategic level. In 2012, the United States listed 3D printing as a “National Manufacturing Innovation Network” program. China established a 3D printing technology industry alliance [3]. South Africa, Japan, and other countries have introduced corresponding policies to support 3D printing industry development [4]. The technology hot spots of 3D printing industry have huge growth potential, and can reflect the development status and dynamics of industry [5]. Thus, predicting the technology hot spots of the 3D printing industry can provide significant technology strategic planning support for the government and enterprises to develop appropriate supportive incentives to achieve a better and faster innovation and development of the 3D printing industry.

The patent contains 90 95% technical information, and the public time is 1–2 years earlier than other carriers [6]. Many scholars agree that patent analysis is an effective tool for technology hot spots analysis, which can provide reasonably strategic planning support [7]. The existing technology hot spots prediction methods analyze the current status and development direction of the industry from different aspects, while most methods have some limitations. For examples, many studies only judge the technology hot spots through the count of patents [8]. However, the count of patents increases accompanied by the technology development process, which cannot predict the development potential of technologies in their early stage, so decision-making is delayed and development opportunities are missed. At the same time, these methods disregard the difference of patent potential, that is, it does not take into account the influence of patent individuals on the development trend of industry, which will lead to a large deviation between the prediction results and the actual development trajectory [9]. Besides, the patents of technology hot spots are minority and have outlier characteristics, which require prediction models either to have good generalization performance or to adjust adaptively according to different samples. However, the existing prediction methods either consider the macro trend of 3D printing technology and ignore the prediction of a single patent trend [10], or are mostly single and rigid, and it is impossible to adaptively construct appropriate robust models according to the outlier characteristics of the patents of technology hot spots, which will affect the accuracy of the prediction results.

In order to make up for the shortcomings of the existing research, this study proposes a model for Robust Composite Attraction Indicator (MRCAI), which reduces the influence of outlier patents on prediction accuracy from two aspects: one is to predict the patent of technology hot spots based on the attractiveness of patent, another is to construct the robust composite attraction indicator (CAI) according to the rough consensus of predicted results of CAIs with high generalization.

Firstly, we develop the patent attraction indicators (AIs) from four dimensions, namely technology group attraction, state attraction, enterprise attraction and inventor attraction, which can evaluate the development potential of patent in detail. Secondly, in order to completely describe the explicit and implicit attractive features of patent, the AIs are reasonably integrated directly and indirectly by various data mining techniques to construct the comprehensive the composite AIs (CAIs). Thirdly, for the sake of getting rid of the influence of outlier patents on prediction accuracy, we measure the generalization performance of the CAIs by calculating the prediction error fluctuation range and screen out the suitable robust CAIs according to the rough consensus of predicted results of CAIs with high generalization. Finally, based on the technology hot spots prediction results of MRCAI, technology attention matrix is constructed to map the effective technology strategic planning, which can finely analyze the technology hot spots of the 3D printing industry in different industry streams in the short-term, medium-term and long-term. The experimental results base on the Derwent Patent Data prove that MRCAI can accurately and timely predict the technology hot spots, and can support the government and enterprises to make the accurate technology strategic planning for 3D printing industry.

The remainder of this study is organized as follows. Section 2 briefly reviews the existing literature. Section 3 presents the technology hot spots prediction. In section 4, we introduce the MRCAI. Section 5 analyzes the results of MRCAI for prediction technology hot spots. Section 6 gives some conclusions and outlooks for future research.

2 Literature review

2.1 Industry analysis

Industry analysis is essential for drawing technology development routes and formulating strategic plans. The results of Sungjoo’s surveys showed that good industrial analysis could increase profits by 30% for enterprises [11]. The existing industry analysis research can be divided into two main aspects: spatial dimension and time dimension. The spatial dimension depicts the development points of technology in different streams of the industrial chain from the macro-level, and the time dimension describes the technology lifecycle of each technology group from the micro-level.

In the spatial dimension, the industry chain is widely adopted to analyze the technology hot spots. The industry consists of streams, the technology in the same stream has a high degree of similarity, and the technology of different streams has obvious differences. Therefore, it is necessary to subdivide the industry streams for comprehensively analyzing the current state of technological progress. To check the progress status of the new energy motor vehicle, Opitz et al. separately studied the activities of each stream based on the patent family data to analyze the technology trends [12]. Karvonen and Kässi analyzed the competitive state of different technologies for decision-makers [13]. Papachristos established competition and linkage between technologies, which proved that every stream had its own technical diversity [14]. Kim and Bae clustered the patents into different technology groups and predicted the promising technologies based on patent citation [15]. Campbell analyzed the strategic plans in different streams to predict the nominal maturity and development direction [16]. Chieh and Hsuan described the role of each technology in the industrial chain through patent citation relationships to make recommendations for the semiconductor R&D strategies [17]. Kimura et al. analyzed the development prospects of some industries by analyzing the development prospects of most enterprises [18].

In the time dimension, the development of technology includes four stages: introduction, growth, maturity, and decline. The industrial analysis in time dimension not only considers the development stage of each technology but also combines the adaptability of technology and market. Using the technology lifecycle to discuss the future development of technology has proved effective in the construction industry [19]. Karvonen et al. traced the development track through patent citation analysis and clarified the status quo of technology in different periods to understand the interaction in different stages [20]. Yeh analyzed the patents from 1996 to 2015 and drew the lifecycle of each technology through the patent quantity [21]. Trappey et al. proposed the patent content clustering method to predict the technology lifecycle and technology development space, and then evaluated market opportunities for inventors and investors [22].

In strategic decision-making or technology development activities, patent information analysis can help developers fully understand the status quo, key technologies and technology life cycle of related technology fields to predict the future technology development trend and core patent distribution of the industry. In addition, the existing research shows that technology can be predicted at least one year ahead of their publishing time [23]. However, most research in this field is limited to a single dimension of space or time, and cannot comprehensively summarize the progress of industry, which seriously hinders the accurate prediction for future industries, especially emerging industries. Therefore, this study predicts the technology hot spots from both the spatial dimension and the time dimension. We build the industrial technology attention matrix to map the effective technology strategic planning based on MRCAI, which can not only analyze the technology hot spots but also reflect the technology potentials.

2.2 Combined prediction model

Since Bates and Granger firstly proposed the combined forecasting theory system in 1969, the method has received extensive attention from many scholars [24]. The single prediction model generally contains part of the information. By combining various models with certain rules, the prediction bias caused by parameters or model errors can be greatly reduced. Therefore, combining different prediction models will improve the prediction accuracy as much as possible.

In order to improve the accuracy of decision-making, most scholars compared and combined many single models through classical statistical methods, for example, Nowotarski et al. considered the diversification characteristics of different models [25]. Fu et al. used the variance-covariance to assign the weight of every single model [26]. Yin et al. combined principal component regression method and partial least squares regression method to improve prediction accuracy [27].

Different from the classical combined methods based on statistical models, many scholars also propose several versatile combined methods. In order to predict the import and export volume in international trade, Xiao et al. proposed an artificial intelligence algorithm to combine the prediction models for accurate prediction results [28]. Liang et al. proposed a combined prediction model based on prediction error correction and regression prediction algorithm to reduce the prediction error [29]. Staron et al. used an improved cuckoo search algorithm to confirm the weight of each model, which effectively improved the accuracy [30]. Aiolfi and Timmermann combined models by model training, and they chose the top 25% of the best performing models for predicting. As a result, the performance of the model was significantly improved [31].

In summary, the combined model has higher accuracy and reliability than a single model [32], which will solve complex practical problems. However, there are still various problems to be solved. First of all, the current combination prediction methods focus on linear combination, and the research on nonlinear combination is relatively less. In addition, the methods of variable weight, weight range and optimal problem are still the focus of future research. Finally, the existing combined prediction methods are fixed and changeless, and as the data characteristics change, the prediction accuracy will be impacted. To fill the research gap, this study proposes the MRCAI, which can adaptively select the suitable robust CAIs with small prediction error fluctuation range and high consensus with other models in different scenarios, and accordingly, ensure the prediction accuracy of outlier patents of technology hot spot.

3 Technology hot spots prediction

3.1 The framework of technology hot spots prediction of the 3D printing industry

In order to provide support for the government and enterprises to make the technology strategic planning to realize a healthy development of the 3D printing industry, this study proposes a framework of the reliable technology hot spots prediction of the 3D printing industry, which can predict the technology hot spots in different technology groups and different industry streams. The backward citation count is adopted to depict the patent potential, namely the patent with the high backward citation count is regarded as the patent of the technology hot spots. As shown in Fig. 1, the framework of technology hotspot prediction consists of four parts: patent AIs, CAI set, CAI selection, and hot spots forecasting. The patent AIs include four dimensions: technology group attraction, state attraction, enterprise attraction and inventor attraction, which can evaluate the development potential of the patent in detail. In the CAI set part, we build various CAIs by reasonably directly and indirectly combining AIs through the data mining techniques. As a result, the explicit and implicit attraction features of patent are completely depicted. In CAI selection portion, for overcoming the influence of outlier patents on prediction accuracy, we measure the generalization performance of the CAI by calculating the prediction error fluctuation range and select the suitable robust CAI by the rough consensus of predicted results of models with high generalization. In the hot spots forecasting part, we analyze the technology lifecycle of different technology groups based on the prediction results of robust CAIs and develop the industrial technology attention matrix for mapping the effective technology strategic planning by making a refined analysis of the hot spots in the upstream, midstream and downstream of the 3D printing industry.

Fig. 1

Framework of technology hot spots prediction of the 3D printing industry.

3.2 Industrial technology attention matrix

In order to map the effective technology strategic planning, this study proposes an industrial technology attention matrix, which analyzes the hot spots in different streams of the 3D printing industry. As shown in Fig. 2, this matrix is divided into eight categories according to the technology potential and technology lifecycle. The technology potential is measured by the short-term predicted backward citation count (y_short). When y_short is larger than ∂, the technology is classified into the hotspot category, otherwise, it is classified into the non-hotspot. And the technology lifecycle has four stages: introduction, growth, maturity, and decline, which are divided by the growth rate (GR) of backward citation count, and the GR (GR₁, GR₂) are defined as follows:

Fig. 2

Industrial technology attention matrix.

$G R_{1} = \frac{y_{medium} - y_{short}}{y_{short}} \times 100 %$ (1) $G R_{2} = \frac{y_{long} - y_{medium}}{y_{medium}} \times 100 %$ (2)

Where, y_medium is the medium-term predicted backward citation count, y_long is the long-term predicted backward citation count, GR₁ is the growth rate of y_medium, while GR₂ is the growth rate of y_long. GR is defined as fast when it is no less than δ, otherwise, is defined as slow.

Next, we introduce each stage of technology lifecycle in detail.

a) Introduction stage

In the introduction stage, GR₁ and GR₂ are both slow, and GR₁ is greater than GR₂. As the market is uncertain and only a few enterprises conduct technical research and market development, the technology is developing slowly.

b) Growth stage

In the growth stage, GR₁ and GR₂ are both fast, and GR₁ is greater than GR₂. With the continuous development of technology, the market is being expended. Because the number of enterprises is increasing and technology distribution is expanding, the patent applications and applicants are growing at a rapid rate.

c) Maturity stage

In the maturity stage, GR₁ and GR₂ are both fast, and GR₁ is less than GR₂. Because technology is maturing and the market is becoming limited and only less enterprises will continue to engage in technical research. In this stage the number oring enterprises is small and the speed of patent application’s growth is slowing down.

d) Decline stage

In the decline stage, GR₁ and GR₂ are both slow, and GR₁ is less than GR₂. When technology is aging, more and more enterprises withdraw from the market due to diminishing returns. Since the technology in the related field is almost no longer doping, the patent applications and applicants have negative growth rate.

Based on the technology attention matrix in Fig. 2, 3D printing technologies can be further clustered into four categories:

a) Potential technology

The technology in this category is in the introduction stage, which has large innovation space and few competitors. Therefore, the enterprises can enter the technology field as soon as possible to carry out technology research and development, and to become the industry pioneer and obtain the higher return.

b) Gold technology

The technology in this category is in the golden age of development. With the upgrading of technology, the demand for the market is increasing. Therefore, the enterprises can consider this technology as a backbone technology for quickly achieving the high returns.

c) Stable technology

The technology of this category is in the stage of blooming, with diversified and differentiated technologies and numerous competitors. However, the slowdown in demand growth causes the market is in a saturated stage with low returns, the technological innovation is also in a bottleneck, and the enterprises need more resources to improve the technology. So, it is not suitable to be a strategic development direction for small enterprises.

d) Loss technology

The technology in this category lacks market demand and R&D space, so it should be avoided when choosing the strategic technology.

3.3 Patent AIs

Backward citation count analysis is the main method of technology hot spots prediction, because the patent with the higher influence will have the greater backward citation [33]. Therefore, this study uses the count of backward citation to indicate the degree of technology hot spots [34], that is, the output (y) of MRCAI.

In order to overcome the influence of the outlier hot spots patents on the model accuracy, this study proposes the AIs to describe the potential of patents as the hot spots. Generally, the more attractive the patent is to other applicants, the more likely it is to be cited, and subsequently the patent technology is more likely to be the hot spot. In order to completely describe the attraction features of patent, this study proposes the AIs from four dimensions: technology group attraction, state attraction, enterprise attraction and inventor attraction. Table 1 shows the AIs for technology hot spots prediction.

Table 1
Patent AIs

Category Indicators Denotation Definition

Technology group attraction IPC Citation IPC_CA The citation count of all IPCs within one patent

IPC Count IPC_CO The number of IPCs within one patent

Top IPC IPC_TOP The number of top IPCs within one patent

Attraction intensity LC The value of technology maturity

State attraction State Citation S_CA The citation count of all states within one patent

State Count S_CO The number of states within one patent

Enterprise attraction Enterprise Citation E_CA The citation count of all patentees (enterprise) within one patent

Enterprise Count E_CO The number of patentees (enterprise) within one patent

Inventor attraction Inventor Citation IN_CA The citation count of all inventors within one patent

Inventor Count IN_CO The number of inventors within one patent

Top Inventor IN_TOP The number of top inventors within one patent

Category	Indicators	Denotation	Definition
Technology group attraction	IPC Citation	IPC_CA	The citation count of all IPCs within one patent
	IPC Count	IPC_CO	The number of IPCs within one patent
	Top IPC	IPC_TOP	The number of top IPCs within one patent
	Attraction intensity	LC	The value of technology maturity
State attraction	State Citation	S_CA	The citation count of all states within one patent
	State Count	S_CO	The number of states within one patent
Enterprise attraction	Enterprise Citation	E_CA	The citation count of all patentees (enterprise) within one patent
	Enterprise Count	E_CO	The number of patentees (enterprise) within one patent
Inventor attraction	Inventor Citation	IN_CA	The citation count of all inventors within one patent
	Inventor Count	IN_CO	The number of inventors within one patent
	Top Inventor	IN_TOP	The number of top inventors within one patent

1) Technology group attraction

Technology group attraction is composed of scope of attraction and attraction lifecycle.

a) Scope of attraction

International Patent Classification (IPC) represents the technology group of a patent and the theoretical knowledge basis. When IPC count of a patent is larger, the scope of the technology involved in the patent is wider, and the scope of its attraction is correspondingly expanded. Patent citation has Matthew effect, that is, the more times a patent is cited, the more attractive it will be to subsequent patent inventors. Therefore, this study selects the top 100 IPC as the TOP IPC. When the patent involves the Top IPC technology, its attraction will be improved. The patents with more Top IPC are more possible to become the technology hot spots.

b) Attraction intensity

The attraction of patent varies with the different stages of technology life. Xiao used the adoption lifecycle to analyze the changes in the attractiveness of patents, which was divided into five stages: innovators, early adopters, early majority, late majority, and laggards [35]. The results showed that when patents were in the early stages, their attractiveness was higher. In this study, the attraction intensity of technology group is determined by its patent application count in three stages, namely 1995–2013 (innovators), 2014–2015 (early adopters), 2016–2018 (early majority). Table 1 shows the AIs for technology hot spots prediction.

2) State attraction

Harhoff et al. confirmed that inventors preferred to cite patents from the authoritative countries, that is, the country with more cited patents had more attraction [36]. In addition, Lanjouw and Schankerman concluded that the market competitiveness and value were greater if the number of patent application states was higher [37], and correspondingly, the patent was more attractive. In this study, the state attraction is described by state citation and state count.

3) Enterprise attraction

The number of citations of the enterprise which owns the patent directly reflects its attraction to other patent applicants. In addition, Du et al. proved that the number of enterprises which owned the patent was positively related to the patent attraction [38]. Therefore, in this study, we use the citation count of patentees (enterprises) and the patentees (enterprises) quantity to describe the attractions of patent application enterprises.

4) Inventors attraction

Agrawal and Henderson proved that the authoritative inventors had the high attraction to other inventors [39]. Obviously, the attractiveness of inventors is significantly related to their patent citations. In this study, in addition to the number of inventors and their citations, we also consider whether they are top inventors, that is, the 100 inventors with the highest citations.

4 The MRCAI

In order to overcome the impact of outlier hot spots patents on prediction accuracy, this study proposes the MRCAI, which can adaptively construct the robust CAIs based on different data characteristics of outlier samples. As shown in Fig. 3, MRCAI consists of three parts: patent AIs, CAIs set, and robust CAI selection. First of all, in the part of patent AIs, we construct the patent AIs from four dimensions, namely technology group attraction, state attraction, enterprise attraction and inventor attraction. Next, in the part of CAIs set, the single normalized AIs are combined into the CAIs by various data mining methods, such as support vector machine (SVM), regression model (RM), radial basis function neural network (RBFNN), general regression neural network (GRNN) and back propagation neural network (BPNN). Finally, in the part of robust CAI selection, the robust CAI is chosen through the prediction error fluctuation range and the rough consensus of each indicator prediction result. When the CAI has the smaller prediction error and is more similar to other models, it is selected as the robust CAI since it can filter the interference of the noise data and overcome the influence of outlier data on prediction accuracy. In this way, the shortcoming of the fixed and rigid combination of indicators, namely low prediction accuracy due to the dynamic complexity of technology development and the diversity of samples characteristics, can be avoid and the excellent prediction results are acquired.

Fig. 3

The framework of MRCAI.

4.1 Verification of patent AIs based on grey relational analysis

The degree of association between the indicator and output is the basis for obtaining the accurate prediction results. Grey relational analysis is a method to measure the association between two indicators, which has no limit on the quantity and the distribution pattern of samples. In general, most backward citation count of the patent is small and even equal to 0. Therefore, this study uses grey relational analysis to measure the association between the patent AIs and the backward citation count (y), which can verify the validity of patent AIs on the samples with a small number of outlier technology hotspot patents. In grey relational analysis method, if the change tendency of the two indicators is more similar, the grey correlation coefficient is larger, otherwise, it is smaller. The grey correlation between AI_i (γ_i) and yis given as follows: $γ_{i} = \frac{1}{N} \sum_{k = 1}^{N} \frac{Δ (\min) + π Δ (\max)}{Δ_{i} (k) + π Δ (\max)}$ (3)

Where π is resolution coefficient and generally is 0.5, N is the number of patent samples, Δ (min) is the minimum difference between the patent AIs and y, similarly, Δ (max) is the maximum difference of that, and Δ_i (k) is the absolute difference between the ith patent AI and y in sample k. When the absolute value of γ_i is larger, the association between the patent AI_i and backward citation count is greater, and the influence of indicators is more significant.

4.2 CAIs set of technology hot spot

Owing to the dynamic complexity of technological evolution, it is difficult to fully describe the technology development trajectories with a single indicator. For these reasons, the CAIs are adopted and they are built in two ways to depict the explicit and implicit attraction features of patent: the directly combing of single AIs using the RMs and indirectly integrating the single AIs based on SVM, BPNN, GRNN and RBFNN. Table 2 shows the method to construct CAIs.

Table 2
CAIs formulation

Combination methods Models Kernel functions Parameters

Direct combinations Regression1(REG1) y = a + bx y is CAI, x is AIS

Regression2(REG2) y = ax^b

Regression3(REG3) $y = \frac{1}{{ax}^{b}}$

Regression4(REG4) y = ae^bx

Regression5(REG5) y = a + lnx

Regression6(REG6) $y = \frac{a}{x} + b$

Regression7(REG7) y = a + be^-x

Regression8(REG8) y = a + bx²

Regression9(REG9) y = a + bx² + cx³

SVM K (χ, z) = (γ < χz > + c) ^d

Indirect combinations BPNN y = f (∑w_i × x - θ_k) f () is nonlinear action function, W_i is the weight of each input factor; θ is the neural unit threshold.

GRNN $y = E (y / X) = \frac{\int_{- \infty}^{+ \infty} yf (X, y) dy}{\int_{- \infty}^{+ \infty} f (X, y) dy}$ f (X, y) dy is an estimated density function

RBF $y = \emptyset (| | x - μ_{i} | |) = exp (- \frac{{| | x - u_{i} | |}^{2}}{2 σ_{t}^{2}})$ μ_i is the center point; ∅ is the distance between x and μ_i; $σ_{t}^{2}$ is the width parameter of the function.

Combination methods	Models	Kernel functions	Parameters
Direct combinations	Regression1(REG1)	y = a + bx	y is CAI, x is AIS
	Regression2(REG2)	y = ax^b
	Regression3(REG3)	$y = \frac{1}{{ax}^{b}}$
	Regression4(REG4)	y = ae^bx
	Regression5(REG5)	y = a + lnx
	Regression6(REG6)	$y = \frac{a}{x} + b$
	Regression7(REG7)	y = a + be^-x
	Regression8(REG8)	y = a + bx²
	Regression9(REG9)	y = a + bx² + cx³
	SVM	K (χ, z) = (γ < χz > + c) ^d
Indirect combinations	BPNN	y = f (∑w_i × x - θ_k)	f () is nonlinear action function, W_i is the weight of each input factor; θ is the neural unit threshold.
	GRNN	$y = E (y / X) = \frac{\int_{- \infty}^{+ \infty} yf (X, y) dy}{\int_{- \infty}^{+ \infty} f (X, y) dy}$	f (X, y) dy is an estimated density function
	RBF	$y = \emptyset (\| \| x - μ_{i} \| \|) = exp (- \frac{{\| \| x - u_{i} \| \|}^{2}}{2 σ_{t}^{2}})$	μ_i is the center point; ∅ is the distance between x and μ_i; $σ_{t}^{2}$ is the width parameter of the function.

4.3 Selection the robust CAI

Since the samples of technology hot spots patent are mostly outliers and abnormal, the accurate predicted results cannot be obtained with a single CAI. Therefore, in order to overcome the interference of outlier samples on prediction accuracy, this study proposes the MRCAI, which can improve the prediction accuracy by adaptively selecting the robust CAI with small prediction error fluctuation range and high rough consensus with other generalized models in different scenarios.

4.3.1 Estimation of the prediction error fluctuation range

In order to comprehensively evaluate the performance of the above-mentioned CAIs in dealing with outlier samples, this study calculates the upper and lower bound of the prediction error of each CAI, that is, estimates the prediction error fluctuation range (ΔY), as shown in the following [40]:

$Δ Y = Y^{+} - Y^{-}$ (4)

$Y^{+} = {\hat{y}}^{0} + t_{\frac{α}{2}} se (e^{0})$ (5)

$Y^{-} = {\hat{y}}^{0} - t_{\frac{α}{2}} se (e^{0})$ (6)

$e^{0} = y^{0} - {\hat{y}}^{0}$ (7)

$Var [e^{0} | X, x^{0}] = σ^{2} + x^{0^{'}} [σ^{2} {(X^{'} X)}^{- 1}] x^{0}$ (8)

$y^{0} = x^{0^{'}} β$ (9)

Where ΔY is the fluctuation range of prediction error fluctuation range, and smaller ΔY indicates higher prediction accuracy. Y⁺ is the upper bound of prediction error, Y^- is the lower bound of prediction error. ${\hat{y}}^{0}$ is the real value of prediction error, y⁰ is predicted value of the prediction error, α is confidence interval. β is the estimated coefficient of the linear regression model, x⁰ represents independent variables in test samples, X indicates independent variables in training samples.

4.3.2 Generalization evaluation of CAIs

Since the citation count of technology hot spots patents deviates so much from the others, it is necessary to evaluate the generalization performance of CAIs and select the generalized CAIs to identify the patents of technology hot spots. In order to verify the generalization performance of each CAI, this study divides the samples into three groups: training samples, generalized samples, and test samples. The training samples are used to estimate the parameters of CAIs in Table 2, which involve 80% of the patent data. 10% of the data are adopted as generalized samples to select the robust CAIs, and the remaining 10% are employed to test the performance of CAIs.

When a CAI has superior reliability and accuracy in different scenarios, it has the excellent generalization performance. In order to have a comprehensive evaluation of the generalization performance of CAI, we consider not only its prediction error fluctuation range in training samples but also that in generalized samples. As a result, the generalization coefficient MP_i is developed, as shown in Eq. (10). ${MP}_{i} = \frac{1}{1 + e^{- a Δ Y_{i}^{T}}} + \frac{1}{1 + e^{- b Δ Y_{i}^{G}}}$ (10)

Where $Δ Y_{i}^{T}$ is the fluctuation range of prediction error of CAI i in training samples, and $Δ Y_{i}^{G}$ is the prediction error fluctuation range in generalized samples. a and bare constant and range from 0 to 1.

Obviously, the CAI with a smaller MP_i acquires the better generalization performance. For the sake of improving the prediction accuracy, we select J CAIs with MP_i less than λ.

4.3.3 Selection robust CAI based on consensus

The outlier samples usually have noisy data, which causes the predicted backward citation count significantly deviates from the real value. In order to filter out the inference of noisy data, this study selects the robust CAI based on the consensus of the predicted results of CAIs with superior generation performance. The rough set can effectively eliminate uncertainty and avoid inaccuracy of samples with lots of noisy data [41]. Therefore, rough consensus coefficient ρ_i,k is defined, as shown in Eq. (11), and the robust CAI is selected with the largest rough consensus coefficient. $ρ_{i, k} = \frac{\bar{γ_{i, k}} L_{i, k}}{\underline{γ_{i . k}}}$ (11)

$\bar{γ_{ik}} = | {Y_{i, k} \subset Y_{j, k} | 1 ⩽ j ⩽ J, j \neq i} |$ (12)

$\underline{γ_{ik}} = | {Y_{i, k} \cap Y_{j, k} \neq \emptyset | 1 ⩽ j ⩽ J, j \neq i} |$ (13)

$L_{i, k} = \frac{C_{J - 1}^{I - 1}}{2^{J - 1}}$ (14)

Where $\bar{γ_{ik}}$ is the number of CAIs, whose fluctuation range of prediction error is bigger than the range of CAI i in sample k, similarly $\underline{γ_{ik}}$ is the number of CAIs, whose prediction error fluctuation range intersects with the range of CAI i in sample k. J is the count of CAIs, whose MP is less than λ. L_i,k is a measure of consensus of CAI. We rank the CAIs in ascending order of its predicted output and I is the ranking index of CAI i. Generally, the medium ranking values result in the large consensus that the CAI shares with others. In Combination Number Formula, the number of possible combinations of I-1 objects from a set of J-1 reaches the largest when I approaches the median of J. So, we use Combination Number Formula $C_{J - 1}^{I - 1}$ as the numerator of L_i,k, and 2^J-1 as the denominator for limiting the value of L_i,k between 0 to 1.

5 Experiment process and results

5.1 Data acquisition

The Derwent Innovations Index is the most comprehensive database of global patent and has more than 14 million basic patents and more than 20 million practical patents. So, we chose it as the experimental data source and the search strategy was “theme=(3D printing)” and “time span = 1963–2018”, and then obtained 7015 related patent data. At last, we selected 1500 valid patents based on the patent AIs.

The results of grey correlation analysis are shown in Fig. 4, where γ_i of the vast majority of AIs is greater than 0.7, even the smallest γ_i is greater than 0.3, and those mean that the proposed patent AIs have a great association with backward citation count, that is, the patent AIs are highly effective.

Fig. 4

Correlation degree between AIs and backward citation count based on grey relational analysis.

5.2 Experiment results

5.2.1 Performance comparison

In this section the performance of robust CAI is compared with other typical prediction models in Table 2. In each experiment, for the benchmark models, we randomly selected 90% samples as the training samples, and the rest samples as the test samples. For MRCAI, we divided the samples into three parts, as shown in Section 4.3.2, namely the training samples, generalized samples, and test samples. All models were implemented in MATLAB with default settings.

The average MSE of 20 random experiments in the short term (within one year), medium term (within three years), and long term (within five years) for various models is shown in Table 3.

Table 3
The MSE of MRCAI and single model

Model MSE

Robust CAI 0.22

BPNN 0.31

GRNN 0.29

RBF1 0.53

RBF2 0.61

REG1 0.35

REG2 0.45

REG3 0.43

REG4 0.43

REG5 1.21

REG6 0.32

REG7 0.32

REG8 1.23

REG9 0.34

SVM 0.44

Model	MSE
Robust CAI	0.22
BPNN	0.31
GRNN	0.29
RBF1	0.53
RBF2	0.61
REG1	0.35
REG2	0.45
REG3	0.43
REG4	0.43
REG5	1.21
REG6	0.32
REG7	0.32
REG8	1.23
REG9	0.34
SVM	0.44

In Table 3, the MSE of the robust CAI is smaller than other models, that is, robust CAI has the highest prediction accuracy and can precisely predict the technology hot spots of the 3D printing industry.

To further evaluate the performance of MRCAI, the numerous common combined models in the current literature are adopted to integrate the predicted outputs of models, such as arithmetic averaging (AA) method [42], additive weighting (AW) method [43], variance reciprocal (VR) method [44], and the minimum error weight coefficient (MEWC) method [45], as shown in Table 4.

Table 4

The typical combination models

Combined prediction model	Formula	Explanation
AA	$Y = \sum_{j = 1}^{J} ω_{j} y_{j}$	$w_{j} = \frac{1}{J}, J$ is the number of prediction models, y_j is the output of model j, the same below.
AW	$Y = \sum_{j = 1}^{J} ω_{j} y_{j}$	$w_{j} = \frac{R_{j}}{\sum_{j = 1}^{J} j} = \frac{2 R_{j}}{J (J + 1)}$ , R_j is the rank index of prediction error of model j in descending order.
VR	$Y = \sum_{j = 1}^{J} ω_{j} y_{j}$	$w_{j} = \frac{D_{j}^{- 1}}{\sum_{j = 1}^{J} D_{j}^{- 1}}, D_{j}$ is the sum of squared errors of model j in training samples.
MEWC	$Y = \sum_{j = 1}^{J} ω_{j} y_{j}$	$min \sum_{i = 1}^{n} e_{i}^{2} = \sum_{i = 1}^{n} {[\sum_{j = 1}^{J} w_{j} y_{j} (i) - {\overset{´}{y}}_{i}]}^{2} s . t . {\begin{matrix} \sum_{j = 1}^{J} w_{j = 1} \\ w_{j} > 0 \end{matrix}$ , e_i is the sum squared residuals of simple i, w_j is the weight of model j, y_j (i) is the predicted value of model j, ${\overset{´}{y}}_{i}$ is the real output of sample i.

The experimental results of these combination models are shown in Table 5.

Table 5

The MSE of combined models

	Robust CAI	AA	AW	VR	MEWC
MSE	0.22	0.35	0.41	0.30	0.25

In Table 5, the MSE of robust CAI is 0.22, which is smaller than other combined models. It shows that compared with most of the existing rigid models, the robust CAI with adaptive selection of appropriate CAI in different scenarios is helpful to deal with outlier samples and improve the prediction accuracy.

Besides MSE, other quantitative index, such as Precision (P), Recall (R) and F-score (F) value, are adopted to further evaluate the performance of Robust CAI, as shown in the following: $P = \frac{TP}{TP + FP}$ (15)

$R = \frac{TP}{TP + FN}$ (16)

$F = \frac{2 \times P \times R}{P + R}$ (17)

Where TP means the number of outcomes where the model correctly predicts the technical hot spot patents, FP indicates the number of outcomes where the model incorrectly forecasts the technical hot spot patents, and FN denotes the number of outcomes where the model incorrectly forecasts the non-technical hot spot patents.

The experimental results are given in Table 6. It can be seen from Table 6 that the precision, recall and F-score value of the Robust CAI model are both higher than other models, which confirms the superior performance of Robust CAI for solving complex prediction problems.

Table 6

The Precision, Recall and F-score value of various models

Model	P	R	F
Robust CAI	0.94	0.96	0.95
BPNN	0.92	0.95	0.93
GRNN	0.92	0.95	0.94
RBF1	0.87	0.91	0.89
RBF2	0.85	0.90	0.87
REG1	0.91	0.94	0.93
REG2	0.89	0.93	0.90
REG3	0.89	0.93	0.91
REG4	0.89	0.93	0.91
REG5	0.71	0.80	0.75
REG6	0.92	0.95	0.93
REG7	0.92	0.95	0.93
REG8	0.71	0.80	0.75
REG9	0.91	0.94	0.93
SVM	0.89	0.93	0.91

Next, in order to verify the validity of calculating CAI consensus based on the prediction error fluctuation range, we design another robust CAI, namely RCAI1, which computes the CAI consensus based on prediction value fluctuation range. Table 7 gives the maximum, average and minimum MSE of robust CAI and RCAI1 in 20 random experiments.

Table 7

The MSE of robust CAI and RCAI1

	Average	Maximum	Minimum
Robust CAI	0.22	0.35	0.18
RCAI1	0.26	0.46	0.24

It can be seen from Table 7 that the maximum, average and minimum MSE of robust CAI are smaller than those of RCAI1, which indicates that the method of computing model consensus based on prediction error fluctuation range is helpful and necessary to improve the model performance when dealing with outlier samples.

5.2.2 Parameter analysis of MRCAI

In this section, we analyze the influence of different values of a, b in formula (10) and λ on the prediction performance of robust CAIs. When analyzing the influence of a and b, we set λ= 1.5, and range a from 0.1 to 1 and b from 0.1 to 1, respectively. When analyzing the influence of λ, we set a = 0.8, b = 0.6 and range λ from 1 to 2. Figure 5 shows the influence of a and b on the performance of robust CAI. Figure 6 demonstrates the influence of λ.

Fig. 5

The impact of a and b on the performance of robust CAI.

Fig. 6

The impact of λ on the performance of robust CAI.

In Fig. 5, when a and b are very small, that is to say, weakening the adaptive selection effect of training samples and generalization samples on CAI, the model will not get the best prediction results. On the contrary, when a and b are very large, although the influence of training samples and generalization samples on the adaptive selection of CAI is strengthened, more generalization models will be screened out, which will affect the quality of robust CAI extracted by consensus, so the model will not obtain good prediction outcomes. However, the median value of a and b can ensure the excellent performance of robust CAI. For instance, when a = 0.7, b = 0.6, the MSE reaches the minimum value. On the other hand, there is little difference in MSE in various values of a and b, basically between 0.2 and 0.4, which can prove that CAI is robust enough to deal with all kinds of outlier samples in technology hot spots prediction, and the model is relatively reliable.

It can be seen from Fig. 6 that the robust CAI does not achieve good performance when λ is larger or smaller. The robust CAI achieves the best performance when λ is at the middle value, namely 1.5. And those imply that selecting the slightly high generalization performance models to calculate the rough consensus of CAI remains fundamental to improve the performance of robust CAI, but calculating the rough consensus of all sort of CAIs to select the suitable CAI is not expected to obtain the high precision.

5.2.3 Hot spots analysis of the 3D printing industry

Using the robust CAI, the short-term, medium-term and long-term backward citation of the patents is predicted, and then the technology attention matrix is developed to support the establishment of significant technology strategic planning to realize a healthy development of the 3D printing industry. According to the backward of citation and growth rate defined according to Eq. (1) and Eq. (2), the attention matrix is constructed, as shown in Fig. 2, where classification coefficient ∂ is 1, and the growth rate coefficient δ is 10%.

Fig. 7

Technology attention matrix of the 3D printing.

As shown in Fig. 7, the technologies in upstream are clustered maturity and are classified as stable technology, while technologies in midstream and downstream are almost distributed in all stages and can be sorted into various categories. Based on the industrial technology attention matrix, some suggestions for the 3D printing industry technology strategic planning are proposed, as shown in the following:

Upstream technologies: These technologies include metal, materials, and high compounds technologies, which are at a mature stage. Any type of innovation of these technologies will cost lots of research and development resources, but their market is in a saturated stage with low returns. So, the technologies in upstream are not suitable to be selected as a strategic technology to be promoted.

Midstream technologies: Among numerous midstream technologies, computation technology is in the decline stage, therefore the depth of innovation is small. While surface processing technology and radio communication technology are in the growth stage, which represents the developing trend of the 3D printing technologies. And government and enterprises should attach great importance to and energetically support these technologies. Moreover, comparing with the radio communication technologies, surface processing has fewer R&D companies and less competition. Developing the monopolistic surface processing technology will result in a leading technology advantage and acquire competitiveness and prosperous returns.

Downstream technologies: Medicine is one of the main application areas of downstream 3D printing technologies, but the technological innovation in this area is entering a bottleneck period and lack of development space. However, clothing accessories and architecture equipment technologies have the large market demand. In addition, the food and electrical technologies in 3D printing industry are in the introduction stage with a small count of R&D companies. Therefore, innovation in these areas can bring about many benefits of enterprise, such as seizing the competitive advantage, avoiding duplication of research efforts, internalizing spillovers of technology.

6 Conclusions

The 3D printing industry is a major part of the emerging industry, which has a positive impact on social and economic development. Accurately predicting the 3D printing industry technology hot spots can provide significant technology strategic planning support for the government and enterprises. However, the existing technology prediction research mainly analyzes the current status of technology based on the patent count, which ignores the potential of each technology. In addition, most patent data with high backward citation count is outlier samples that are difficult to predict accurately, so this study proposes the MRCAI, predicts the technology hot spots based on the idea of patent attraction, and subsequently the technology attention matrix is developed to identify the 3D printing technologies with greatest potentiality and map the effective technology strategic planning. In order to avoid the detrimental effect of outlier characteristics of the backward citation count of technology hot spots patent on the prediction accuracy, the AIs based on the technology group, state, enterprise and inventor attraction are proposed to depict the attractiveness of patents. Moreover, the robust CAIs are adaptively built based on the characteristics of outlier samples in various scenarios, which shares the largest consensus with other optimal CAIs.

Comparing with the existing research, the MRCAI has the following advantages: 1) Different from the previous research which depends on the patent count to analyze the technology hot spots, we predict the hot spots based on the predicted backward citation count, which helps government and enterprises accurately identify the technology hot spots in advance, so as to seize the opportunities and avoid delaying in technology strategic planning. 2) Unlike the most existing studies which judge the technology potential through the time series of the patent count, such as S-Curve, the theoretical basis of this study is the attractiveness of patent, that is, this study predicts the development potential of the technology based on its patent AIs, which can improve the efficiency and accuracy of processing complex, cumbersome and outlier patent data. 3) Distinct from the existing research which predicts technology hot spots through the single fixed model, we use MRCAI to adaptively construct the robust CAI, which shares the large rough consensus with other generalized models in different scenarios and can increase the generalization and stability for handling out outlier samples. 4) Last but not least, we not only predict the backward citation count in different development time period, but also propose a technology attention matrix for subdividing the technology hot spots in different industry streams to identify the potential technology of the 3D printing industry, which can offer the government and enterprises practical support for technology strategic planning and making accurate decisions in advance.

Besides technology strategic planning, the proposed model can be applied in many other fields. For example, by anticipating hot patents in advance based on our model, enterprises can successfully formulate patent competition strategies. In addition, because there are some time lags between patent publication and market launch, our model can provide correct decision support for market investment by identifying hot patents in advance.

Technology development is not only related to technology but also affected by various factors such as its competitors and user demand. This study predicts the 3D printing industry development only from the technical perspective, and there is a lack of market information support, so the prediction results have certain limitations. In the follow-up study, we can comprehensively predict the technology hot spots of the 3D printing industry by integrating market factors.

Footnotes

Acknowledgments

This work was supported by the Chinese National Natural Science Foundation (No. 71871135).

References

Research Institute (Iri), Research-Technology Management (Rtm), R&D Magazine, global R&D funding forecast, (2016). http://www.iriweb.org

Warnecke

, Gevorkjan

G.D.

and Teuteberg

, Amalgamation of 3D Printing Technology and the Digitalized Industry – Development and Evaluation of an Open Innovation Business Process Model, International Conference on Business Information Systems, Springer, Cham, (2018), 148–159.

Gebler

, Uiterkamp

A.J.S.

and Visser

, A global sustainability perspective on 3D printing technologies, }, Energy Policy 74 (2014), 158–167.

Wimpenny

D.I.

, Pandey

P.M.

and Kumar

L.J.

, Advances in 3D Printing & Additive Manufacturing Technologies, Springer Publishing Company, Incorporated, 2016.

Roisin

, The Significance and Path of Accelerating the Development of Strategic Emerging Industries, , Knowledge Economy 5 (2016), 26–26.

Hall

B.H.

, Jaffe

and Trajtenberg

, Market value and patent citations, RAND Journal of Economics 36 (1) (2005), 16–38.

Angelou

, Maragakis

and Argyrakis

, A structural analysis of the patent citation network by the k-shell decomposition method, , Physica A: Statistical Mechanics and its Applications 521 (2019), 476–483.

Evangelista

, Ardito

, Boccaccio

, Fiorentino

, Petruzzelli

A.M.

and Uva

A.E.

, Unveiling the technological trends of augmented reality: A patent analysis, , Computers in Industry 118 (2020), 1–15.

Daim

T.U.

, Rueda

, Martin

and Gerdsri

, Forecasting emerging technologies: Use of bibliometrics and patent analysis, Technological Forecasting and Social Change 73 (8) (2006), 981–1012.

10.

Wei

, Lin

C.R.

, Li

C.Y.

, Kong

L.K.

and Yang

Z.L.

, Tracing the Evolution of 3-D Printing Technology in China Using LDA-Based Patent Abstract Mining, IEEE Transactions on Engineering Management, (2020), 1–11.

11.

Cho

and Lee

, How firms can get ideas from users for sustainable business innovation, Sustainability 7 (12) (2015), 16039–16059.

12.

Opitz

, Badami

, Shen

, Vignarooban

and Kannan

A.M.

, Can Li-Ion batteries be the panacea for automotive applications? , Renewable and Sustainable Energy Reviews 68 (2017), 685–692.

13.

Karvonen

and Kässi

, Patent analysis for analysing technological convergence, Foresight 13 (5) (2011), 34–50.

14.

Papachristos

, Diversity in technology competition: The link between platforms and sociotechnical transitions, , Renewable and Sustainable Energy Reviews 73 (2017), 291–306.

15.

Kim

and Bae

, A novel approach to forecast promising technology through patent analysis, Technological Forecasting and Social Change 117 (17) (2017), 228–237.

16.

Campbell

R.S.

, Patent trends as a technological forecasting tool, World Patent Information 5 (3) (1983), 137–143.

17.

Chieh

and Hsuan

, Knowledge spillovers among semiconductor companies with different technology positions and with different roles in the industry chain, International Conference on Scientometrics and Informetrics, (2017), 741–752.

18.

Kimura

, Goto

, Shintani

, Ishizaka

, Arima

T.H.

and Tokura

, Magnetic control of ferroelectric polarization, , Nature 426 (2003), 55–58.

19.

Aslani

, Bakhtiar

and Akbarzadeh

M.H.

, Energy-efficiency technologies in the building envelope: Life cycle and adaptation assessment, , Journal of Building Engineering 21 (2019), 55–63.

20.

Karvonen

, Kapoor

, Uusitalo

and Ojanen

, Technology competition in the internal combustion engine waste heat recovery: a patent landscape analysis, , Journal of Cleaner Production 112 (2016), 3735–3743.

21.

Yeh

Y.L.

, Using the Patent Analysis to Examine the Future Development Trend of the Life Cycle of Taiwan’s Fitness Equipment and Bicycle Industries, Global Academy of Training and Research (GATR) Enterprise 6 (1) (2018), 37–43.

22.

Trappey

C.V.

, Wu

H.Y.

, Taghaboni-Dutta

and Trappey

A.J.

, Using patent data for technology forecasting: China RFID patent analysis, Advanced Engineering Informatics 25 (1) (2011), 53–64.

23.

Kyebambe

M.N.

, Cheng

, Huang

, He

and Zhang

, Forecasting emerging technologies: A supervised learning approach through patent analysis, , Technological Forecasting and Social Change 125 (2017), 236–244.

24.

Bates

J.M.

and Granger

C.W.J.

, The combination of forecasts, Journal of the Operational Research Society 20 (4) (1969), 451–468.

25.

Nowotarski

, Liu

, Weron

and Hong

, Improving short term load forecast accuracy via combining sister forecasts, , Energy 98 (2016), 40–49.

26.

, Lai

, Shan

and Geng

, A Spatial Forecasting Method for Photovoltaic Power Generation Combined of Improved Similar Historical Days and Dynamic Weights Allocation, In 2018 IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia), IEEE, (2018), 1195–1198.

27.

Yin

, Liu

and Hou

, A multivariate statistical combination forecasting method for product quality evaluation, , Information Sciences 355 (2016), 229–236.

28.

Xiao

, Wang

, Dong

and Wu

, Combined forecasting models for wind energy forecasting: A case study in China, , Renewable and Sustainable Energy Reviews 44 (2015), 271–288.

29.

Liang

, Liang

, Wang

, Dong

and Miao

, Short-term wind power combined forecasting based on error forecast correction, , Energy Conversion and Management 119 (2016), 215–226.

30.

Staron

, Medin

and Söderqvist

, A method for forecasting defect backlog in large streamline software development projects and its industrial evaluation, Information and Software Technology 52 (10) (2010), 1069–1079.

31.

Aiolfi

and Timmermann

, Persistence in forecasting performance and conditional combination strategies, Journal of Econometrics 135 (1-2) (2006), 31–53.

32.

Liu

B.C.

, Lai

M.Z.

, Wu

J.L.

and Binaykia

, Patent analysis and classification prediction of biomedicine industry: SOM-KPCA-SVM model, Multimed Tools & Applications 79 (15-16) (2020), 10177–10197.

33.

Gui

B.X.

, Ju

Y.B.

and Liu

, Mapping technological development using patent citation trees: an analysis of bogie technology, Technology Analysis & Strategic Management 31 (2) (2019), 213–226.

34.

Hall

S.G.

and Mitchell

, Combining density forecasts, International Journal of Forecasting 23 (1) (2007), 1–13.

35.

Xiao

H.W.

, Patent Map Method and Application, Shanghai Jiao Tong University Press, 2011.

36.

Harhoff

, Narin

, Scherer

F.M.

and Vopel

, Citation frequency and the value of patented inventions, Review of Economics and statistics 81 (3) (1999), 511–515.

37.

Lanjouw

J.O.

and Schankerman

, The quality of ideas: measuring innovationwithmultiple indicators (No.w7345), National bureau of economic research, 1999.

38.

Y.P.

, Yao

C.Q.

and Li

, Using heterogeneous patent network features to rank and discover influential inventors, Journal of Zhejiang University Science C 16 (7) (2015), 568–578.

39.

Agrawal

and Henderson

, Putting patents in context: Exploring knowledge transfer from MIT, Management science 48 (1) (2002), 44–60.

40.

Green

W.H.

, Econometric Analysis.(5th edition), Prentice Hall, 2002.

41.

Pawlak

, Rough Sets, International Journal of Computer and Information Sciences 11 (5) (1982), 341–356.

42.

T.C.

, Fan

H.Q.

, Jesús

and Corchado

J.M.

, Second-order statistics analysis and comparison between arithmetic and geometric average fusion: Application to multi-sensor target tracking, , Information Fusion 51 (2019), 233–243.

43.

Biswas

, Kumar

, Jain

and Chandra

, Measuring Supply Chain Reconfigurability using Integrated and Deterministic Assessment Models, , Journal of Manufacturing Systems 52 (2019), 172–183.

44.

Yan

, Zhang

, Feng

and Zhang

, Research on Logistics Demand Forecast of Port Based on Combined Model, Journal of Physics: Conference 1168 (3) (2019), 1–6.

45.

Vlasov

P.V.

, Khrapov

F.I.

, Subota

L.S.

and Cheremina

E.M.

, Comparative Evaluation of the Accuracy and Robustness of Various Information Processing Algorithms from Tripled Measuring Channel, Measurement Techniques 62 (9) (2019), 754–761.