q-Rung orthopair fuzzy TOPSIS method and the application to information service quality evaluation in online health community

3.1 q-Rung orthopair fuzzy sets and operational rules

Definition 1 [28]. Let X be a universe of discourse, a q-ROFS A in X is given by A = { 〈 x , μ A ( x ) , ν A ( x ) 〉 | x ∈ X }

where μ _A  (x) ∈ [0, 1] denotes the membership degree and ν _A  (x) ∈ [0, 1] denotes the non-membership degree, satisfying 0 ⩽ μ _A  (x)  ^q  + ν _A  (x)  ^q  ⩽ 1, q ⩾ 1. And the degree of indeterminacy is given by π A ( x ) = ( 1 - μ A ( x ) q - ν A ( x ) q ) 1 / q .

For convenience, Liu and Wang [25] called (μ _A  (x) , ν _A  (x)) a q-rung orthopair fuzzy number (q-ROFN) which can be denoted by α = (μ _A , ν _A ).

Yager [28, 29] proposed q-ROFSs and gave the related basic operations, based on the score function, Liu and Wang [25] gave the accuracy function, as shown below.

Definition 2 [25]. Suppose P₁ = (μ₁, ν₁) and P₂ = (μ₂, ν₂) are any two q-rung orthopair fuzzy numbers, the mathematical operations are defined as follows:

P 1 ⊕ P 2 = ( ( μ 1 q + μ 2 q - μ 1 q μ 2 q ) 1 / . q , ν 1 ν 2 )

Definition 3 [25]. For any q-ROFN p = (μ, ν), a score function of p is defined as

S (p) = μ ^q  - ν ^q , where S (p) ∈ [-1, 1].

An accuracy function of p is defined as

A (p) = μ ^q  + ν ^q , where A (p) ∈ [-1, 1].

Then the comparison of two q-ROFNs can be calculated, suppose p₁ and p₂ are any q-ROFNs, then

If S (p₁) < S (p₂), then p₁ < p₂.

Definition 4 [25]. Let {β₁, β₂, . . . , β _n } be a collection of q-ROFNs, q-rung orthopair fuzzy weighted aggregation (q-ROFWA) operator is defined by Liu, q - ROFWA ( β 1 , β 2 , . . . , β n ) = w 1 β 1 ⊕ w 2 β 2 ⊕ ⋯ ⊕ w n β n = ( ( 1 - ∏ k = 1 n ( 1 - ( μ k ) q ) w k ) 1 / . q , ∏ k = 1 n ( v k ) w k ) , where W = (w₁, w₂, . . . , w _n )  ^T is the weighted vector related to β _k  = (μ _{β k} , v _{β k} ), k = (1, 2, . . . , n) with the condition that w _k  ∈ [0, 1], ∑ k = 1 n w k = 1 .

By taking different values of the parameter q, the q-ROFWA operator can be applied to different cases.

When q = 1, the q-ROFWA operator reduces to IFWA operator [43]: q - ROFWA ( β 1 , β 2 , . . . , β n ) ( q = 1 ) Imacr ; FWA ( β 1 , β 2 , . . . , β n ) = ( 1 - ∏ k = 1 n ( 1 - μ k ) w k , ∏ k = 1 n ( v k ) w k ) .

3.2 Distance measure of fuzzy sets

Definition 5 [44]. Suppose A = {A₁, A₂, . . . , A _n } and B = {B₁, B₂, . . . , B _n } are any two IFSs, and the generalized distance measure is defined as follows, d ( A , B ) = 1 2 ∑ j = 1 n ( | ( μ A j ) - ( μ B j ) | λ + | ( ν A j ) - ( ν B j ) | λ + | ( π A j ) - ( π B j ) | λ ) 1 / . λ .(1)

Based on above geometrical distance model, Biswas [1] improved this distance measure between two PFNs, which can be reduced to Hamming distance (λ = 1) and Euclidean distance (λ = 2).

Definition 6 [36]. For any two PFSs P = {P₁, P₂, . . . , P _n } and Q = {Q₁, Q₂, . . . , Q _n }, let P _j  = (μ _{p j} , ν _{p j} ) and Q _j  = (μ _{Q j} , ν _{Q j} ), (j = 1, 2, . . . , n) be two PFNs, the weighted generalized distance is defined as follows, D ( P , Q ) = 1 2 ∑ j = 1 n w j ( | ( μ p j ) 2 - ( μ Q j ) 2 | + | ( ν p j ) 2 - ( ν Q j ) 2 | + | ( π P j ) 2 - ( π Q j ) 2 | ) , where w _j  = (w₁, w₂, . . . w _j )  ^T indicates the weight vector of the jth PFN.

The principle of TOPSIS method defined by Hwang and Yoon [5] is based on the closest distance to the positive ideal solution (PIS) and the farthest distance from the negative ideal solution (NIS). The main steps of this method include:

Construct the original matrix.

The detailed steps of TOPSIS with q-ROFS will be described in Section 5.

4.1 Distance measure for q-ROFS

Based on the PFS weighted generalized distance from Definition 6, this research extends the definition to q-ROFS environments.

Definition 7. Let P = {P₁, P₂, . . . , P _n } and Q = {Q₁, Q₂, . . . , Q _n } be any two q-ROFSs, where P _j  = (μ _{p j} , ν _{p j} ) and Q _j  = (μ _{Q j} , ν _{Q j} ) are two q-ROFNs (j = 1, 2, . . . , n), the weighted generalized distance measure is defined as follows, D ( P , Q ) = 1 2 ∑ j = 1 n ω j ( | ( μ p j ) q - ( μ Q j ) q | + | ( ν p j ) q - ( ν Q j ) q | + | ( π P j ) q - ( π Q j ) q | ) , π P j = ( 1 - ( μ P j ) q - ( ν P j ) q ) 1 / . q , π Q j = ( 1 - ( μ Q j ) q - ( ν Q j ) q ) 1 / . q , , where ω _j  = (ω₁, ω₂, . . . , ω _n ) indicates the weight vector of the jth q-ROFN with the condition that and ∑ j = 1 n ω j = 1 .

4.2 Entropy weight model for q-ROFS

The entropy measure is an objective weighting method based on the amount of information contained in each criterion, which can measure the amount of information uncertainty in a system. Generally, the greater the amount of information, the higher the degree of variation in data criteria. Then the information entropy of this criterion is smaller; on the contrary, the criterion with larger information entropy indicates that the uncertainty of the data is greater, and the variation degree of criterion is lower. Based on the Pythagorean fuzzy entropy defined by Xue [32], a generalized q-rung orthopair fuzzy entropy (q-ROFE) is defined by the relationship between membership and non-membership and the definition of indeterminacy degree.

Definition 8. Let P = {P₁, P₂, . . . , P _n } be any q-ROFS, where P _i  = (μ _{P i} , ν _{P i} ) (i = 1, 2, . . . , n) is a q-ROFN, the entropy measure for q-ROFS is defined as follows, E ( P ) = 1 n ∑ i = 1 n [ 1 - | μ P q ( x i ) - ν P q ( x i ) | + π P q ( x i ) - π P q ( x i ) { 1 - | μ P q ( x i ) - ν P q ( x i ) | } ] = 1 n ∑ i = 1 n [ 1 - ( π P q ( x i ) - 1 ) | μ P q ( x i ) - ν P q ( x i ) | ] = 1 n ∑ i = 1 n [ 1 - ( μ P q ( x i ) + ν P q ( x i ) ) | μ P q ( x i ) - ν P q ( x i ) | ] .

According to the Pythagorean fuzzy entropy (PFE) [32], it can be proved that the q-rung orthopair fuzzy entropy measure proposed in this paper can also satisfy these following properties:

Definition 9. Let P and Q be two q-ROFSs on universe of discourse X. An entropy measure E(P) is a mapping E: q-ROFS(X) ⟶ [0,1], satisfying the following properties:

E (P) =0 if and only if P is a crisp set.

Proof. 1. For a crisp set P, μ _P  (x _j ) =1, ν _P  (x _j ) =0 or μ _P  (x _j ) =0, ν _P  (x _j ) =1, then E (P) =0. Conversely, let E (P) =0, i.e., 1 - ( μ p q ( x j ) + ν p q ( x j ) ) | μ p q ( x j ) - ν p q ( x j ) | = 0 . Then 0 ⩽ μ p q ( x j ) + ν p q ( x j ) ⩽ 1 , 0 ⩽ | μ p q ( x j ) - ν p q ( x j ) | ⩽ 1 , given that μ P q ( x j ) + ν P q ( x j ) = 1 , | μ P q ( x j ) - ν P q ( x j ) | = 1 . Since μ _P  (x _j ) , ν _P  (x _j ) ∈ [0, 1], so μ _P  (x _j ) =1, ν _P  (x _j ) =0 or μ _P  (x _j ) =0, ν _P  (x _j ) =1. Hence, P is a crisp set.

Proof. 2. For ∀x _j  ∈ X, let μ _P  (x _j ) = ν _P  (x _j ), then ( μ p q ( x j ) + ν p q ( x j ) ) | μ p q ( x j ) - ν p q ( x j ) | = 0 , hence E (P) =1. Conversely, if E (P) =1, then ( μ p q ( x j ) + ν p q ( x j ) ) | μ p q ( x j ) - ν p q ( x j ) | = 0 . Hence μ _P  (x _j ) = ν _P  (x _j ).

Proof. 3. E ( P c ) = 1 n ∑ i = 1 n [ 1 - ( ν P j q + μ P j q ) | ν P j q - μ P j q | ] = 1 n ∑ i = 1 n [ 1 - ( μ P j q + ν P j q ) | μ P j q - ν P j q | ] = E ( P )

Proof. 4. If P is less fuzzy than Q, then μ _P  ⩽ μ _Q and ν _P  ⩾ ν _Q for μ _Q  (x _j ) ⩽ ν _Q  (x _j ), ∀x _j  ∈ X, then μ _P  (x _j ) ⩽ μ _Q  (x _j ) ⩽ ν _Q  (x _j ) ⩽ ν _P  (x _j ). E ( Q ) - E ( P ) = 1 n ∑ j = 1 n [ ( μ P q ( x j ) + ν P q ( x j ) ) | μ P q ( x j ) - ν P q ( x j ) | - ( μ Q q ( x j ) + ν Q q ( x j ) ) | μ Q q ( x j ) - ν Q q ( x j ) | ] = 1 n ∑ j = 1 n [ ( μ P q ( x j ) + ν P q ( x j ) ) ( ν P q ( x j ) - μ P q ( x j ) ) - ( μ Q q ( x j ) + ν Q q ( x j ) ) ( ν Q q ( x j ) - μ Q q ( x j ) ) ] = 1 n ∑ j = 1 n [ ( ν P 2 q ( x j ) - μ P 2 q ( x j ) ) - ( ν Q 2 q ( x j ) - μ Q 2 q ( x j ) ) ] = 1 n ∑ j = 1 n [ ν P 2 q ( x j ) - μ P 2 q ( x j ) - ν Q 2 q ( x j ) + μ Q 2 q ( x j ) ] = 1 n ∑ j = 1 n [ ( ν P q ( x j ) - ν Q q ( x j ) ) ( ν P q ( x j ) + ν Q q ( x j ) ) + ( μ Q q ( x j ) - μ P q ( x j ) ) ( μ P q ( x j ) + μ Q q ( x j ) ) ]

Since ( ν P q ( x j ) - ν Q q ( x j ) ) ⩾ 0 and ( μ Q q ( x j ) - μ P q ( x j ) ) ⩾ 0 , then E (Q) - E (P) ⩾0, that is E (P) ⩽ E (Q).

6.1 Information service quality evaluation in online health community

In this section, the performance of the proposed method is validated by a real-word case about the evaluation of information service quality in online health communities in China. Online health communities play a positive role in alleviating hospital congestion and improving medical resources utilization, especially in China with a large population base. Because the information quality in these communities is not perfect, evaluating the information service quality of the online health community has a good reference significance for ordinary users, medical enterprises and the government.

In order to ensure the validity and scientificity of the evaluation criteria system, an evaluation criteria system model was established, which adopted the previous scales from related literature and integrated additional criteria based on medical industry characteristics. This research carried out an online questionnaire survey with a 5-point Likert-type response format through Wenjuanxing, a professional and popular questionnaire platform in China. After deleting invalid data from a total of 211 responses, we received 190 valid responses, and verified that the data passed the reliability and validity test and met the requirements for PCA. The PCA method was adopted to reduce the dimensions and optimizes the criteria model. After revising and correcting the criteria model, the evaluation criteria model with 7 principal criteria and 41 sub-criteria was built, as shown in Table 3. The relevant questionnaire data and detailed result of PCA are presented in Appendix.

In this MADM problem, these seven evaluation criteria were used as evaluation attributes and the evaluation preference were obtained for four online health communities through expert interview. Then three experts with experiences in online health communities were invited to make decisions and express their evaluation degree assigned in the form of q-ROFNs, including professors and associate professors, the weight of experts is assigned according to their titles. This research set the experts’ importance degree as “very important”, “very important” and “important”, and convert linguistic variables into fuzzy numbers according to the formula [20]. The reference value can provide authoritative judgments about the quality of online health community information services, which satisfies the research needs. The evaluation results are displayed in the following q-ROFS decision matrices (q-ROFDMs). In the following, this paper adopts the proposed q-ROF TOPSIS method to deal with the practical decision-making problem. Moreover, a detailed comparative analysis with other typical methods is handled to prove the applicability and superiority of the new method.

The four domestic online health communities are denoted as A₁, A₂, A₃, A₄, respectively. According to the evaluation criteria system model established by previous questionnaire survey, the following criteria (attributes) are identified:

C₁: Internal quality of information.

The weight of attributes is shown as ω = {ω₁, ω₂, ω₃, ω₄, ω₅, ω₆, ω₇}, satisfying ω _j  ∈ [0, 1] and ∑ j = 1 7 ω j = 1 ; the weight of DMs is shown as σ = {σ₁, σ₂, σ₃}, satisfying σ _k  ∈ [0, 1] and ∑ k = 1 3 σ k = 1 . The weights of the three experts are shown as linguistic variables in the Table 4. For convenience, the relative importance ratings of expert will be transformed into q-ROFNs according to the rule from Table 1. The experts give the assessment value with respect to each criterion (attribute) in q-ROFNs, and D k = [ d ij k ] 4 × 7 ( k = 1 , 2 , 3 ) expresses separate q-ROFDMs from three experts, as shown in Table 5, Tables 6 and 7. To guarantee the effective evaluation information and accurate conclusion, the specific decision-making process for online health community information service quality evaluation is shown as follows. And the proposed evaluation framework of online health community is shown in Fig. 3.

OPEN IN VIEWER

Fig. 3

The proposed evaluation framework of online health community.

Step   1. All the evaluation criteria of online health communities are benefit attributes, so the q-ROFDM does not have to be normalized.

Step   2. Identify the weight of DMs (experts) based on the formula. The result is shown in Table 8.

Step   3. The q-ROFDMs from different experts are aggregated by q-ROFWA (q = 2). The results are shown in the Table 9. x ij = q - ROFWA σ ( x ij 1 , x ij 2 , . . . , x ij k ) = σ 1 d ij 1 ⊕ σ 2 d ij 2 ⊕ . . . ⊕ σ k d ij k = ( ( 1 - ∏ t = 1 k ( 1 - ( μ ij ) 2 ) σ k ) 1 / . 2 , ∏ t = 1 k ( v ij ) σ k )

Step   4. Apply the entropy measure for q-ROFS to determine the weights of attributes, the weight of attributes is shown in Table 10.

Step   5. Calculate the PIS and NIS of q-ROFS. The result is shown in Table 11.

Step   6. Calculate the distance from each alternative to the PIS and NIS. The result is showen in Table 12.

Step   7. Calculate the revised relative closeness index, the result is also shown in Table 12.

According to the revised relative closeness index of the alternatives, the final ranking of alternatives is: A 2 ≻ A 1 ≻ A 4 ≻ A 3 .

Therefore, A₂ is the best alternative, which means A₂ has the best quality of information service among the selected online health communities. It also explains that A₂ provides better references for users seeking information. Online health communities with higher evaluation scores have better information services, however, for communities with lower rankings, the seven attributes proposed in this study should be used to make up for the lack of development, improve the ability to serve users and create better service experience. As a whole, this method solves the evaluation and selection about online health communities and provides a convincing reference for users to make an optimal choice.

The above numerical example can prove the practicability of the proposed method, which can effectively solve the comprehensive evaluation problem in reality. In the following, this research compares the q-ROF TOPSIS method proposed in this paper with other existing methods to further verify this method’s optimality. The detailed comparative analysis is mainly carried out from the reduction of information loss, the determination of attribute set, the expression space of fuzzy numbers, the obviousness of ranking results, the way to determine the attribute weight, the number of decision matrices, and the simplicity of calculation process. Table 13 shows the methods selected for comparison.

The ranking results are shown in Table 14. Based on the above ranking result, it can be found that the result of M2-M6 is consistent, which is A₂ ≻ A₁ ≻ A₄ ≻ A₃. It is worth noting that the results of M1 are different from other methods, and the overall ranking results are same except M1. Although the result of M1 is slightly different, the best and worst alternative is still the same. The overall stable result can prove the accuracy and reliability of the method proposed in this paper. The following is a detailed analysis of each method.

The first three methods apply the highly aggregation operators and determine the optimal alternative based on the score values, the last three methods combine with traditional TOPSIS method, and rank alternatives according to the relative closeness indices. In addition, it is worth noting that the above methods did not explain the source of the attribute, but directly set the attributes (criteria) for practical application. In contrast, this paper builds an evaluation criteria system through PCA method focusing on the information quality evaluation problem, which ensures that comprehensive and accurate research results are obtained. Because the M1, M2 and M3 methods can deal with the situation where only one expert or multiple experts’ weight is known, in these methods, each expert is equally important and the weight is 1/3; The M4 method can only calculate intuitionistic fuzzy numbers, so the evaluation data is adjusted so that the sum of membership and non-membership does not exceed one for comparative analysis; The M5 method does not involve the integration of multiple experts in the first step, so this paper applies one of the most important expert decision matrices for calculation; and the M6 method is new one that proposed in this paper. The following comparative analysis explains the superiority of the proposed method (M6).

(1) Compared with the M1 method, M2 method and M3 method.

Firstly, the first three methods apply q-ROFNs to express the fuzzy information, which can describe a larger decision-making space with stronger information capacity. Secondly, these three methods use different aggregation operators to integrate multiple times and rank the alternatives, which inevitably lead to some information distortion, it can be found that the gap among scores values are not significant. In contrast, the M6 method can reduce the loss of information and ensure the accuracy of decision result by evaluating the distance between the different alternatives to the PIS and NIS. In addition, while using the q-ROFWBM operator of the M3 method, this method sets s = t = 1 and q = 2, but the final result can be influenced by the different parameters set. The aggregation operators have additional requirements and specific functions, they are suitable for specific application purposes, for example, q-ROFWBM operator is suitable for considering interrelationships between parameters. Thus, the application scope of these aggregation operators is limited and lacks universality. Thirdly, these three methods focus on proposing a new aggregation operator and directly provide the weight value, which has a certain degree of subjectivity. There is a lack of research on determining expert weights or attribute weights in actual MADM problems. The M6 method in this paper proposes an entropy measure method to determine the weight of criteria, as an objective assignment method, it determines the weight value according to the amount of information contained in each criterion, which can better ensure the scientificity and accuracy of the decision-making results.

(2) Compared with the M4 method and M5 method.

The comparison results of the M4-M6 methods are shown in Fig. 4.

OPEN IN VIEWER

Fig. 4

The ranking result of the M4-M6 method.

Firstly, in terms of q-ROFS, the expression space of fuzzy information is bigger and it can flexibly adjust the parameter q to over more hesitant fuzzy evaluation information and adapt to more complex and dynamic decision-making environments. The M4 method involves multiple decision matrices from a group of experts and the evaluation preference need to satisfy the tighter restrictions, that is, the sum of membership degree and non-membership degree does not exceed one, for example, the value (0.75,0.29) is hard to calculate by IFNs. Secondly, it is worth noting that the gap among four relative closeness indices from M4 is smaller than the result from M6. It is obvious that the result of M6 is easier to distinguish. Thirdly, the M5 method is similar to the case of q = 2 in this paper. However, the weight of each attribute and the decision matrices are known quantities in M5 before decision-making, so it cannot indicate the source of information and has a certain degree of subjectivity and randomness. In contrast, the M6 method utilizes the first aggregation technology to process the decision matrices of multiple experts, integrates them into a comprehensive decision matrix, which can not only avoid the inaccurate evaluation of too few experts, but also reduce the information loss and distortion. Moreover, from the perspective of practical application, especially in multi-attribute decision-making (MADM) problems, the input of multiple attributes with different weights will also affect the decision result. The M6 method uses the objective entropy measure model to determines the attribute weights, which can reduce the impact of unreasonable evaluation values and ensure the fairness and accuracy of the evaluation results.

(3) Comparing with q-ROF ELECTRE method.

Concentrate on other MADM methods, the ELECTRE, as a multi-criteria decision analysis technique, has been applied to some fuzzy MADM problems [13, 19]. Pinar [4] has proposed the q-ROF ELECTRE method and gave the decision-making steps, which involves a selection process based on the outranking relationship. Under the q-ROF ELECTRE method, the standardization and construction of decision matrix in the decision-making steps are consistent with the q-ROF TOPSIS method. The difference is that the ELECTRE method needs to determine the concordance set and discordance set and then construct the strong, moderate and weak sets. After constructing a comprehensive judgment matrix, it eliminates the inferior alternatives which are outranked by others until the best one is left. Here shows the strong concordance set C _kl , moderate concordance set C _kl , weak concordance set C _kl of A _k and A _l , and the strong discordance set D _kl , moderate discordance set D _kl , weak discordance set D _kl .

C kl = [ - - 2 , 3 , 4 7 1 , 2 , 4 , 6 , 7 - 1 , 2 , 3 , 4 , 5 , 6 1 , 2 , 6 , 7 - - - - 3 , 4 , 6 - 3 , 4 , 5 , 6 - ] , C kl ′ = [ - 3 1 , 5 , 6 1 - - - - - - - 7 2 3 , 5 1 , 2 - ] , C kl ″ = [ - - 7 - 5 - 7 - - - - 1 5 4 7 - ] , D kl = [ - 1 , 2 , 4 , 6 , 7 - 3 , 4 , 6 - - - - 2 , 3 , 4 1 , 2 , 3 , 4 , 5 , 6 - 3 , 4 , 5 , 6 7 1 , 2 , 6 , 7 - - ] , D kl ′ = [ - - - 2 3 - - 3 , 5 1 , 5 , 6 - - 2 1 - - - ] , D kl ″ = [ - 5 - 5 - - - 4 7 7 - - - - - - ] .

The result of concordance matrix and discordance matrix are calculated as follows, C = [ - 0.119 0.773 0.235 0.745 - 0.902 0.536 0 0 - 0.142 0.606 0.252 0.819 - ] , D = [ - 1 0 0.799 0.451 - 0 0.387 0.884 1 - 1 0.667 1 0 - ] .

Based on the threshold value of the concordance and discordance indexes, which is the average value of the matrix ( c - = 0.4275 , d - = 0.599 ). The concordance and discordance dominance matrices and aggregate dominance matrix can be calculated and represented by the F, G and E respectively. F = [ - 0 1 0 1 - 1 1 0 0 - 0 1 0 1 - ] , G = [ - 0 1 0 1 - 1 1 0 0 - 0 0 0 1 - ] , E = [ - 0 1 0 1 - 1 1 0 0 - 0 0 0 1 - ] .

According to the principle of q-ROF ELECTRE, if the elements E_kl in the matrix are equal to one, then A_k is preferred to A_l for both the concordance and discordance criteria, so the ranking result is A₁ ≻ A₃ and A₂ ≻ A₁, A₂ ≻ A₃, A₂ ≻ A₄ an A₄ ≻ A₃. Regarding the information service quality in China’s online health communities, A₂ is still the optimal alternative and A₃ is the least ideal alternative. However, this calculation process cannot rank all of the alternatives because of the middle alternatives A₁ and A₄.

This case expresses the evaluation value provided by the three DMs for four alternatives with respect to the seven attributes for selection and evaluation. It can be seen that this process begins with a comparison of the seven evaluation criteria of any two alternatives. However, if the number of alternatives and attributes is large, M7 will become complicated and hard to calculate, while the TOPSIS method applied in this paper is simple and general. Moreover, although the q-ROF ELECTRE method can also obtain the outranking results through gradual elimination and selection, it cannot express the degree of quantity. On the contrary, the q-ROF TOPSIS method can obtain the specific value through closeness indices, displaying a clear ranking and allowing decision-makers to see the specific information and make a better comparison.

Therefore, the q-ROF TOPSIS method has advantages in the expression and processing of fuzzy decision information. Based on the comparisons and analysis above, it can be verified that the method proposed in this paper is better than the other methods.