An integrated QFD and 2-tuple linguistic method for solution selection in crowdsourcing contests for innovative tasks

Abstract

Solution selection plays an important role in crowdsourcing and is an imperative work for requesters. However, to the best of our knowledge, there is few studies focus on the problem of solution selection, especially in crowdsourcing contests for innovative tasks. This paper aims to develop a methodology incorporating quality function deployment (QFD) with 2-tuple linguistic method to assist requesters to select the right solution from a large pool of potential solutions efficiently. The methodology includes three phases. The first phase, i.e. pre-selection, is to screen potential solutions by employing the rule of non-compensatory. The second phase is to construct relationships between requester’s requirements and solution features using quality function deployment (QFD), and further to determine the weights of solution features using 2-tuple linguistic weighted average operator and fuzzy weighted average method. The last phase is to evaluate the performance of potential solutions with respect to solution features, and further estimate their overall performance. Finally, an illustrative application case on the crowdsourcing platform-Taskcn is presented to demonstrate the implementation and effectiveness of the proposed approach.

Keywords

Crowdsourcing contests innovative tasks solution selection quality function deployment 2-tuple linguistic method

1. Introduction

The term “crowdsourcing” refers to a model that outsources tasks to a large and possibly undefined group of people. In recent years, it has attracted much attention from organizations [1], and gained in popularity for performing diversities of task [2]. Specifically, innovative tasks, such as product development [3], logo design [4], public transportation stop design project [5], and t-shirt design [6], as one of utmost task type [7, 8], its crowdsourcing has become an extremely promising domain of crowdsourcing [9]. Meanwhile, some crowdsourcing sites are structured as contests in support of publishing the tasks, such as InnoCentive, Crowd Flower, and Taskcn in china.

In crowdsourcing contests for innovative tasks, requesters publish a task with details of its requirements, awards, and expected duration. Potential participants who have interests and skills contribute to the task and submit their solutions. Finally, one solution which meets requester’s demands best is selected to carry out the task [10]. Generally, judging and choosing the best solution is the last step of crowdsourcing. Its result has directly great impacts on the quality of task completion and requester’s satisfaction. However, selecting the right solution is not an easy but a difficult work costing time and energy for requesters, because the number of submitted solutions provided by participants is always large and varying from each other in terms of features, contents, and forms. Requesters are required to evaluate each submitted solutions carefully and effectively.

Currently, in crowdsourcing systems, some approaches have been proposed to assess outcomes submitted by participants. The commonly used approaches are voting/rating mechanisms, text mining techniques, and machine learning approaches. For example, Ipeirotis et al. [11] presented an algorithm for evaluating and managing the quality of labeling process on Amazon Mechanical Turk. The algorithm is capable of generating a scalar score representing the inherent quality of each worker, allowing for separating the intrinsic error rate from the bias in worker answers. Le et al. [12] suggested an approach including an initial training period and subsequent sporadic insertion of predefined gold standard data in order to ensure quality of worker judgments. The experiment results on relevance categorization task on Amazon Mechanical Turk verified that the approach enables to ensure the accuracy of outcomes produced by individual worker. With regard to Le et al’s approach, Oleson et al. [13] noted that it has the weakness of requiring extensive collection of gold standard data manually, and further developed an algorithm to generate gold standard data automatically and ensure important gold properties as well. Besides, Burmania et al. [14] developed a reference set with predetermined ground-truth to monitor annotators’ accuracy and fatigue. By evaluating each worker’s quality in real-time and automatically stopping evaluation when worker’s quality does not meet the predetermined threshold, the system presented in the paper can help to collect accurate annotations.

The studies mentioned above mainly focus on simple tasks, such as image annotation and labels processing, and present some approaches to assess the quality of outcomes provided by workers. There is, to the best of our knowledge, few studies investigate the problem of solution evaluation and selection in crowdsourcing contests for innovative tasks. Unlike simple tasks which have one (or more) correct answer(s) out of a finite set of possible answers [15], there is no totally correct solutions for an innovative task. Consequently, it may lead to the models and approaches, such as gold-standard-based quality evaluation approach [11 –13] and reference sets with predetermined ground-truth [14], cannot be applied to address the problem of solution selection in crowdsourcing contests for innovative tasks. New approaches are required to enable requesters to select qualified solutions appropriately and effectively. In the meantime, the following properties need to be considered while developing the approach. First, the aim of requesters is to select the best solution which has a higher satisfaction to meet their requirements. However, in practical, participants generally transfer these requirements described vague and implicit into the features of solutions that they developed. It may be more suitable and feasible to assess each solution against each solution feature rather than requesters’ requirements because the solution features are clear, concrete, and easy to be observed and assessed. Under this condition, to achieve the goal, we should comprehensively take into account the relationships between requesters’ requirements (RRs) and solution features (SFs) as well as the correlations among SFs that are always not independent. Second, during the process of evaluating solutions, linguistic labels rather than crisp number are commonly preferred by requesters to impress their opinions on the performance of solutions and relation degree between RRs and SFs. There is a need for proposed approach that can be able to deal with the imprecise and qualitative information. Third, due to the fact that the submitted solutions may be large, the approach is required to assess and select the right solution more effectively.

In view of above conditions, this paper develops a methodology with three phases for solution selection in crowdsourcing contests for innovative tasks. The first is pre-selection stage. The purpose of this stage is to screen a set of solutions that meet the minimum requirements so they can be qualified as the potential solutions and will be evaluated in the next two phases. In the second phase, we construct the relationships between RRs and the potential solutions features by using the house of quality (HOQ), the first matrix in QFD. Afterwards, the weight of each SF is determined by considering the importance of RRs, the inner dependencies of SFs, and the relations between RRs and SFs. Therein, 2-tuple linguistic method [16 –18] which allows to reduce the loss of information is employed to represent, process, aggregate flexible information. The last phase is to assess the performance of each potential solution with respect to each SF by using 2-tuple linguistic variables. The priority of each solution is further determined by aggregating its local score against on each SF and its weight. The final evaluation results enable requesters to select the appropriate solution to carry out their tasks.

The rest of this paper is organized as follows. In the next section, the QFD and 2-tuple linguistic method are illustrated in order to provide a base of knowledge. Section 3 presents the methodology consisting of three phases to address the problem of solution selection in crowdsourcing contests for innovative tasks. Subsequently, in Section 4, an illustrative application case is conducted to demonstrate the implementation and feasibility of the proposed approach. Section 5 will discuss advantages and disadvantages of the proposed methodology comparing with other related methods. Finally, we conclude the paper in Section 6.

2. Preliminaries

This section will illustrate the QFD and 2-tuple linguistic method introduced in this paper in order to make the study as self-contained as possible.

2.1. Quality function deployment

The QFD is an important approach of developing product which integrates customer needs throughout the total process of product development [19, 20]. It offers tools for transferring customer needs into technical requirements, and further into part characteristics, process plans and production requirements [21, 22]. The QFD not only provides a structure approach for quality-driven product development, but also has advantages in prioritizing and defining information to be transformed from customer requirements to specific stages of product development.

The QFD is implemented by employing four matrices [23], which are product planning, part deployment, process plans, and production planning, respectively. Each matrix is corresponding to a stage of product development cycle. This paper mainly concentrates on HoQ, the first matrix in QFD. Generally, a typical HoQ is composed of seven parts, i.e., customer needs (CNs) and their relative importance, technical attributes (TAs) and their correlations, the relationships between CNs and Tas, competitive benchmarking and prioritized technical descriptors and targets, respectively [21]. The HoQ matrix is shown in Fig. 1.

CNs, which are also called customer requirements or customer attributes, are the initial inputs in HoQ. They offer the company a guideline to estimate what characteristics that product should possess. The first step of constructing a HoQ matrix is to investigate, identify, and determine CNs. With respect to the CNs, their importance must be assessed to assist companies to determine the priority of each CN to be satisfied. Some methods, such as AHP and simple direct rating, can be employed to measure their importance.

TAs, which also refer to product features, design requirements, or technical requirements, are the product requirements that have directly relations with CNs. They reflect how well the developed product matches with CNs. Commonly, TAs are interrelated rather than independent from each other [24]. To account for interrelations among TAs and facilitate informed trade-offs, their inner independencies are marked in the roof of HoQ matrix [22].

Fig. 1.

House of quality matrix.

The relationships between CNs and TAs are presented in the body of HoQ matrix, which is used to illustrate how much a TA affect a CN. Generally, linguistic variables and fuzzy linguistic methods are employed to measure the correlation degree between them [2, 19].

Competitive benchmarking is used to compare the performance of a product in terms of CNs with those of its competitors. Prioritized technical descriptors and targets are used to identify the relative importance of TAs with respect to CNs. The more important of a technical attribute, the more attention it will be got in the process of product development.

2.2. The 2-tuple linguistic method

For some situations with qualitative aspects that are difficult to assess employing exact and precise values, we prefer to use inaccurate words in a natural language, such as “very likely”, “nearly”, and “probably so”, to express our evaluation information and opinions. To model and deal with the qualitative information, the fuzzy linguistic method [25] is a very feasible approach and has been proved to be useful in many problems [17 , 26–28]. In this paper, the 2-tuple linguistic method will be introduced to deal with fuzzy linguistic information. The method [16–18 , 29] is a continuous model of representation of information avoiding the loss of information and lack of precision in the aggregation process. To define it, the 2-tuple linguistic variables representing linguistic information and the 2-tuple linguistic computation operators that transform and aggregate linguistic information are presented as follows.

(1) Definition of 2-tuple linguistic variables

Suppose S ={ s₀, s₁, …, s_u } is a finite linguistic term set with odd cardinality, where s_i ∈ S is ith linguistic term. All terms distribute on a scale which a total order is defined, s_i ≤ s_j ⇔ i ≤ j.

Let β be a numerical value representing aggregation result of linguistic symbolic, where β ∈ [0, u] and β∉ { 0, …, u }. Then a transformation function (Δ) is defined to realize transfer between 2-tuple linguistic variables and numerical values [17]: $\begin{matrix} Δ : [0, u] \to s \times [- 0.5, 0.5) \\ Δ (β) = (s_{i}, α), with {\begin{matrix} s_{i}, i = round (β) \\ α = β - i, α \in [- 0.5, 0.5) \end{matrix} \end{matrix}$ (1)

Where round (•) is the usual round operation. s_i has the closest index label to “ β”. “ α” is the value of the symbolic translation, where i ∈ [0, u] and α ∈ [- 0.5, 0.5).

Meanwhile, a 2-tuple linguistic variable (s_i, α) can also be transferred into numerical value β ∈ [0, u] by the following equation. $\begin{matrix} Δ : S \times [- 0.5, 0.5) \to [0, u] \\ Δ^{- 1} (s_{i}, α) = i + α = β \end{matrix}$ (2)

By using symbolic translation process mentioned above, we can transform 2-tuple linguistic variables into its equivalent numerical value β and vice versa. Furthermore, linguistic information can be expressed by means of the 2-tuple linguistic variable denoted as (s_i, α_i), where α_i is a numerical value indicating the distance between the original result β and the central value of the closest term s_i. For example, the linguistic term s_i can be converted into a 2-tuple linguistic variable by adding a value zero as symbolic translation, i.e. s_i ∈ S ⇒ (s_i, 0).

(2) Computation of 2-tuple linguistic variables

Assume that (s_i, α_i) and (s_j, α_j) are two 2-tuple linguistic variables, the computation of them are described as follows [17]:

Algebraic operation:

$\begin{matrix} (s_{i}, α_{i}) \oplus (s_{j}, α_{j}) = Δ (Δ^{- 1} (s_{i}, α_{i}) + Δ^{- 1} (s_{j}, α_{j})), \\ (s_{i}, α_{i}) \otimes (s_{j}, α_{j}) = Δ (Δ^{- 1} (s_{i}, α_{i}) \cdot Δ^{- 1} (s_{j}, α_{j})), \end{matrix}$

where ⊗ and ⊕ are the symbolic of multiplication and addition operations.

Negative operation: Neg ((s_i, α_i)) = Δ (u - (Δ^-1 (s_i, α_i))).

Comparative operation: if Δ^-1 ((s_i, α_i)) > Δ^-1 ((s_j, α_j)), then (s_i, α_i) is bigger than (s_j, α_j). If Δ^-1 ((s_i, α_i)) = Δ^-1 ((s_j, α_j)), then (s_i, α_i) and (s_j, α_j) represent the same information.

To handle the situation of the computational results be out of the scope of s_u, we can extend the granularity u large enough [30], and the semantic of the terms out of linguistic term set do not be considered since the operation results only appear in computation [31].

(3) Aggregation of 2-tuple linguistic variables

For 2-tuple linguistic variables, some existing aggregation operators, such as arithmetic mean, weighted average operator, and linguistic weighted average operator, can be extended to process them.

Arithmetic mean: suppose a 2-tuple linguistic variable set x ={ (r₁, α₁), …, (r_n, α_n) }, their 2-tuple arithmetic mean $\bar{x}$ can be computed as [27]:

$\begin{matrix} \bar{x} ((r_{1}, α_{1}), \dots, (r_{n}, α_{n})) = Δ (\frac{1}{n} \sum_{i = 1}^{n} Δ^{- 1} (r_{i}, α_{i})) \\ = Δ (\frac{1}{n} \sum_{i = 1}^{n} β_{i}) \end{matrix}$ (3)

Weighted average operator: suppose a 2-tuple linguistic variable set x ={ (r₁, α₁), …, (r_n, α_n) }, and ω ={ ω₁, …, ω_n } be their associated weights, then the 2-tuple weighted average ${\bar{x}}^{ω}$ can be computed as [17, 27]:

\begin{matrix} {\bar{x}}^{ω} ((r_{1}, α_{1}), \dots, (r_{n}, α_{n})) \\ = Δ (\frac{\frac{1}{n} \sum_{i - 1}^{n} (Δ^{- 1} (r_{i}, α_{i}) \cdot ω_{i})}{\sum_{i - 1}^{n} ω_{i}}) \end{matrix}

$\begin{matrix} = Δ (\frac{\frac{1}{n} \sum_{i - 1}^{n} β_{i} \cdot ω_{i}}{\sum_{i - 1}^{n} ω_{i}}) \end{matrix}$ (4)

Linguistic weighted average operator: suppose a 2-tuple linguistic variable set x ={ (r₁, α₁), …, (r_n, α_n) }, and ω = ${(ω_{1}, α_{1}^{ω}), \dots, (ω_{n}, α_{n}^{ω})}$ be their linguistic 2-tuple weights, then the 2-tuple linguistic weighted average ${\bar{x}}_{l}^{ω}$ can be computed as [17, 27]:

$\begin{matrix} {\bar{x_{l}}}^{ω} (((r_{1}, α_{1}), (ω_{1}, α_{1}^{ω}), \dots, (r_{n}, α_{n}), (ω_{n}, α_{n}^{ω}))) \\ = Δ (\frac{\sum_{i = 1}^{n} (Δ^{- 1} (r_{i}, α_{i}) \cdot (ω_{i}, α_{i}^{ω}))}{\sum_{i = 1}^{n} Δ^{- 1} (ω_{i}, α_{i}^{ω})}) = Δ (\frac{\sum_{i = 1}^{n} β_{i} \cdot β_{ω_{i}}}{\sum_{i = 1}^{n} β_{ω_{i}}}) \end{matrix}$ (5)

3. Methodology

In this section, a methodology with three phases is firstly developed for solution selection in crowdsourcing contests for innovative tasks. They are solution pre-selection, weighting of solution features, and solution evaluation and prioritization.

3.1. Solution pre-selection

In crowdsourcing systems, innovative tasks, unlike simple tasks, are complex and require specific skills, knowledge, and experience of participants [10 , 33]. Different participants with different levels of skill, reputation, and motivation may develop various solution varying from each other in terms of quality [15, 34]. For the solutions owning low quality may be proposed by unqualified or cheatable participants, they do not have great values for requesters. To improve the efficiency of solution selection, it is a feasible way to conduct per-selection initially that aims to screen solutions meeting minimum requirements.

Currently, basic screening rules of compensatory, non-compensatory, and lexicographical, are commonly applied to deal with the problem of alternatives ranking and selection such as supplier selection [35 –38]. In the first case, decision making units (DMUs) are assessed against each criterion. A bad performance of a DMU on a specific criterion can be compensated by high performance on other criteria, and therefore its overall performance will reflect this. On the contrary, decisions making on the basis of non-compensatory rule assumes that no compensation for a DMU that does not satisfy a minimum requirement of a specific criterion. In the third case, the criteria for DMUs evaluation are first ranked, and the DMUs meeting the requirement of the most important criterion are selected to evaluate further.

In the phase of solution pre-selection in this paper, the main purpose is to screen the solutions meeting basic requirements proposed by requesters so they can be selected as potential solutions and moved on to the next phase. Base on the above view, the screening rule of non-compensatory is more appropriate than the other two rules to perform solution pre-selection. The solutions do not meet minimum requirement of a specific criterion will not be considered further. For an innovative task, the criteria used for solution pre-selection are basic and minimum demands which are obtained from requesters. These demands are commonly focus on the whole performance of each submitted solution.

3.2. Weighting of solution features

In this phase, a HoQ model for solution selection in crowdsourcing contests for innovative tasks is constructed at first. Then 2-tuple linguistic method and fuzzy weighted average method [35, 39] are used to determine the weight of each SF.

(1) Construct the HoQ model for the solution selection problem

The initial step of constructing the HoQ model in this paper is to identify RRs and SFs. In general, RRs always be published initially by requesters in a crowdsourcing contest along with expected duration and awards. It is noted that some other RRs may be added by requesters as time goes on. In this paper, these RRs with different priorities are collected and regarded as inputs in HoQ.

Afterwards, participants start to design or develop their solutions according to RRs from the most importance to the least. To a certain extent, the features of a submitted solution are derived from RRs, and RRs in turn are reflected by these features. On the basis of the view, this paper tries to identify SFs as follows: first, requesters collect all the potential solutions screened in the first phase, and extract features that most solutions possess. Second, some participants who have a good performance in performing the task are invited to collaborate with requesters to identify SFs as well, because they know the solutions well and may have a better understanding about SFs and their relations with RRs than requesters [40, 41]. By communicating and interviewing, requesters can acquire participants’ opinions on the SFs estimated initially and other added SFs. Finally, requesters summarize and sort out the SFs which are going to be viewed as criteria for solution evaluation.

According to the collected RRs and SFs, we present the HoQ matrix for the solution selection problem which is shown in Fig. 2. Therein, RRs ={ rs₁, rs₂, …, rs_i, …, rs_m } is a set of requesters’ requirements, and SFs ={ sf₁, sf₂, …, sf_j, …, sf_n } is the set of identified solution features.

Fig. 2.

The HoQ for the solution selection problem.

In Fig. 2, w_i (1 ≤ i ≤ m), v_ij, and t_lj (1 ≤ l, j ≤ n) denote the relative importance of rs_i, the relationship score between rs_i and sf_j, and the inner dependence value between sf_l and sf_j, respectively. In this paper, linguistic variables and 2-tuple linguistic method are used to represent, process, and aggregate subjective assessment information given by requesters. For the sake of simplicity, a uniform linguistic term set S ={ s₀, s₁, …, s_u } is defined for evaluation. It is noted that other multi-granularity linguistic term sets can also be used in this paper in order to reflect different backgrounds and expertise of requesters. For the multi-granularity linguistic term sets, they can be transferred into a defined linguistic term set by using a transformation function so as to aggregate all evaluation information and further obtain final evaluation results [17 , 42].

Subsequently, using linguistic terms in S, requesters are requested to evaluate the importance of each RR, the relationships between RRs and SFs, and inner dependencies among SFs. Regarding evaluation results represented by linguistic terms, we can transform them into 2-tuple linguistic variables by employing the transformation function (Δ). As a result, we can obtain evaluation results denoted as 2-tuple linguistic variables given by each requester.

Let $(w_{i}^{k}, α_{i}^{k})$ , $(v_{ij}^{k}, α_{ij}^{k})$ , and $(t_{lj}^{k}, α_{lj}^{k})$ be the evaluation results of the importance value of rs_i, the relationship score between rs_i and sf_j, and the inner dependence value between sf_l and sf_j provided by kth requester, where $w_{i}^{k}, v_{ij}^{k}, t_{lj}^{k} \in S$ and $α_{i}^{k}, α_{ij}^{k}, α_{lj}^{k} \in [- 0.5, 0.5)$ . So the final values of w_i, v_ij, and t_kj can be estimated by aggregating evaluation results given by all the requesters as follows. $w_{i} = (w_{i}^{'}, α_{i}^{'}) = Δ (\sum_{k = 1}^{k} (Δ^{- 1} (w_{i}^{k}, α_{i}^{k}) \times w_{k}))$ (6) $v_{ij} = (v_{ij}^{'}, α_{ij}^{'}) = Δ (\sum_{k = 1}^{k} (Δ^{- 1} (v_{ij}^{k}, α_{ij}^{k}) \times ω_{k}))$ (7) $t_{lj} = (t_{lj}^{'}, α_{lj}^{'}) = Δ (\sum_{k = 1}^{k} (Δ^{- 1} (t_{lj}^{k}, α_{lj}^{k}) \times ω_{k}))$ (8)

Where K is the number of requesters who involving in solution assessment and selection. ω_k is the weight assigned to kth requester, and 0 ≤ ω_k ≤ 1, $\sum_{k = 1}^{K} ω_{k} = 1$ ., $(v_{ij}^{'}, α_{ij}^{'})$ , and $(t_{lj}^{'}, α_{lj}^{'})$ are 2-tuple linguistic variables representing the importance value of rs_i, the relationship score between rs_i and sf_j, and the inner dependence value between sf_l and sf_j, where and.

(2) Determine the weights of solution features

The weight of each SF is impacted and determined by the importance values of RRs, the inner dependencies of SFs, and the relationship measures between RRs and SFs. Since they are represented by 2-tuple linguistic variables, 2-tuple linguistic weighted average operator [17, 27] incorporating fuzzy weighted average (FWA) [35 , 44] are employed to compute the overall weight of each SF.

Firstly, the relationships between RRs and SFs should be modified and normalized by multiplying the inner dependencies among SFs [17, 43], i.e. $v_{ij}^{*} = Δ (\frac{\sum_{l = 1}^{n} (Δ^{- 1} (v_{il}^{'}, α_{lj}^{'}) Δ^{- 1} (t_{lj}^{'}, α_{lj}^{'}))}{\sum_{j = 1}^{n} \sum_{l = 1}^{n} (Δ^{- 1} (v_{il}^{'}, α_{lj}^{'}) Δ^{- 1} (t_{lj}^{'}, α_{lj}^{'}))})$ (9)

Where $v_{ij}^{*}$ is the normalized relationship value between rs_i and sf_j, 1 ≤ l, j ≤ n; 1 ≤ i ≤ m.

Benefiting from Equation (9), the weight of each SF can be computed as follows [17, 44]: $y_{j} = Δ (\sum_{i = 1}^{m} w_{i} v_{ij}^{*} / - \sum_{i = 1}^{m} w_{i})$ (10)

Where y_j is the weight of sf_j.

3.3. Solution evaluation and prioritization

This phase will firstly assess the performance of each potential solution with regard to each SF, and further determine their overall performance and prioritize them.

(1) Assess the performance of potential solutions

Suppose PS ={ ps₁, ps₂, …, ps_g, …, ps_G } be the set of potential solutions, where G is the number of solutions. For each potential solution, requesters are required to evaluate the performance of it against each SF. For simplicity, linguistic terms in S is provide to requesters to evaluate each potential solution. Consequently, the assessment results given by requesters can be characterized as follows: $p_{gj} = (p_{gj}^{'}, α_{gj}^{'}) = Δ (\sum_{k = 1}^{K} (Δ^{- 1} (p_{gj}^{k}, α_{gj}^{k}) \times ω_{k}))$ (11)

Where $p_{gj}^{k}$ is the performance of ps_g with respect to sf_j assessed by kth requester. p_gj is the performance of ps_g with respect to sf_j by aggregating all the requesters evaluation results, where $p_{gj}^{'} \in S$ and $α_{gj}^{'} \in [- 0.5, 0.5)$ .

(2) Prioritize and select solutions

Through 2-tuple linguistic weighted average operator, we can estimate the overall performance of each potential solution by aggregating the local performance of it with respect to SFs and the local weights of SFs, i.e. $p_{g} = Δ (\sum_{j = 1}^{n} p_{gj} y_{j})$ (12)

Where p_g is the overall performance of ps_g.

Then we can prioritize all solutions according to their evaluation values. Since only one solution in crowdsourcing contests can be selected as the win one, the solution with the highest value is selected on the basis of the rules of 2-tuples comparison to carry out the task.

4. Illustrative application case

In order to demonstrate implementation procedures of the proposed methodology and its feasibility, this section will briefly introduce a case of solution selection in crowdsourcing contests for innovative tasks. The case is conducted on Taskcn, the most widely used Chinese online crowdsourcing market that was founded in 2006. On Taskcn, innovative tasks, such as logo design, website design, article editing, and advertisement design, are important and major tasks posted by requesters and contributed to by participants. By the end of July 29, 2017, Taskcn has accumulated 62,679 tasks, with rewards totaling CNY 40,963,920. Additionally, there are 3,657,540 registered workers on Taskcn, and about 273,108 workers have won at least a reward.

The innovative task in the case is a logo design for a bilingual kindergarten named “zhisheng” which was published on Taskcn. The requirements of the task are the designed logo should reflect characteristics of modern kindergarten (rs₁), logo’s color and shape should satisfy preferences of children who ages are from two to six years old (rs₂), embody meanings of the kindergarten’s name, i.e. gain knowledge every day and grow up happily every day (rs₃), original (rs₄), concise and full of energy and hope (rs₅). Furthermore, the reward of the task is CNY 300, and interested participants are asked to submit their solutions in 21 days. Finally, there are 2122 workers browse the task, and 35 solutions are received.

Regarding to the received solutions, we adopt the proposed methodology in this paper to evaluate them and further select the most suitable one. The procedures of selection are given below.

Step 1. Pre-selection. The aim of per-selection is to screen the solutions that meet the minimum requirements. In this section, the five requirements mentioned above are taken as criteria for screening. According to the screening rule of compensatory, the solutions which do not meet the minimum demand of any criterion will be removed, and those solutions will be moved on to the next phase if they meet the minimum demand of each criterion. For example, in the following Fig. 3, of the four submitted solutions, the first solution does not satisfy requirement rs₁ because it’s hard to judge whether the logo represents a kindergarten directly or not. The second solution is deleted because it is not a logo and do not have other information requested by requesters. The third is so complex that does not meet requirement of rs₅. The last does not reflect kindergarten’s theme and name.

Fig. 3.

The examples of deleted solutions.

Through pre-selection, as a result, there are 14 solutions do not satisfy the minimum requirement of one or more criteria, and will not be considered further. The rest 21 solutions, which denoted as {ps₁, ps₂, …, ps₂₁}, are moved to the next phase.

Step 2. Identifying solution features. Regarding to the remainder solutions, requesters browse each solution and communicate with each other, and discuss with some participants to identify solution features which are used to evaluate each solution. Consequently, eight features are identified and presented as: precise expression of kindergarten’s name (sf₁), it’s a logo of the kindergarten (sf₂), reflect the feature of bilingual teaching (sf₃), easy to understand and recognize (sf₄), key information embedding in the logo matches with children characteristics, i.e. innocent, hope, and lovely (sf₅), color matches (sf₆), shape and size of logo and text (sf₇), the position of logo and text (sf₈).

Step 3. Constructing the HoQ matrix. According to the determined requirements and features in the aforementioned two steps, we construct the HoQ matrix including part 1, part 2, part 3, and part 4 for solution selection problem in this case which is shown in Fig. 4.

Fig. 4.

The HoQ model and evaluation results in the case.

For the sake of simplicity, a linguistic term set with seven terms, i.e. S = {s₀ : Null, s₀ : Very Low, s₂ : Low, s₃ : Medium, s₄ : High, s₅ : Very High, s₆ : Total}, is presented for three requesters to assess the importance of RRs, the inner dependencies among SFs, the relationship between each RR and each SF. In practice, with ever increasing of amount of solutions and their features, it is hard for requesters to indicate precisely the linguistic evaluation information on the above three aspects using the given linguistic term set in advance. The numerical value α provides flexibility for requesters to applying the linguistic terms. For example, a requester may think that the relationship between sf₁ and sf₂ may be very high (the defined linguistic term is s₅) but less than s₆, and thus their inner dependence value can be presented using 2-tuple linguistic variable (s₅, 0.2). Furthermore, it is noted that the inner dependence between two features is symmetric. A feature has the strongest dependence on itself, their dependence value is (s₆, 0). If there is no dependence between lth and jth feature, their dependence value would be (s₀, 0).

For instance, the evaluation outcomes given by the three requesters about the importance value of rs₁, the relationship value between rs₁ and sf₁, and the inner dependence between sf₁ and sf₂ are ((s₅, 0.2), (s₅, 0), (s₄, 0.3)), ((s₅, 0), (s₄, 0.2), (s₅, 0)), and ((s₆, 0), (s₅, 0.3), (s₄, 0)), respectively. In this paper, we assume that the weights of the three requesters are 0.4, 0.3, and 0.3. By using Equations (6–8), we can obtain aggregation results for w₁, v₁₁, and t₁₂: $\begin{matrix} w_{1} = Δ (0.4 \times Δ^{- 1} (s_{5}, 0.2) + 0.3 \times Δ^{- 1} (s_{5}, 0) \\ + 0.3 \times Δ^{- 1} (s_{6}, 0)) = Δ (5.38) = (s_{5}, 0.38) \\ v_{11} = Δ (0.4 \times Δ^{- 1} (s_{5}, 0) + 0.3 \times Δ^{- 1} (s_{4}, 0.2) \\ + 0.3 \times Δ^{- 1} (s_{5}, 0.3)) = Δ (4.85) = (s_{5}, - 0.15) \\ t_{12} = Δ (0.4 \times Δ^{- 1} (s_{4}, 0.3) + 0.3 \times Δ^{- 1} (s_{5}, 0) \\ + 0.3 \times Δ^{- 1} (s_{4}, 0)) = Δ (4.42) = (s_{4}, 0.42) \end{matrix}$ (13)

Meanwhile, we can also determine the importance of five RRs, the inner dependencies among eight SFs, and their relationship values. All the computation results are presented in Fig. 4.

Step 4. Determining the weights of solution features. Based on the original relationship values between RRs and SFs, and the inner dependencies scores among SFs in Fig. 4, we can obtain the modified and normalized relationship value between each RR and SF by using Equation (9). The normalized values between the five RRs and eight SFs are shown in the following matrix.

${(v_{ij}^{*})}_{5 \times 8} = [\begin{matrix} Δ (0.090) & Δ (0.107) & Δ (0.110) & Δ (0.186) & Δ (0.170) & Δ (0.120) & Δ (0.109) & Δ (0.107) \\ Δ (0.067) & Δ (0.089) & Δ (0.106) & Δ (0.184) & Δ (0.176) & Δ (0.137) & Δ (0.125) & Δ (0.116) \\ Δ (0.076) & Δ (0.094) & Δ (0.104) & Δ (0.181) & Δ (0.170) & Δ (0.133) & Δ (0.125) & Δ (0.116) \\ Δ (0.064) & Δ (0.090) & Δ (0.119) & Δ (0.183) & Δ (0.179) & Δ (0.130) & Δ (0.114) & Δ (0.120) \\ Δ (0.069) & Δ (0.088) & Δ (0.108) & Δ (0.182) & Δ (0.175) & Δ (0.135) & Δ (0.124) & Δ (0.119) \end{matrix}]$

Benefiting from the above matrix, the weight of each feature can be calculated incorporating the importance values of RRs. For example, by employing Equation (10), the weight of sf₁ is computed as follow:

$y_{1} = Δ (\frac{0.090 \times 5.38 + 0.067 \times 5.49 + 0.076 \times 5.46 + 0.064 \times 4.06 + 0.069 \times 5.30}{5.38 + 5.49 + 5.46 + 4.06 + 5.30}) = Δ (0.074)$

Also, the weights of the other seven features can be got, i.e. Δ (0.094), Δ (0.109), Δ (0.183), Δ (0.174), Δ (0.131), Δ (0.120), Δ (0.115), respectively. They are presented in part 4 in Fig. 4.

Step 5. Assessing the performance of each potential solution with respect to each feature. For the rest 21 potential solutions, the three requesters assess the performance of each of them with respect to SFs using linguistic terms in the set S. Then, Equation (11) is employed to aggregate requesters’ evaluation results, and the final results are presented in part 5 in Fig. 4.

Step 6. Determining the priorities of the potential solutions. On the basis of the local performance of each solution with respect to SFs and the weights of SFs, we can determine the overall performance of each potential solution. For example, as shown in Fig. 4, the overall performance of solution ps₁ is computed using Equation (12) as follow: $\begin{matrix} p_{1} = Δ (0.074 \times 4.7 + 0.094 \times 3.6 + 0.109 \times 2.6 \\ + 0.183 \times 3.4 + 0.174 \times 3.7 + 0.131 \times 3 \\ + 0.120 \times 4 + 0.115 \times 4.4) = Δ (3.615) \end{matrix}$

In the meantime, the performance scores of other solutions can be obtained and presented in part 6 in Fig. 4.

Step 7. Ranking and selection. According to the evaluation results in part 6 in Fig. 4, we can rank 21 solutions as ps₃ ≻ ps₁₁ ≻ ps₁₆ ≻ ps₁₉ ≻ ps₉ ≻ ps₁₂ ≻ ps₈ ≻ ps₂₀ ≻ ps₂ ≻ ps₆ ≻ ps₁₄ ≻ ps₁ ≻ ps₇ ≻ ps₁₅ ≻ ps₁₇ ≻ ps₁₈ ≻ ps₄ ≻ ps₁₀ ≻ ps₁₃ ≻ ps₂₁ ≻ ps₅, where the symbol “≻” means superior to. Finally, the solution ps₃ is selected as the most suitable solution to carry out the task.

5. Discussion

In this section, we compare the proposal in this paper with other related studies in order to demonstrate the advantages and disadvantages in our proposal.

Essentially, solution selection in crowdsourcing contests for innovative tasks belongs to multi-criteria decision making (MCDM) problems. The method of AHP or other methods combining AHP are the most widely used approach. For example, Kilincci et al. [45] proposed fuzzy AHP to address supplier selection problem in a washing machine company. Hadi-Vencheh et al. [26] developed an integrated method of AHP-DEA to deal with the problem of multi-criteria ABC inventory classification. Therein, AHP is applied to determine the weights of criteria, and DEA is employed to determine the values of the linguistic terms which are used to assess each item under each criterion. Ju et al. [46] evaluated emergency response capacity by fuzzy AHP and 2-tuple linguistic method, and Santos et al. [47] segmented suppliers depending on a model based on 2-tuple linguistic representation and AHP. In these two studies, 2-tuple linguistic variables are used to represent experts’ opinions and AHP is used to determine the weights of criteria and overall score of each item. Conducting pairwise comparisons is an imperative step when implementing the methods mentioned above. However, obtaining pairwise comparisons in this paper may become quite complex because the number of submitted solution in a contest is large. Moreover, to assist requesters to select the solution which meets their requirements, this study takes RRs, the inner dependencies among SFs, and the relationship between RRs and SFs into account. It may lead to the AHP method or other approaches, such as fuzzy AHP, fuzzy TOPSIS, and fuzzy VIKOR, that assume the mutual independencies among criteria may not suitable to determine the weights of SFs properly.

The QFD employed in this paper is common and widely used method connecting customer demands/requirements and design alternatives/requirements. A lot of studies focus on scoring customer requirements, measuring design requirements, and determining the relationships between customer requirements and design requirements by combining QFD with other techniques. For example, integrating QFD and DEA to measure the relative importance of design alternatives or design requirements aiming to satisfy customer requirements and other additional factors, such as cost, environment, and producer constraints [23 , 48]. The proposed approaches in these studies offer valuable insights to determine the scores of RRs, SFs, and their relationships in this paper. However, they mainly are used to handle the information in QFD, but cannot used to determine the priorities of solutions with a large number. Furthermore, to our knowledge, the QFD also are integrated with other methods such as AHP [42 , 49], TOPSIS [50, 51], ANP [52, 53], and DEA [21, 54], etc. to address MCDM problems. As previously mentioned, the AHP, ANP methods which require to carrying out pairwise comparisons to derive the weights of criteria or scores of alternatives cannot be applied to deal with the problem of solution selection in this paper.

To the best of our knowledge, the proposed methodology in [21, 22] is similar, to some extent, to the QFD and 2-tuple linguistic method developed in this paper. These two studies all employed QFD to construct the relationship between customer needs and supplier assessment criteria and their inner dependencies. In [21], an imprecise DEA is used to compute the lower and upper bounds of weights of supplier assessment criteria based on evaluation information in the QFD. In [22], multi-granularity linguistic term sets and 2-tuple linguistic method is used to represent and aggregate experts’ evaluation information presented in QFD to determine the weights of supplier assessment criteria and suppliers priorities. Comparing with the two studies, our research differs from them in the following aspects: (i) The methodology in this paper includes pre-selection phase considering the practical condition of the large number of submitted solutions. Pre-selection plays an important role in the process of solution selection and has several merits: first, reducing the risk of selecting some unqualified solutions in the pool of potential solutions. Second, reducing the workload, time, and energy the requesters must spend to evaluate and integrate information about the performance of each solution with respect to solution features. Third, reducing the risk of neglecting a few qualified solutions in the evaluation and selection phase. By screening each submitted solution with requester requirements in a systematic and transparent way, requesters may not exclude some solutions intuitively and directly and decrease the chance of losing good potential solutions. (ii) For the methodology developed in [21], there is a need to solve the amount of LP models that is equal to the number of alternative users. Consequently, it is a heavy workload of calculation when the number of alternative users is large. On the contrary, our methodology is simple, easy to use, and applicable to complex MCDM problems with a large number of alternatives. (iii) Unlike Karsak and Dursun’s approach adopts multi-granularity linguistic term sets to express various opinions of experts who may vary from each other in cultural and educational background, this study introduces 2-tuple linguistic variables (s_i, α_i), which s_i denote linguistic terms and α_i ∈ [- 0.5, 0.5) represent uncertainty degrees of requesters in qualifying a phenomenon using the given linguistic term set in advance. It may enable requesters to assess and calculate more easily without transforming multi-granularity linguistic term set into a uniform linguistic term set. (IV) Linguistic terms are all used in these two studies to present decision makers expressions. Karsak et al. [22] transform them into triangular fuzzy numbers and then 2-tuple linguistic variables to deal with them, which possibly result in information loss and distortions to some degree. This study utilizes 2-tuple linguistic weighted average operator to aggregate linguistic evaluation information, which does not require such transformation between linguistic terms and fuzzy numbers.

Not unlike any modeling approach, the methodology put forward in this paper exists the following disadvantages:

In this paper, we prioritize potential solutions and select one with a total highest score. However, the performance of the selected solution is unlikely to have the highest scores on all the features. The future study will focus on extracting the highlights of each submitted solution and combining them together to form a new solution for requesters.

During the process of solution selection, more than one requester may involve in assessing solution performance, the importance of requirements and their relations with solution features. Under the situation, non-cooperative behaviors exhibiting by requesters probably occur due to their different interests and preferences [55, 56]. This study aggregates requesters’ evaluation information representing by linguistic variables by using 2-tuple linguistic weighted average operator but does not take non-cooperative behaviors by requesters into account.

In section of discussion, we do not provide the illustrative examples and numerical analysis to verify the merits and demerit of our proposal comparing with other related studies because the structures and procedures used in different approaches are different. This paper only demonstrates advantages and disadvantages of our methodology comparing with some other MCDM approaches. Future study will try to develop some criteria to compare the proposed approach with other related methods.

6. Conclusion

In crowdsourcing contests for innovative tasks, selecting the right solutions plays an important role in the process of crowdsourcing tasks for requesters. This paper develops a methodology with three phases for solution selection. The first phase is to screen the solutions meeting the minimum requirements. The second phase is to determine the weights of solution features using QFD, 2-tuple linguistic method, and fuzzy weighted average. The last phase is to evaluate the performance of potential solutions and aggregate them with the weights of solution features. To summary, the proposed methodology in this paper enables to incorporate the relationships between requester’s requirements and solution features, and also the inner correlations among solution features in determining the weights of solution features, which is helpful for achieving higher satisfaction to meet requester’s requirements. It also utilizes the advantages of 2-tuple fuzzy linguistic representation to capture human thinking style and to express requester’s vague and subjective opinions. The 2-tuple linguistic weighted average operator is used to aggregate fuzzy linguistic evaluation information avoiding the problem of loss of information. Furthermore, implementation process and results of the illustrate case show that the methodology including three phases proposed in this paper is easy to use, simple, and suitable for requesters to seek the best solution from any number of submitted solutions in crowdsourcing contests for innovative tasks.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 71802002), Key Project of Academic Humanities and Social Science of Anhui Education Department (No. SK2017A0120), the Science Research Starting Foundation of Anhui Polytechnic University for Talent Introduction (No. 2016YQQ008), and the Scientific and Technology Research Program of Chongqing Municipal Education Commission, China (No. KJQN201800832). The author would like to thank the anonymous reviewers and the editor for their constructive comments and suggestions.

References

N.H.

Thuan ,

Antunes and

Johnstone , Factors influencing the decision to crowdsource: A systematic literature review, Information Systems Frontiers 18(1) 2016, 47–68.

C.K.M.

Lee ,

C.Y.

Chan ,

Ho ,

K.L.

Choy and

W.H.

Ip , Explore the feasibility of adopting crowdsourcing for innovative problem solving, Industrial Management & Data Systems, 115(5) (2015), 803–832.

M.K.

Poetz and

Schreier , The value of crowdsourcing: Can users really compete with professionals in generating new product Ideas? Journal of Product Innovation Management 29(2) (2012), 245–256.

T.X.

Liu ,

Yang ,

L.A.

Adamic and

Chen , Crowdsourcing with all-pay auctions: A field experiment on taskcn, Management Science 60(8) (2014), 2020–2037.

D.C.

Brabham , Motivations for participation in a crowd-sourcing application to improve public engagement in transit planning, Journal of Applied Communication Research 40(3) (2012), 307–328.

D.C.

Brabham , Moving the crowd at threadless, Information, Communication & Society 13(8) (2010), 1122–1145.

Schenk and

Guittard , Towards a characterization of crowdsourcing practices, Journal of Innovation Economics 7(1) (2011), 93–107.

Luz ,

Silva and

Novais , A survey of task-oriented crowdsourcing, Artificial Intelligence Review 44(2) (2015), 187–213.

Kittur ,

J.V.

Nickerson ,

Bernstein ,

Gerber ,

Shaw ,

Zimmerman , et al., The future of crowd work, Proceedings of the 2013 Conference on Computer Supported Cooperative Work, 2013, pp. 1301–1318.

10.

Gong , Estimating participants for knowledge-intensive tasks in a network of crowdsourcing marketplaces, Information Systems Frontiers 19(2) (2017), 301–319.

11.

P.G.

Ipeirotis ,

Provost and

Wang , Quality management on Amazon Mechanical Turk, Proceedings of the ACM SIGKDD Workshop on Human Computation, 2010, pp. 64–67.

12.

Le ,

Edmonds ,

Hester and

Biewald , Ensuring quality in crowdsourced search relevance evaluation, In Proceedings of the ACM SIGIR 2010 Workshop on Crowd-sourcing for Search Evaluation, 2010, pp. 17–20.

13.

Oleson ,

Sorokin ,

Laughlin ,

Hester ,

Le and

Biewald , Programmatic gold: Targeted and scalable quality assurance in crowdsourcing, Proceedings of the 11th AAAI Conference on Human Computation, 2011, pp. 43–48.

14.

Burmania ,

Parthasarathy and

Busso , Increasing the reliability of crowdsourcing evaluations using online quality assessment, IEEE Transactions on Affective Computing 7(4) (2016), 374–388.

15.

Basu Roy,

Lykourentzou ,

Thirumuruganathan ,

Amer-Yahia and

Das , Task assignment optimization in knowledge-intensive crowdsourcing, The VLDB Journal 24(4) (2015), 467–491.

16.

Herrera-Viedma , Modeling the retrieval process for an information retrieval system using an ordinal fuzzy linguistic approach, Journal of the American Society for Information Science & Technology 52(6) (2001), 460–475.

17.

Herrera and

Martinez , A 2-tuple fuzzy linguistic representation model for computing with words, IEEE Transactions on Fuzzy Systems 8(6) (2000), 746–752.

18.

Herrera-Viedma ,

Herrera ,

Martinez ,

J.C.

Herrera and

A.G.L.

Pez , Incorporating filtering techniques in afuzzy linguistic multi-agent model for information gathering on the web, Fuzzy Sets and Systems 148(1) (2004), 61–83.

19.

Al-Mashari ,

Zairi and

Ginn , Key enablers for the effective implementation of QFD: A critical analysis, Industrial Management & Data Systems 105(9) (2005), 1245–1260.

20.

Bouchereau and

Rowlands , Methods and techniques to help quality function deployment (QFD), Benchmarking An International Journal 7(1) (2000), 8–20.

21.

E.E.

Karsak and

Dursun , An integrated supplier selection methodology incorporating QFD and DEA with imprecise data, Expert Systems with Applications 41(16) (2014), 6995–7004.

22.

E.E.

Karsak and

Dursun , An integrated fuzzy MCDM approach for supplier evaluation and selection, Computers & Industrial Engineering 82 (2015), 82–93.

23.

Ramanathan and

Yunfeng , Incorporating cost and environmental factors in quality function deployment using data envelopment analysis, Omega 37(3) (2009), 711–723.

24.

C.H.

Wang and

H.S.

Wu , A novel framework to evaluate programmable logic controllers: A fuzzy MCDM perspective, Journal of Intelligent Manufacturing 27(2) (2016), 315–324.

25.

L.A.

Zadeh , The concept of a linguistic variable and its application to approximate reasoning, Information Sciences 8(3) (1975), 199–249.

26.

Hadi-Vencheh and

Mohamadghasemi , A fuzzy AHP-DEA approach for multiple criteria ABC inventory classification, Expert Systems with Applications 38(4) (2011), 3346–3352.

27.

Porcel ,

A.G.L.

Pez-Herrera and

Herrera-Viedma , A recommender system for research resources based on fuzzy linguistic modeling, Expert Systems with Applications 36(3) (2009), 5173–5183.

28.

Herrera-Viedma ,

Pasi ,

A.G.

Lopez-Herrera and

Porcel , Evaluating the information quality of Web sites: A methodology based on fuzzy computing with words, Journal of the American Society for Information Science & Technology 57(4) (2006), 538–549.

29.

Zhang ,

Dong and

Chen , The 2-Rank Consensus Reaching Model in the Multigranular Linguistic Multiple-Attribute Group Decision-Making, IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018, pp. 1–15.

30.

S.-P.

Wan , 2-Tuple linguistic hybrid arithmetic aggregation operators and application to multi-attribute group decision making, Knowledge-Based Systems 45 (2013), 31–40.

31.

Zhang ,

Wang and

Zhao , A new emergency decision support methodology based on multi-source knowledge in 2-tuple linguistic model, Knowledge-Based Systems 144 (2018), 77–87.

32.

Satzger ,

Psaier ,

Schall and

Dustdar , Auction-based crowdsourcing supporting skill management, Information Systems 38(4) (2013), 547–560.

33.

Terwiesch and

Xu , Innovation contests, open innovation, and multiagent problem solving, Management Science 54(9) (2008), 1529–1543.

34.

Cheng ,

Lian ,

Chen ,

Han and

Zhao , Task assignment on multi-skill oriented spatial crowdsourcing, IEEE Transactions on Knowledge & Data Engineering 28(8) (2016), 2201–2215.

35.

Chen ,

R.Y.K.

Fung and

Tang , Rating technical attributes in fuzzy QFD by integrating fuzzy weighted average method and fuzzy expected value operator, European Journal of Operational Research 174(3) (2006), 1553–1566.

36.

F.R.

Lima ,

Osiro and

L.C.R.

Carpinetti , A fuzzy inference and categorization approach for supplier selection using compensatory and non-compensatory decision rules, Applied Soft Computing 13(10) (2013), 4133–4147.

37.

Rezaei ,

Nispeling ,

Sarkis and

Tavasszy , A supplier selection life cycle approach integrating traditional and environmental criteria using the best worst method, Journal of Cleaner Production 135 (2016), 577–588.

38.

Dong ,

Wu ,

Zhang and

Zhang , Multi-granular unbalanced linguistic distribution assessments with interval symbolic proportions, Knowledge-Based Systems 82 (2015), 139–151.

39.

Chen ,

R.Y.K.

Fung and

Tang , Fuzzy expected value modelling approach for determining target values of engineering characteristics in QFD, International Journal of Production Research 43(17) (2005), 3583–3604.

40.

Kittur ,

Smus and

Kraut , CrowdForge: Crowdsourc-ing complex work, CHI '11 Extended Abstracts on Human Factors in Computing Systems, 2011, pp. 1801–1806.

41.

Kulkarni ,

Can and

Hartmann , Collaboratively crowdsourcing workflows with turkomatic, Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, 2012, pp. 1003–1012.

42.

Chadawada ,

Sarfaraz ,

Jenab and

Pourmo-hammadi , Integration of AHP-QFD for selecting facility location, Benchmarking An International Journal 22(3) (2015), 411–425.

43.

Bhattacharya ,

Geraghty and

Young , Supplier selection paradigm: An integrated hierarchical QFD methodology under multiple-criteria environment, Applied Soft Computing 10(4) (2010), 1013–1027.

44.

Zare Mehrjerdi ,

M.S.

Owlia and

Tanha , Evaluation and ranking the relative importance of design requirements by combining QFD and DEA techniques (case study: Tile industry of Iran), International Journal of Industiral Engineering & Producion Management 23(2) (2012), 175–186.

45.

Kilincci and

S.A.

Onal , Fuzzy AHP approach for supplier selection in a washing machine company, Expert Systems with Applications 38(8) (2011), 9656–9664.

46.

Ju ,

Wang and

Liu , Evaluating emergency response capacity by fuzzy AHP and 2-tuple fuzzy linguistic approach, Expert Systems with Applications 39(8) (2012), 6972–6981.

47.

L.F.D.O.M.

Santos ,

Osiro and

R.H.P.

Lima , A model based on 2-tuple fuzzy linguistic representation and Analytic Hierarchy Process for supplier segmentation using qualitative and quantitative criteria, Expert Systems with Applications 79 (2017), 53–64.

48.

W.-C.

Ko , Construction of house of quality for new product planning: A 2-tuple fuzzy linguistic approach, Computers in Industry 73 (2015), 117–127.

49.

Dai and

Blackhurst , A four-phase AHP-QFD approach for supplier assessment: A sustainability perspective, International Journal of Production Research 50(19) (2012), 5474–5490.

50.

Akbas and

Bilgen , An integrated fuzzy QFD and TOP-SIS methodology for choosing the ideal gas fuel at WWTPs, Energy 125 (2017), 484–497.

51.

A.H.

Kumaraswamy ,

Bhattacharya ,

Kumar and

Brady , An Integrated QFD-TOPSIS Methodology for Supplier Selection in SMEs, 2011 Third International Conference on Computational Intelligence, Modelling & Simulation, 2011, pp. 271–276.

52.

Bottani ,

Centobelli ,

Murino and

Shekarian , A QFD-ANP method for supplier selection with benefits, opportunities, costs and risks considerations, International Journal of Information Technology & Decision Making 17(3) (2018), 911–939.

53.

N.A.

Tourzani ,

Keramati and

Apornak , Supplier selection model using QFD-ANP methodology under fuzzy multi-criteria environment, International Journal of Productivity & Quality Management 24(1) (2018), 59–83.

54.

Zhang , User selection for collaboration in product development based on QFD and DEA approach, Journal of Intelligent Manufacturing (2017). doi: 10.1007/s10845-017-1386-3

55.

Zhang ,

Palomares ,

Dong and

Wang , Managing non-cooperative behaviors in consensus-based multiple attribute group decision making: An approach based on social network analysis, Knowledge-Based Systems (2018). doi: 10.1016/j.knosys.2018.06.008

56.

Dong ,

Zhao ,

Zhang ,

Chiclana and

Herrera-Viedma , A self-management mechanism for non-cooperative behaviors in large-scale group consensus reaching processes, IEEE Transactions on Fuzzy Systems (2018), 1–1. doi: 10.1109/tfuzz.2018.2818078