A Modified Pugh Matrix Method for Continuous Performance Evaluation of Employees

Abstract

Although researchers recommend the continuous performance evaluation of employees (CPEE), literature is scant about a procedural framework for CPEE. This article aims to develop a simple and robust method for CPEE using the modified Pugh matrix method (MPMM). The criteria for CPEE are compared pairwise using the analytical hierarchy process (AHP), followed by a comparison of employees using MPMM to determine and rank the employees based on their performance. The proposed method is successfully validated in an organisation, and the working mechanism of MPMM is demonstrated. Implications of the proposed method for CPEE and the scope for future work are also discussed.

Keywords

Performance appraisal system continuous performance evaluation of employees multi-criteria decision-making modified Pugh matrix method analytical hierarchy process

Introduction

Performance of organisations chiefly depends on the performance of their employees. Employee performance can be enhanced only by continuously evaluating and calibrating their performance (Boice & Kleiner, 1997). The performance of employees is typically assessed using a performance appraisal system (PAS). The PAS is a confidential system that managers use to evaluate the performance of employees once a year (Aguinis et al., 2011). Rather than waiting a year to learn about their performance, employees value timely information about their current performance (Fawcett, 2015; Jawahar, 2006). Therefore, there is a necessity for a method to facilitate continuous employee performance evaluation (CPEE). Besides, the criticism related to the subjectivity with PAS (Grund & Przemeck, 2012; Kleiman et al., 1987) can be addressed by bringing in a series of evaluations that result in objective (quantitative) scores. These scores from CPEE could act as a reliable input to the existing PAS.

Various criteria determine the overall performance of the employees. An effective CPEE process should incorporate all these criteria. Researchers have proposed utilising the multi-criteria decision-making (MCDM) method for employee performance evaluation for PAS (Haddad et al., 2019; Shaout & Yousif, 2014). Although there is literature about using MCDM for the annual employee performance appraisal, literature on MCDM, specifically for CPEE, appears virtually non-existent. This article explores the efficacy of a relatively lesser known MCDM method—the Pugh matrix method (PMM)—for CPEE.

In the remaining part of the article, the existing literature on using MCDM for the annual appraisal process is reviewed, and PMM as a possible method for CPEE is proposed. A procedural framework for CPEE is explained, and the proposed method’s validation in an organisation is detailed. Finally, implications of the CPEE method are elaborated, and the scope for future research is also included.

Methods of Employee Performance Evaluation

Performance evaluation of employees has been an active area of research for the past few decades (Pichler et al., 2020). Most of the issues related to the practice of annual performance appraisal (Kondrasuk, 2011) can be addressed to an extent by reducing the frequency of the appraisals. Despite the importance researchers’ thrust towards developing the theory of CPEE and its benefits (Palaiologos et al., 2011; Rivera et al., 2021; Sreejith, 2015), CPEE is not known to be practised in organisations. The managers may fear that the CPEE is yet another exercise that takes a considerable time to execute (Aguinis et al., 2011). Besides, there is no known procedural framework for a formal CPEE.

There are informal performance evaluation methods carried out by managers (Biernat et al., 2010) with varying frequency. However, such informal evaluations are often highly subjective, random, non-transparent and non-informative (Komendat & Didona, 2016). Organisations across the globe are moving towards a continuous feedback mechanism (Tripathi et al., 2021). Many organisations such as Adobe, Cisco and Infosys adopt software- and mobile-based applications for offering timely feedback (Fawcett, 2015; Levit, 2019). One drawback of subjective-feedback-only methods is that they are hardly quantitatively recorded. In the long run, the feedback process and appraisal data may lack proper linkage, which handicaps the organisations from effective use of such methods. Organisations want a robust mechanism that keeps track of such frequent performance feedback that can be easily integrated with the organisation’s existing PAS. In this study, a formal, periodic, cyclical, transparent and simple process for CPEE is proposed.

Since the past two decades, various MCDM methods (as in Table 1) are being increasingly utilised for the annual performance evaluation of employees. Table 1 indicates that analytical hierarchy process (AHP), and its variations have been extensively used. Besides, popular outranking methods are almost always combined with AHP for employee performance evaluation. Yet, none of such MCDM methods are known to be used for CPEE. This article attempts to contribute to the literature by exploring the possibility of utilising a modified version of one such outranking MCDM method for CPEE.

Pugh Matrix Method

The PMM is an MCDM method proposed by Stuart Pugh (1981a). In PMM, one alternative is chosen as a baseline, and the remaining alternatives are compared with this baseline against each criteria using a linear arithmetic scale. On a given criterion, −1 implies if the alternative performs worse than the baseline, 0 implies same as the baseline and +1 implies better than the baseline. This is repeated for all criteria, and the sum of these scores is used to rank each alternative (Burge, 2009; Pugh, 1981b). Although simple, PMM is not known to be used widely as compared to other MCDM methods. This could be due to limitations such as the lack of differential weights for criteria, the scale being too short and a possible bias in choosing the baseline alternative (Mullur et al., 2003). These limitations are addressed in the proposed modified PMM (MPMM).

Table 1.

MCDM Methods Used for Performance Evaluation of Employees.

No.	Researcher(s)	MCDM Method	Industry	Purpose
1	Taylor et al. (1998)	AHP	Service	Selection and promotion
2	Hemaida and Everett (2003)	AHP	Service	Salary revision
3	Islam and bin Mohd Rasad (2006)	AHP	Service	Performance appraisal
4	Manoharan et al. (2009)	DEA	Manufacturing	Performance appraisal
5	Gürbüz (2010)	CI and MACHBETH	Manufacturing	Performance appraisal
6	Afshari et al. (2010)	ELECTRE	Manufacturing	Performance appraisal
7	Nader (2010)	Pugh Matrix Method (PMM)	Manufacturing	Evaluation of ideas from employees
8	Taufiq and Sugiharto (2011)	AHP	Service	Promotion
9	Ardabili (2011)	Fuzzy AHP, SAW, TOPSIS	Service	Performance appraisal
10	Jati (2012)	PROMETHEE	Service	Performance prediction and selection
11	Wu et al. (2012)	Fuzzy AHP and VIKOR	Service	Performance appraisal
12	Zbranek (2013)	DEA	Service	Performance efficiency
13	Islam et al. (2013)	Fuzzy AHP, TOPSIS	Service	Performance appraisal
14	Morte et al. (2013)	PROMETHEE and MMASSI	Service	Performance appraisal
15	Ahmed et al. (2013)	SAW, TOPSIS and VIKOR	Service	Performance appraisal
16	Gürbüz and Albayrak (2014)	ANP and CI	Manufacturing	Performance appraisal
17	Rahmati and Noorbehbahani (2017)	Fuzzy AHP and Fuzzy TOPSIS	Manufacturing	Performance appraisal
18	Ishizaka et al. (2018)	PROMETHEE	Service	Employee productivity
19	Tanti et al. (2018)	PROMETHEE	Service	Performance evaluation
20	Bongo et al. (2018)	DEMANTEL-ANP and PROMETHEE	Service	Job stress and performance
21	(Hsien Yu and Kuo Yan (2020)	AHP and VIKOR	Service	Employee promotion
22	Cholil et al. (2020)	ELECTRE	Service	Best employee selection
23	Kafabih and Budiyanto (2020)	Fuzzy-TOPSIS	Manufacturing	Salary increase and reward
24	Watrianthos et al. (2021)	POMETHEE	Service	Performance appraisal

Source: The authors.

The Proposed Modified PMM for CPEE

As with other MCDM methods, AHP can be used to determine the criteria weights in PMM. The evaluation scale can be extended to five points with a rating as follows: much worse than (−2), worse than (−1), equal (0), better than (+1) and much better than (+2). The limitation concerning the bias in baseline could be addressed by considering each of the alternatives as baseline in a rotational manner.

In the process of CPEE, the performance of each of the employees i (i = 1, 2, …, m employees) is compared among each other employees on every evaluation criteria j (j = 1, 2, …, n criteria) by making each of the k employee (k = 1, 2, …, m) as a baseline employee, one at a time. The sequence of the proposed MPMM is detailed:

Step 1: Identify the evaluation criteria for CPEE and determine the relative importance (weights) of each criteria using AHP.

Step 2: Using MPMM, obtain the pairwise performance ratings (PR _ij ) of each employee i (i = 1, 2, …, m) concerning each of the criterion j (j = 1, 2, …, n), comparing a baseline employee k using the following scale:

{PR}_{i j} = [\begin{matrix} + 2, if the ith employee performs much better than the baseline employee in criterion j \\ + 1, if the ith employee performs slightly better than the baseline employee in criterion j \\ 0, if the ith employee performs as good as the baseline employee in criterion j \\ - 1, if the ith employee performs slightly worse than the baseline employee in criterion j \\ - 2, if the i th employee performs much worse than the baseline employee in criterion j \end{matrix}]

The score of the baseline employee k will always be 0. The proposed MPMM for CPEE will have the pairwise numerical performance rating of employees (PR _ij ) as m × n matrix, considering k as baseline employee.

Step 3: Determine the normalised performance score (NPS _ik ) for each of the employees i (i = 1, 2, …, m), considering each of the k employee (k = 1, 2, …, m) as baseline employee.

Step 3.1: Compute performance score (PS _ik ) for each employee i (where i = 1, 2, …, m), considering k as baseline employee using the following Equation (1):

{PS}_{i k} = \sum_{j}^{n} (w_{j} \times {PR}_{i j}) for given k and for all i = 1, 2, \dots, m, j = 1, 2, \dots, n

(1)

This results in a column vector representing each employee’s performance For m employees’ performance score, there would be m iterations, considering each of the employees as a baseline employee to obtain PS _ik , resulting in an m × m matrix. Each column vector in the PS _ik matrix represents the performance score of each employee i (i = 1, 2, …, m), considering each employee k (k = 1, 2, …, m) as baseline.

Step 3.2: Eliminate the negative scores: the column elements are added with the absolute of the minimum value of that column vector.

Step 3.3: Normalise the scores (NPS _ik ) by dividing each element by the sum of the corresponding column vector.

Step 4: Determine the mean normalised performance score (MNPS _i ) by considering the row average, using Equation (2):

{MNPS}_{i} = \frac{\sum_{k}^{m} {NPS}_{i k}}{m} for all i = 1, 2, \dots, m

(2)

Based on the computed score on MNPS _i , the employees are ranked. This completes one round of evaluation.

Step 5: Repeat steps 2–4 for CPEE.

Validation of the Working Process of the Proposed MPMM for CPEE

The proposed MPMM is validated on a team of seven employees (including the manager) working in an Indian organisation. The following five-step approach is used:

Step 1: Identify evaluation criteria for CPEE and determine the relative importance (weights) of each criteria using AHP.

Six evaluation criteria: proactive, prompt, responsible, resourceful, diagnostic and dynamic are considered from the literature for CPEE. Sreejith and Mathirajan (2020) defined these six criteria for CPEE by conducting exploratory factor analysis among 26 latent variables, which were identified by carrying out a detailed descriptive study among 443 employees in the Indian IT organisations. The evaluation criteria with their corresponding latent variables are listed in Table 2.

The manager was asked to compare the criteria pairwise, and accordingly, weights were obtained using AHP using Saaty’s scale (Saaty, 1990) as presented in Table 3.

Table 2.

Evaluation Criteria and the Latent Variables for Each of the Criteria for CPEE.

Evaluation Criteria	Latent Variables Mapped Against Each Criterion
Proactive	Knowledge updation	Initiative	Self-learning	Leadership
Prompt	Timeline adherence	Timely reporting	Process adherence	Punctuality
Resourceful	Understanding big picture	Ideas and suggestions	Creativity	Cost savings
Responsible	Additional responsibilities	Knowledge sharing	Commitment	Teamwork	Mentoring	Improving morale
Diagnostic	Quality of the job	Documentation	Analytical ability	Reviewing	Presentation
Dynamic	Customer interaction	Communication	Negotiation

Source: The authors.

Table 3.

Pairwise Comparison of Evaluation Criteria using AHP.

	Proactive	Prompt	Responsible	Resourceful	Diagnostic	Dynamic
Proactive	1	0.33	0.5	1	4	2
Prompt	3	1	2	3	7	7
Responsible	2	0.50	1	2	3	5
Resourceful	1	0.33	0.50	1	5	3
Diagnostic	0.25	0.14	0.33	0.20	1	1
Dynamic	0.50	0.14	0.20	0.33	1	1

Source: The authors.

Several software are available to calculate the criteria weights concerning the goal defined in AHP (Ossadnik & Lange, 1999). Microsoft Excel (2016 version) was used in this study. This pairwise comparison resulted in criteria weights as indicated in Table 4.

Table 4.

Evaluation Criteria Weights Identified Using Pairwise Comparison Using AHP.

Evaluation Criteria	Proactive	Prompt	Responsible	Resourceful	Diagnostic	Dynamic
Weight (or importance) for CPEE	0.1334	0.3896	0.2266	0.1501	0.0479	0.0517

Source: The authors.

Step 2: Obtain pairwise numerical performance ratings (PR _ij ) of each of six employees i (i = 1, 2, …, 6) concerning each of the six criteria j (j = 1, 2, …, 6), comparing a baseline employee k.

Choosing the first employee in the list—Revathy—as baseline employee k, the manager performed a pairwise comparison of all other five employees with that of Revathy for all six criteria using the extended scale. The comparison is presented in Table 5.

Table 5.

Pairwise Numerical Performance Ratings of Six Employees for Six Criteria Using MPMM with one Employee (Revathy) as Baseline.

Employee (i)	Proactive	Prompt	Responsible	Resourceful	Diagnostic	Dynamic
Revathy* = k	0	0	0	0	0	0
Sajith	1	0	−1	1	2	1
Praseetha	0	−2	1	2	−1	−1
Divya	2	1	1	−1	−1	1
Adarsh	0	0	1	1	−1	0
Sumesh	1	−1	0	−2	0	1

Source: The authors.

Note: * Represents the baseline employee.

Step 3: Determine the NPS _ik for each of the employees i, considering each of the employee k as baseline employee.

Using Equation (1), the PS _ik for each employee i with Revathy as baseline is determined and presented in Table 6.

Table 6.

Performance Score of six employees with Revathy as the Baseline Employee.

Employee (i)	Performance Score (PS _i )
Revathy* = k	0.000
Sajith	−0.260
Praseetha	0.677
Divya	−1.091
Adarsh	−0.815
Sumesh	0.106

Source: The authors.

Note: * Represents the baseline employee.

With all other employees, each as baseline, six such scores are obtained and are combined in a 6 × 6 matrix as presented in Table 7.

Table 7.

Performance Scores for Employees When Each Employee Acts as Baseline Employee.

Employee (i)	Revathy	Sajith	Praseetha	Divya	Adarsh	Sumesh
Revathy	0.000	0.260	−0.677	1.091	0.815	−0.106
Sajith	−0.260	0	0.324	−0.842	−0.041	−1.054
Praseetha	0.677	−0.324	0	0.218	−0.047	0.327
Divya	−1.091	0.842	−0.218	0	0.31	−0.894
Adarsh	−0.815	0.041	0.047	−0.31	0	−0.075
Sumesh	0.106	1.054	−0.327	0.894	0.075	0

Source: The authors.

The negative signs can be eliminated using step 3.2. In the first column, each element is subtracted using the absolute of the minimum value of that column (1.091).

The performance scores in Table 7 will be modified into that, as presented in Table 8.

Table 8.

The Modified Performance Score of Each of the Employees When Each of the Employee is Considered as Baseline Employee.

Employee (i)	Revathy	Sajith	Praseetha	Divya	Adarsh	Sumesh
Revathy	1.091	0.584	0.000	1.951	0.862	0.948
Sajith	0.831	0.324	1.001	0.000	0.006	0.000
Praseetha	1.768	0.000	0.677	1.060	0.000	1.381
Divya	0.000	1.166	0.459	0.842	0.357	0.160
Adarsh	0.276	0.365	0.724	0.532	0.047	0.979
Sumesh	1.197	1.378	0.350	1.736	0.122	1.054

Source: The authors.

From Table 8, the sum of each column value is determined for each column, and the corresponding column elements are divided by this sum to arrive at an NPS _ik score for that column. This is repeated for every column that results in a combined NPS _ik for all employees. The computed NPS _ik is presented in Table 9.

Table 9.

The Normalised Performance Score of Each of the Employees When Each of the Employee is Considered as Baseline Employee.

Employee (i)	Revathy	Sajith	Praseetha	Divya	Adarsh	Sumesh
Revathy	0.211	0.153	0.000	0.319	0.618	0.210
Sajith	0.161	0.085	0.312	0.000	0.004	0.000
Praseetha	0.342	0.000	0.211	0.173	0.000	0.305
Divya	0.000	0.305	0.143	0.138	0.256	0.035
Adarsh	0.053	0.096	0.225	0.087	0.034	0.216
Sumesh	0.232	0.361	0.109	0.284	0.088	0.233

Source: The authors.

Step 4: The MNPS _i ) is computed by calculating the row average using Equation (2) (as shown in Table 10).

Table 10.

Performance Evaluation of Employees Based on the Mean Normalised Performance Score and Rank of a Particular Evaluation Period.

Employee (i)	Performance Evaluation Score of an Employee (i) and Their Ranking in the Team at a Particular Evaluation Period
Employee (i)	Mean Normalised Performance Score (MNPS _i )	The Rank of the Employee
Revathy	0.252	1
Sajith	0.094	6
Praseetha	0.172	3
Divya	0.146	4
Adarsh	0.119	5
Sumesh	0.218	2

Source: The authors.

From Table 10, Revathy has the highest score, while Sajith scores the lowest. This completes one round of CPEE.

Step 5: For a CPEE, steps 2–4 should be repeated at short but regular intervals. Three more repetitions were conducted weekly, resulting in a monthly score, as presented in Table 11.

Due to time limitations, CPEE scores only for a month were captured. It is recommended that the organisation continue this CPEE method for the rest of the year.

Table 11.

Performance Evaluation Score for Six Employees over 4 Weeks.

Employee	Mean Performance Score for the Week				Monthly Mean Performance Score	Monthly Rank
Employee	Week 1	Week 2	Week 3	Week 4	Monthly Mean Performance Score	Monthly Rank
Revathy	0.252	0.204	0.185	0.229	0.217	2
Sajith	0.094	0.118	0.178	0.184	0.143	6
Praseetha	0.172	0.152	0.181	0.178	0.171	3
Divya	0.146	AB	0.114	0.192	0.151	5
Adarsh	0.119	0.241	0.127	0.188	0.169	4
Sumesh	0.218	0.241	0.217	0.195	0.218	1

Source: The authors.

Note: AB—Divya was absent during the majority of Week 2.

Discussions and Managerial Implications

The proposed MPMM for CPEE is effortless to execute and simple to comprehend, thus making it easy to administer continuously. The proposed method offers the following grounds for discussion about administering CPEE:

Frequency of CPEE: The frequency of CPEE depends on the organisation’s culture, policy and manager’s willingness. It can be as short as a weekly basis (as demonstrated) or as long as a quarterly basis. The CPEE frequency should not be more than quarterly because the continuity may be lost. Shorter the frequency, the better and more objective the results could be (Pichler et al., 2020). Assuming the CPEE to be a weekly process, over 1 year, there would be at least 50 data points providing information about the performance of the employees. The performance trend of the employee can be identified using a time series analysis.

Reward and recognition: After every cycle of CPEE, a rank of employees based on their performance score is obtained. The top-ranked employee(s) can be rewarded and appreciated to continue their good performance (Sumesh and Revathy as per Table 11). To facilitate better performance continuously, the employee(s) with the top performance score or the employee(s) who exceeds a predefined threshold score can be encouraged with suitable reward and recognition. This reinforcement will act as a feed-forward mechanism for better performance (Bradler et al., 2016). Such short-term achievements would bring in the sense of healthy competitiveness and may motivate others in the team to vie for such recognition (Doucette, 2018).

Training and performance improvement programmes: The rankings arrived at in Table 11 indicate that Sajith has the least score and ranked last among six. Upon comparing this with Table 10, it can be identified that Sajith has scored the least during the first cycle of CPEE and continued to perform poorly during the subsequent three cycles of CPEE. If the poor performance continues for further cycles of CPEE, the manager should intervene and try to understand the reasons behind his underperformance. By making the CPEE scores transparent, employees in the team would realise how much they need to improve. Employee(s) with a low score can be asked to undergo appropriate performance improvement training immediately rather than wait for the output from the PAS after the annual appraisal. This helps eliminate any performance gaps, and corrective action can be taken at the right time (Gravina et al., 2021). The underperformers could be offered training, mentoring or counselling support to work better. This early detection of performance deficiency can also help employee retention (Ibrahim et al., 2017).

Input to existing PAS: The scores obtained using the proposed MPMM at the end of every cycle of CPEE that is cumulatively stored in a database can act as an objective input to the existing PAS. This reduces the recency effect (Spina & Spina, 2021) and reveals accurate information about the performance of employees across the appraisal period.

Rather than relying on the manager’s memory (Steiner & Rain, 1989) during the annual appraisal process, the periodic outputs from the proposed method for CPEE could help managers easily trace employees’ performance, and necessary inputs can be passed on to the PAS with increased credibility.

Task delegation and interim decisions: The periodic CPEE scores show a performance distribution of employees in a team, which can be helpful for a new manager for task delegation and load scattering (Wakhyuni, 2018). The performance distribution and the cumulative performance score can be appropriately utilised for interim decisions related to transfers, promotion or incentive distribution (Susilowati et al., 2018).

Gamification: Enterprising managers can design the proposed method for CPEE into a gamification process (Mollick & Werbach, 2015). As the relative performance scores are known to all employees, just like in a game, the employees challenge each other to score better during the forthcoming cycles of CPEE.

Challenges in implementing MPMM for CPEE: While there are several advantages of the proposed method for CPEE, there are also challenges. The most important one is creating an acceptance and familiarity among managers to use the proposed method for CPEE. The method was successfully implemented in an organisation, and the initial feedback received were encouraging. Organisations should make conscious efforts to imbibe the method of CPEE into their culture. Team size is another challenge for CPEE. Larger the team size, complex is the calculation. However, a process template for the proposed MPMM can be easily developed using a spreadsheet or software that eases the computational process. The team size should ideally be small enough (Laughlin et al., 2006) so that the administrative effort in generating the required input to MPMM is minimum.

The proposed MPMM process helps generate relative performance scores of employees of a homogeneous team. The managers need to interpret the performance score judiciously and use it to improve the performance of employees in a real-time manner. Importantly, managers should imbibe and own the entire process.

Conclusion

Contemporary organisations need to have an agile process to continuously evaluate their employees’ performance. A well-designed CPEE reduces the subjectivity of evaluation and informs the employees about their performance on a real-time basis. This article proposed and validated a procedural framework for CPEE using the MPMM. The following are the contributions from the article:

Although there is literature supporting the necessity and importance of CPEE, there is no known process to operationalise CPEE. The literature is augmented by designing a procedural framework for employee performance evaluation.

An MPMM is proposed by eliminating the limitations of PMM.

The proposed MPMM for CPEE is validated by demonstrating the workability of the proposed method in an organisation.

The cumulative performance score computed from the proposed MPMM for CPEE can act as an objective input to the existing PAS.

Limitations and Future Scope

In this method, the large team size was not accounted for. As with all other MCDM methods, it is proposed that the team size (alternatives) should be less (Mukhametzyanov & Pamucar, 2018). The team composition, size and perseverance are the challenges for managers in implementing CPEE using the proposed MPMM.

This study does not address the organisational alignment or generalisability of the proposed method. Although an initial endorsement is offered, the proposed method needs further validation from an organisation-wide setting to empirically analyse the workability of the process. The linkage of the proposed method for CPEE with the organisation’s existing PAS also needs investigation. A working software or template for CPEE, using the proposed MPMM process, is an immediate future work.

Footnotes

Declaration of Conflicting Interests

The authors declared no possible conflicts of interest concerning this article’s research, authorship and publication.

Funding

The authors received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Sreejith S. S.

References

Afshari

, Mojahed

, Yusuff

, Hong

, & Ismail

(2010). Personnel selection using ELECTRE. Journal of Applied Sciences , 10(23), 3068–3075.

Aguinis

, Joo

, & Gottfredson

R. K.

(2011). Why we hate performance management—And why we should love it. Business Horizons , 54(6), 503–507.

Ahmed

, Sultana

, Paul

S. K.

, & Azeem

(2013). Employee performance evaluation: A fuzzy approach. International Journal of Productivity and Performance Management .

Ardabili

F. S.

(2011). New framework for modeling performance evaluation for bank staff departments. Australian Journal of Basic and Applied Sciences , 5(10), 1037–1043.

Biernat

, Fuegen

, & Kobrynowicz

(2010). Shifting standards and the inference of incompetence: Effects of formal and informal evaluation tools. Personality and Social Psychology Bulletin , 36(7), 855–868.

Boice

D. F.

, & Kleiner

B. H.

(1997). Designing effective performance appraisal systems. Work Study . https://doi.org/10.1108/00438029710367622

Bongo

M. F.

, Alimpangog

K. M. S.

, Loar

J. F.

, Montefalcon

J. A.

, & Ocampo

L. A.

(2018). An application of DEMATEL-ANP and PROMETHEE II approach for air traffic controllers’ workload stress problem: A case of Mactan Civil Aviation Authority of the Philippines. Journal of Air Transport Management , 68, 198–213.

Bradler

, Dur

, Neckermann

, & Non

(2016). Employee recognition and performance: A field experiment. Management Science , 62(11), 3085–3099.

Burge

(2009). The systems engineering tool box . https://www.burgehugheswalsh.co.uk/uploaded/1/documents/pugh-matrix-v1.1.pdf

10.

Cholil

S. R.

, Pinem

A. P. R.

, Hendrawan

, Handayani

, & Nugroho

(2020). Best employee selection in Central Java using the ELECTRE method. In Cholil

S. R.

, Pinem

A. P. R.

, Hendrawan

, Handayani

, & Nugroho

(Eds.), Engineering, information and agricultural technology in the global digital revolution (1st ed.). CRC Press.

11.

Doucette

J. N.

(2018). Advice on meaningful recognition. Nursing Management , 49(8), 56.

12.

Fawcett

(2015). 7 reasons why you should be conducting performance appraisals more often . HAYS. https://social.hays.com/2015/08/31/7-reasons-why-you-should-be-conducting-performance-appraisals-more-often/

13.

Gravina

, Nastasi

, & Austin

(2021). Assessment of employee performance. Journal of Organizational Behavior Management , 124–149. https://doi.org/10.1080/01608061.2020.1869136

14.

Grund

, & Przemeck

(2012). Subjective performance appraisal and inequality aversion. Applied Economics , 44(17), 2149–2155.

15.

Gürbüz

(2010). Multiple criteria human performance evaluation using Choquet integral. International Journal of Computational Intelligence Systems , 3(3), 290–300.

16.

Gürbüz

, & Albayrak

Y. E.

(2014). An engineering approach to human resources performance evaluation: Hybrid MCDM application with interactions. Applied Soft Computing , 21, 365–375.

17.

Haddad

, Sanders

, & Bausch

(2019). Selecting a robust decision making method to evaluate employee performance. International Journal of Management and Decision Making , 18(4), 333–351.

18.

Hemaida

, & Everett

(2003). Employee performance evaluation using the analytic hierarchy process. Academy of Information and Management Sciences Journal , 6(2), 67–76.

19.

Hsien Yu

, & KuoYan

(2020). The application of a managerial competency model in employee promotion—Combination of AHP and VIKOR methodology. International Journal of Business and Management , 15(7), 1–77.

20.

Ibrahim

, Boerhannoeddin

, & Bakare

K. K.

(2017). The effect of soft skills and training methodology on employee performance. European Journal of Training and Development . https://doi.org/10.1108/EJTD-08-2016-0066

21.

Ishizaka

, Resce

, & Mareschal

(2018). Visual management of performance with PROMETHEE productivity analysis. Soft Computing , 22(22), 7325–7338. https://doi.org/10.1007/s00500-017-2884-0

22.

Islam

, & bin Mohd Rasad

(2006). Employee performance evaluation by the AHP: A case study. Asia Pacific Management Review , 11(3), 163–176.

23.

Islam

, Kabir

, & Yesmin

(2013). Integrating analytic hierarchy process with TOPSIS method for performance appraisal of private banks under fuzzy environment. Studies in System Science , 1(4), 57–70.

24.

Jati

(2012). A study on the performance appraisal method of vocational education teachers using Promethee II. Journal of Education , 5(1). https://journal.uny.ac.id/index.php/joe/article/view/1760

25.

Jawahar

(2006). An investigation of potential consequences of satisfaction with appraisal feedback. Journal of Leadership and Organizational Studies , 13(2), 14–28.

26.

Kafabih

, & Budiyanto

(2020). Determination of annual employee salary increase and best employee reward using the fuzzy-TOPSIS method (pp. 1–5). https://ieeexplore.ieee.org/document/9166188

27.

Kleiman

L. S.

, Biderman

M. D.

, & Faley

R. H.

(1987). An examination of employee perceptions of a subjective performance appraisal system. Journal of Business and Psychology , 2(2), 112–121.

28.

Komendat

, & Didona

(2016). Performance evaluations and employee’s perception of job security. International Journal of Scientific and Research Publications , 6(1), 646–649.

29.

Kondrasuk

J. N.

(2011). So what would an ideal performance appraisal look like? Journal of Applied Business and Economics , 12(1), 57–71.

30.

Laughlin

P. R.

, Hatch

E. C.

, Silver

J. S.

, & Boh

(2006). Groups perform better than the best individuals on letters-to-numbers: Effects of group size. Journal of Personality and Social Psychology , 90(4), 644–651. https://doi.org/10.1037/0022-3514.90.4.644

31.

Levit

(2019). Viewpoint: How to collect continuous feedback . SHRM. https://www.shrm.org/resourcesandtools/hr-topics/employee-relations/pages/viewpoint-how-to-collect-continuous-feedback.aspx

32.

Manoharan

, Muralidharan

, & Deshmukh

(2009). Employee performance appraisal using data envelopment analysis: A case study. Research and Practice in Human Resource Management , 17(1). http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.731.874

33.

Mollick

, & Werbach

(2015). Gamification and the enterprise. In Walz

S. P.

, & Deterding

(Eds.), The gameful world: Approaches, issues, applications (p. 439). MIT Press.

34.

Morte

, Pereira

, & Fontes

D. B.

(2013). MCDA applied to performance analysis and evaluation of road drivers: A case study in the road transport company . BS’13-Third International Conference on Business Sustainability. https://repositorio-aberto.up.pt/bitstream/10216/70496/2/50217.pdf

35.

Mukhametzyanov

, & Pamucar

(2018). A sensitivity analysis in MCDM problems: A statistical approach. Decision Making: Applications in Management and Engineering , 1(2), 51–80.

36.

Mullur

, Mattson

, & Messac

(2003). Pitfalls of the typical construction of decision matrices for concept selection (p. 466). http://www.csun.edu/~rsh12598/415/additional/messac_Pitfalls_reno_2003.pdf

37.

Nader

(2010). A framework for systematic application and measurement of the innovation management processes. Bilgi Ekonomisi ve Yönetimi Dergisi , 5(2), 57–69.

38.

Ossadnik

, & Lange

(1999). AHP-based evaluation of AHP-Software. European Journal of Operational Research , 118(3), 578–588.

39.

Palaiologos

, Papazekos

, & Panayotopoulou

(2011). Organizational justice and employee satisfaction in performance appraisal. Journal of European Industrial Training . https://doi.org/10.1108/03090591111168348

40.

Pichler

, Beenen

, & Wood

(2020). Feedback frequency and appraisal reactions: A meta-analytic test of moderators. The International Journal of Human Resource Management , 31(17), 2238–2263.

41.

Pugh

(1981a). Concept selection: A method that works. In Review of design methodology. Proceedings international conference on engineering design. Heurista, Zürich, 497–506. https://www.scirp.org/(S(i43dyn45teexjx455qlt3d2q))/reference/ReferencesPapers.aspx?ReferenceID=84003

42.

Pugh,

(1981b). Design decision – How to succeed and know why. Design Engineering Conference (DES 81), Birmingham, England.

43.

Rahmati

, & Noorbehbahani

(2017). A new hybrid method based on fuzzy AHP and fuzzy TOPSIS for employee performance evaluation . 0165–0171. https://doi.org/10.1109/KBEI.2017.8324965

44.

Rivera

, Qiu

, Kumar

, & Petrucci

(2021). Are traditional performance reviews outdated? An empirical analysis on continuous, real-time feedback in the workplace. Information Systems Research . https://doi.org/10.1287/isre.2020.0979

45.

Saaty

T. L.

(1990). How to make a decision: The analytic hierarchy process. European Journal of Operational Research , 48(1), 9–26.

46.

Shaout

, & Yousif

M. K.

(2014). Performance evaluation–Methods and techniques survey. International Journal of Computer and Information Technology , 3(5), 966–979.

47.

Spina

J. D.

, & Spina

L. J.

(2021). Revisiting your performance appraisal system. In Spina

L. J.

, & Spina

J. D.

(Eds.), Beyond succession planning: New ways to develop talent (pp. 17–24) . Emerald Publishing. https://doi.org/10.1108/978-1-83982-078-620211005

48.

Sreejith

(2015). The X factor that can help to motivate Generation Y: Performance evaluation in IT needs to move from the manufacturing model. Human Resource Management International Digest . https://doi.org/10.1108/HRMID-12-2014-0167

49.

Sreejith

S. S.

, & Mathirajan

(2020). Development of a conceptual framework for continuous performance evaluation of employees to offer reward and recognition. Journal of Management Research , 20(4), 225–256.

50.

Steiner

D. D.

, & Rain

J. S.

(1989). Immediate and delayed primacy and recency effects in performance evaluation. Journal of Applied Psychology , 74(1), 136.

51.

Susilowati

, Anggraeni

E. Y.

, Andewi

, Handayani

, & Maseleno

(2018). Using profile matching method to employee position movement. International Journal of Pure and Applied Mathematics , 118(7), 415–423.

52.

Tanti

, Rahmad

I. F.

, & Thanri

(2018). Monitoring and evaluation of flight instructor performance with PROMETHEE method (pp. 1–6). https://doi.org/10.1109/CITSM.2018.8674261

53.

Taufiq

, & Sugiharto

(2011). The decision support system design of employee performance appraisal using analytical hierarchi process (AHP) method . Proceedings of The 1st International Conference on Information Systems For Business Competitiveness (ICISBC). https://core.ac.uk/download/pdf/11734661.pdf

54.

Taylor

F. A.

, Ketcham

A. F.

, & Hoffman

(1998). Personnel evaluation with AHP. Management Decision . https://doi.org/10.1108/00251749810245336

55.

Tripathi

, Thite

, Varma

, & Mahapatra

(2021). Appraising the revamped performance management system in Indian IT multinational enterprises: The employees’ perspective. Human Resource Management . https://doi.org/10.1002/hrm.22061

56.

Wakhyuni

(2018). An empirical investigation of the effect of workload and sops on employees work morale. International Conference of ASEAN Prespective and Policy (ICAP) , 1(1), 251–257. https://jurnal.pancabudi.ac.id/index.php/ICAP/article/view/299

57.

Watrianthos

, Ritonga

W. A.

, Rengganis

, Wanto

, & Indrawan

M. I.

(2021). Implementation of PROMETHEE-GAIA method for lecturer performance evaluation. Journal of Physics: Conference Series , 1933(1), 012067. https://iopscience.iop.org/article/10.1088/1742-6596/1933/1/012067

58.

H. -Y.

, Chen

J. -K.

, & Chen

I. -S

. (2012). Performance evaluation of aircraft maintenance staff using a fuzzy MCDM approach. International Journal of Innovative Computing, Information and Control , 8(6), 3919–3937.

59.

Zbranek

(2013). Data envelopment analysis as a tool for evaluation of employees’ performance. Acta Oeconomica et Informatica , 16(394-2016-24293), 12–21.