How Efficient Are Emotional Intelligence Trainings: A Meta-Analysis

Abstract

This multilevel meta-analysis examines whether emotional intelligence (EI) can be enhanced through training and identifies training effects’ determinants. We identified 24 studies containing 28 samples aiming at increasing individual-level EI among healthy adults. The results revealed a significant moderate standardized mean change between pre- and post-measurement for the main effect of EI training, and a stable pre- to follow-up effect. Additionally, the type of EI model, dimensions of the four branch model, length, and type of publication turned out to be significant moderators. The results suggest that EI trainings should be considered effective interventions.

Keywords

emotional intelligence intervention meta-analysis training

Introduction

EI has become a very important psychological construct within the last 10 years. There are several meta-analyses stating the relevance of EI as it has been shown to be related to health outcomes (Martins, Ramalho, & Morin, 2010; Schutte, Malouff, Thorsteinsson, Bhullar, & Rooke, 2007) and subjective well-being (Sánchez-Álvarez, Extremera, & Fernández-Berrocal, 2016). It has been also associated with academic (Perera & DiGiacomo, 2013) and work performance (Joseph, Jin, Newman, & O’Boyle, 2015).

Due to these findings stressing the importance of EI in everyday life, important growth in EI interventions has been occurring in recent years. These interventions have had different target groups, such as children and adolescents (Nélis et al., 2011; Ruini et al., 2009), and managers and employees (Cherniss, Grimm, & Liautaud, 2010; Slaski & Cartwright, 2003). In addition, these interventions have been conducted in various contexts (educational, clinical, and organizational), and aiming at various outcomes. We found one review containing a meta-analysis exploring the effects of four EI interventions (Schutte, Malouff, & Thorsteinsson, 2013). Schutte and colleagues concluded that EI interventions are promising and called for further exploration of interventions’ efficacy and utility.

The present study presents the first extensive meta-analysis on the efficacy of EI interventions. After a systematic literature review, we analyzed 28 samples from 24 studies aiming at increasing the level of EI through different interventions. Current theoretical approaches, models, and interventions are presented briefly here. We then provide meta-analytically derived results on the efficacy of the EI interventions and moderating effects. Finally, we discuss findings and make suggestions for future research.

Theoretical Approaches to EI

The most common theoretical approaches to emotional intelligence (EI) in the literature are trait EI and ability EI. The ability approach considers EI to be composed of specific emotional abilities. One of the most acknowledged and scientifically rigorous ability models of EI is Mayer and Salovey’s four branch model that distinguishes between perceiving emotions, facilitating thought, understanding emotions, and managing emotions (Mayer, Salovey, & Caruso, 2004). The trait EI approach, on the other hand, views EI as emotion-related dispositions, at a hierarchically lower position than personality traits that determine the way people behave in emotional situations (Petrides, Pita, & Kokkinaki, 2007).

Besides these two approaches, there are so-called mixed models of EI, such as Bar-On’s emotional social intelligence (ESI) model (Bar-On, 2006) and Goleman’s model of emotional competences (Boyatzis, Goleman, & Rhee, 2000), which include other noncognitive features like social skills, motivation, self-esteem, and personality aspects. Finally, integrative models of EI attempt to reconcile and combine different theoretical approaches to EI. Integrative EI models are, for instance, Mikolajczak, Quoidbach, Kotsou, and Nélis’s (2009) tripartite model of EI, Fiori’s (2009) dual process approach to EI, and Joseph and Newman’s (2010) cascading model of EI. Independent from theoretical approach or model, most EI experts agree that EI refers to measureable individual differences in experiencing and processing emotions and emotion-related information.

EI Trainings

Recent years have brought an important increase in EI interventions. These interventions are set in different contexts and vary in terms of the theoretical models serving as their bases (e.g., Cherniss et al., 2010; Groves, McEnrue, & Shen, 2008; Kirk, Schutte, & Hine, 2011; Nélis et al., 2011; Ruiz-Aranda, Salguero, Cabello, Palomera, & Fernández-Berrocal, 2012; Vesely, Saklofske, & Nordstokke, 2014). These interventions have aimed at improving different outcomes, such as life satisfaction, perceived health, stress reduction, emotional self-efficacy, mental health, quality of interpersonal relationships, and even employability (Dacre Pool & Qualter, 2012; Kirk et al., 2011; Kotsou, Nélis, Gregoire, & Mikolajczak, 2011; Nélis et al., 2011). The first attempt to obtain an estimate of the effect size of EI trainings for adults through a meta-analysis was made by Schutte et al. (2013), who took into account m = 4 experimental intervention studies with random assignment to intervention and control groups and comprised o = 6 effect sizes. The results of their findings yielded a moderate overall effect size for the impact of training on emotional intelligence (g = 0.46), yet their significance is limited due to the low number of studies included.

Campo, Laborde, and Weckemann (2015) also conducted a review of studies that aimed at improving EI in adults. They concluded that the most promising results were obtained from the interventions designed using the Mikolajczak and colleagues’ tripartite model of EI (Mikolajczak et al., 2009), not only for increasing the level of EI, but also for bringing positive changes in several other psychological variables.

These reviews hint at the plausibility that EI interventions developed using different theoretical approaches and conducted in different contexts can increase the level of EI. Nevertheless, Schutte et al.’s (2013) meta-analysis only included studies with random assignment to intervention and control conditions, which limited the number of studies to be included in the analysis. Moreover, several studies on new EI interventions have been published during the last two years. In the present work, a higher number of studies was included (m = 24) in order to increase the analysis’ statistical power and to be able to identify determinants of the efficacy of EI training.

Although there is still a lot of disagreement about the conceptual delimitation and definition of EI, many of the researchers in this field agree that some of the aspects of EI can be developed, which is supported by earlier research. We hypothesize that EI interventions will have a significant effect on increasing the level of EI (Hypothesis 1).

Moderators of Intervention Effects

Standpoints on the plasticity of EI also depend on the theoretical model of EI. According to the trait EI perspective, EI is conceptually more similar to stable personality traits (Petrides et al., 2007) and is, thus, more resistant to change. On the other hand, EI defined as an ability or set of abilities that can be learned and taught (Salovey & Mayer, 1990) has more potential for change and development. Moreover, mixed models assume more components to be taken into account in an intervention, such as social skills and motivation, and these would probably be more difficult to train. Therefore, we hypothesize that the effects of EI interventions will differ depending on the kind of model underlying the intervention. In interventions based on ability models, the effects of EI trainings will be higher than in those based on trait and mixed models (Hypothesis 1a).

Moreover, regarding the four branches of the Mayer and Salovey model (Mayer et al., 2004), it seems reasonable that perceiving and understanding emotions might have a higher potential to be taught than facilitating thought and regulating emotions. Understanding emotions is associated with cognitive processing the most (MacCann, Joseph, Newman, & Roberts, 2014; Mayer, Salovey, Caruso, & Sitarenios, 2001; Roberts, Zeidner, & Matthews, 2001) and reflects accumulated emotion-related knowledge (Mayer, Caruso, & Salovey, 2016); while perceiving emotions reflects semantic or lexical knowledge (Lindquist, Gendron, Feldman Barrett, & Dickerson, 2014). On the other hand, facilitating thought and regulating emotions include motivational, emotional, and cognitive factors (Mayer et al., 2016; Mayer et al., 2001), which makes them harder to train. For these reasons, EI trainings should be more effective for the perceiving and understanding emotions dimensions than for those of facilitating thought and regulating emotions (Hypothesis 1b).

Finally, because increasing the length and duration of an intervention is one of the most important recommended practices for intervention success (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011), we examined the duration of the training as a possible moderator of the intervention effects. For example, Sin and Lyubomirsky (2009) found that duration of the positive psychology interventions was a significant determinant of their effectiveness, and that longer interventions were more beneficial for well-being. We hypothesize that length of training (defined as hours per week during which the training takes place) will have an effect on the level of its efficacy: the longer the EI intervention, the higher its efficacy (Hypothesis 1c).

To summarize, this meta-analysis is the first to investigate the efficacy of EI interventions in an extensive way. Detailed results are yielded by analyzing a sufficient number of studies and by testing different moderators and control variables. Thus, more in-depth knowledge on the optimization of effects of EI interventions was obtained.

Method

Inclusion and Exclusion Criteria

In order to include only studies that address the research questions outlined before, we defined the following inclusion and exclusion criteria for eligibility of studies: (a) an emotional intelligence training based on an empirically validated model (psychometrically tested and validated) took place in the course of the study; (b) EI was measured pre- and post-training; (c) there were control and intervention comparable groups. Contrary to Schutte et al.’s (2013) study, randomization to the experimental conditions is not one of the inclusion criteria in the present meta-analysis. Still, we excluded studies that did not use control groups or whose control groups are not comparable with the experimental groups. We controlled for randomization to intervention conditions by comparing effect sizes from randomized and non-randomized studies by moderator analysis. General quality of the study was also analyzed as a control variable. Further, an additional analysis using only randomized controlled trials is reported. (d) Individual-level data had to be given or computable. (e) The EI measure had to be of good psychometric quality. (f) Participants had to be older than 16 years. Plasticity of children’s brain might be higher than that of adults (Campo et al., 2015), and there is evidence that children gradually acquire competence in understanding emotions and that develop this competence until adulthood (Widen, 2013). The efficacy of EI trainings for children, therefore, cannot be compared to that for adults, as there might be a higher change in the latent variable in children, which is not due to the training, compared to adults. Additionally, participants had to be (g) mentally and physically healthy, and (h) nondelinquents. Previous studies showed that people with health-related problems might suffer from increased emotional distress, and these health problems might cause negative emotional reactions (Yalcin, Karahan, Ozcelik, & Igde, 2008). In addition, Adetunji, Soezin, and Margaret (2015) showed that offenders have lower levels of EI and might have lower levels of emotional competences. Therefore, EI training might have a different effect in this context. (i) Moreover, we excluded studies that used and reported the same data as in any previously published study.

Literature Search

The search term (“emotional intelligence” OR “emotional competenc*” OR “emotional interven*”) AND (interven* OR train* OR program*) yielded 1,672 results from the databases PsychInfo, PsyCritiques, and PsycArticles, of which titles and abstracts were scanned. The search was limited to studies published in the English language. In addition, we checked studies cited by reviews on the topic (e.g., Campo et al., 2015; Schutte et al., 2013) and searched for studies citing these reviews. The database search in combination with the forward and backward search led to m = 97 studies that were shortlisted and checked for eligibility in detail. The overall literature search resulted in m = 24 studies containing k = 28 samples fitting our criteria.

Coding

A coding scheme was developed based on Cochrane collaboration standards (Higgins & Green, 2011). Besides study, sample, and intervention characteristics, the relevant moderators defined in the hypotheses and control variables were added to the scheme. Variables indicating the quality of the study were also included (Higgins & Green, 2011) and used to form a quality index. All variables included in the coding scheme and their explanations are presented in the Appendix.

One of the authors coded all of the studies. In case of tentativeness, coding decisions were discussed between the authors. If necessary information was missing, authors of the eligible studies were contacted and missing information was added.

Extraction of Effect Sizes

Effect sizes were calculated based on means (M), standard deviations (SD), and correlations (r) as follows: Firstly, standardized mean changes (SMCs) between pre- and post-measurements were computed for the experimental and control groups separately for each outcome by means of the formula:

S M C_{p r e - p o s t} = \frac{M_{p o s t} - M_{p r e}}{\sqrt (S D_{p r e}^{2} + S D_{p o s t}^{2} - 2 r_{p r e - p o s t} S D_{p r e} S D_{p o s t})}

(Cooper, Hedges, & Valentine, 2009). Because r_pre-post, which is the correlation between pre and post measurement, was not reported in any of the eligible studies, it was necessary to estimate it. We used r _pre-post = .50, which can be considered rather conservative. Secondly, standardized mean changes (SMC_pre-post) between experimental and control groups were calculated as the difference between SMC_cg:pre-post and SMC_eg:pre-post. Thirdly, we corrected SMC_pre-post and their variances for lack of reliability by using Cronbach’s alpha (α) and by the formula of Hunter and Schmidt (2004). For 67% of the outcomes, α was given. For these, the mean α was 0.80 and we used this value if α was not given by the study. These corrected SMC_pre-post outcome values compare pre to post changes between experimental and control groups. The same calculations were conducted for follow-up measurements: we calculated SMC_pre-fu (as the difference between SMC_cg:pre-fu and SMC_eg:pre-fu) and SMC_post-fu (as the difference between SMC_cg:post-fu and SMC_eg:post-fu) and corrected them for reliability. If outcomes on the level of the EI dimensions were reported, we extracted effect sizes on the dimension level as required for moderator analyses regarding differences between dimensions. If these were not given, effect sizes were computed for overall EI outcomes.

Analyses

Analyses were run with the program R (Version 3.2.5) and the package metafor (Viechtbauer, 2010). Because many studies report several measures or differentiate between dimensions of EI as outcome variables, there were dependencies between the outcomes from the studies. If several outcomes are reported for the same sample, it is necessary to control for this dependency by robust variance estimation (RVE; Cooper et al., 2009; Cooper, Hedges, & Valentine, 2009). Thus, we applied random effect models that take into account the correlations between outcomes stemming from one sample. The correlation between outcomes was estimated as r_dim = .40 (e.g., Mayer, Salovey, Caruso, & Sitarenios, 2003; Petrides, 2009).

In order to test the hypotheses and control for further potential moderating effects, we conducted several moderator analyses and meta-regressions. Two-sided p-values are reported and an alpha level of .05 was used. Beforehand, we analyzed correlations between moderators and conducted Q-tests that, if significant, justified random effects and moderator analyses. However, due to the heterogeneity of interventions, random effects are necessary because of theoretical assumptions as well.

Publication bias was analyzed by funnel plots and Egger tests (Egger, Smith, Schneider, & Minder, 1997) and, if necessary, the trim and fill method was applied (Cooper et al., 2009). Unfortunately, a univariate random effects model that does not account for dependencies between outcomes had to be used to control for publication bias, because these methods are not yet implemented for multilevel meta-analyses in metafor. To our knowledge, there is no other program or R package that is able to check for publication bias in random effects meta-analyses when RVE is applied. For this reason, an analysis based on the univariate model regarding publication bias seemed plausible.

Sensitivity analyses tested for differences in the size of the resulting effect size SMC due to the choice of correlations and reliabilities if they were not given in the primary study. We chose to set the correlations between pre-, post-, and follow-up measurements to r = .50 for our analyses. In order to account for differences in results due to this choice, the correlations were set to r = .30 and r = .70. For each choice of correlation, resulting SMCs were compared. The correlations between outcome variables were set to r_dim = .40 in the analysis reported in the previous lines. We compared our results to those resulting from r_dim = .20 and r_dim = .80. In the same way, reliabilities that were set to α = .80, if not reported by the studies, were set to α = .70 and α = .90 for the sensitivity analysis. We inspected differences in SMCs when all of the estimates were changed at the same time.

Results

Sample, Intervention, and Study Characteristics

This meta-analysis is based on k = 28 samples from m = 24 studies, and the overall sample size was N = 1,986. The sample consisted of 64.03% females, and the mean age was 26.59 years (min = 18.00, max = 43.00). Studies were published between 2006 and 2016. Most of the samples were from studies published in peer-reviewed journals (78.57%; k = 22), 17.86% (k = 5) were from theses or dissertations, and 0.04% (k = 1) were research articles published by a university.

On average, the EI trainings consisted of 6.09 sessions that lasted 2.57 hours each. The average training had a length of 4.46 hours per week. A fixed schedule was given in 92.86% (k = 26) of studies, and 35.29% (k = 17) defined individual goals for participants. Diary writing was requested for 42.68% (k = 12), and 25.00% (k = 7) had personal coaches. Feedback was given for 35.71% (k = 10) of the samples, and in the majority of the cases (82.14%, k = 23) the trainings were both experience-based (skill practice by role-plays or in actual life, reflective writing, talking about emotions) and theory-based (lectures, group discussions, story analyzing, video vignettes, reading texts, case studies, workbook exercises, tests).

Six different EI models were observed serving as the basis for the trainings. Those that were used the most were Mayer and Salovey’s four branch model (Mayer et al., 2004; 64.29%, k = 18; ability model), and Bar-On’s ESI model (Bar-On, 2006; 17.86%, k = 5; mixed model). Besides these, two interventions used Palmer and Stough’s Swinburne emotional intelligence model (see Gignac, 2010; 7.14%, k = 2; mixed model), and one intervention was based on Petrides and Furnham’s trait emotional intelligence model TEIQue (Petrides, Furnham, & Frederickson, 2004; 3.6%, k = 1; trait model). Finally, the EI model of the collaborative for academic, social, and emotional learning (CASEL; Jennings & Greenberg, 2009; 3.6%, k = 1; ability model) and the model of Bisquerra and Pérez-Escoda (2007; 3.6%, k = 1; trait model) served as a basis for one intervention each.

The EI measures applied were diverse as well, as we observed 18 different measures from the k = 28 samples, of which the most frequently used ones were the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT; o = 26, k = 8), the TEIQue (o = 10, k = 8), and the Emotional Quotient Inventory (EQ-I; o = 13, k = 5). Self-report measures were used in k = 8 samples (o = 57), peer-report measures in k = 5 samples (o = 7), and ability measures in k = 7 samples (o = 20). The average time interval between pre and post measurement was 2.06 months with a range from 45 minutes to 9 months. Follow-up measurements were conducted on k = 7 (25%) samples only, with an average time interval of 4.06 months between the end of the training and the follow-up measurement (min = 6.83 days, max = 12.73 months).

Main Training Effect

Hypothesis 1

Training had a moderate effect on emotional intelligence. The multilevel random-effects meta-analysis of k = 28 samples and o = 84 outcomes resulted in an overall effect size of SMC_pre–post = 0.51 with a 95% confidence interval of [0.41, 0.60] (all confidence intervals reported are 95%). The standard deviation of the effect size was τ = 0.17. The corresponding forest plot is shown in Figure 1. The overall training effect is significantly different from zero (p < .001), thus, Hypothesis 1 is supported.

Figure 1.

Forest plot for SMC_pre–post. Effect sizes per EI dimension and sample, their 95% CIs, and weight of each study (represented by the thickness of the lines and boxes) for the main training effects comparing pre- to post-changes between experimental and control groups are illustrated. The meta-analytically computed average effect of SMC_pre–post = 0.51 is displayed as a diamond-shaped figure at the bottom of the plot.

Hypothesis 1a

The three most common kinds of models, which are ability, mixed, and trait models of EI were compared. For ability models, we found an effect of SMC_pre–post = 0.60, 95% CI [0.48, 0.71], whereas the effect for mixed models was SMC_pre–post = 0.31, 95% CI [0.12, 0.50] and SMC_pre–post = 0.31 [−0.04, 0.66] for trait models. Hence, trainings that are based on an ability model show significantly higher effects than trait models (p = .017), which supports Hypothesis 1a. This random effect model is based on k = 28 and o = 84 (τ = 0.16).

Hypothesis 1b

A significant moderating effect of the EI dimensions from the Mayer and Salovey model (Mayer et al., 2004) was found. The moderator analysis based on k = 16 samples and o = 55 outcomes showed a significant difference between the training effects of the dimensions understanding emotions (SMC_pre–post = 0.69, 95% CI [0.48, 0.91]) and facilitating thought (SMC_pre–post = 0.42, 95% CI [0.21, 0.63]; p = .038) in a random effects model (τ = 0.15). There were no significant differences between the other dimensions. Hence, Hypothesis 1b was partially supported.

Hypothesis 1c

Meta-regression supported Hypothesis 1c, as the length of the training had an impact on the size of the training effect (b_length = 0.03, 95% CI [0.01, 0.05], p = .006). The random effects analysis is based on k = 27 samples and o = 80 outcomes (τ = 0.15). Thus, if the length of the training is increased by one more hour per week, the resulting effect size of the training grows by 0.03.

Control variables

We controlled for the influence of the type of publication by applying a random effects model (k = 28, o = 84, τ = 0.15) comparing articles published in peer-reviewed journals with theses. This was a significant moderator. Peer-reviewed articles showed a significantly higher effect size (SMC_pre–post = 0.57, 95% CI [0.47, 0.67]) than theses (SMC_pre–post = 0.12, 95% CI [−0.15, 0.40]; p = .003). We did not find any influence of the other control variables.

Publication bias

A significant publication bias was found for the univariate model SMC.uni_pre–post = 0.52, 95% CI [0.44, 0.59] (p < .001, τ = 0.19) as indicated by the funnel plot (see Figure 2) and a significant Egger test (p < .001). According to the trim and fill method, k = 11 samples should be added on the left side of the funnel plot accounting for asymmetry. The overall effect would then decrease to SMC.uni_pre–post = 0.46, 95% CI [0.38, 0.54] (p < .001; τ = 0.26). Because the univariate model is not the appropriate one, as dependencies between outcomes are not accounted for, these results have to be treated with caution. Unfortunately, we cannot give an exact estimate of the correct effect size based on the multilevel model. Still, the true effects should be considered smaller than SMC_pre–post = 0.51 because of publication bias.

Figure 2.

Funnel plot indicating publication bias for the main effect of training by its asymmetry. Effect sizes from the single EI dimensions comparing pre- to post-changes between experimental and control groups are plotted as observed outcomes against standard errors. On the lower left side, a few effect sizes are missing for the plot to be symmetric.

Follow-Up Effects

Out of the overall k = 28 samples, in k = 7 samples, follow-up measurements were conducted, which resulted in o = 16 follow-up outcomes. If we take only follow-up studies into account, we find a pre- to post-effect of SMC_pre–post = 0.62, 95% CI [0.43, 0.81] (p < .001; τ = 0.14). Comparing pre- to follow-up, the effect remained at SMC_pre–fu = 0.55, 95% CI [0.33, 0.77] (p < .001; τ = 0.23), which was supported by a zero effect from post- to follow-up (SMC_post–fu = −0.06, 95% CI [−0.21, 0.09], p = 0.47; τ = 0.00). We did not conduct moderator analyses due to the low number of samples, and we did not find a publication bias for the follow-up effects.

Randomized Controlled Trials (RCTs)

Moderator analyses showed no significant effect between samples randomly assigned to experimental and control groups and those assigned quasi-randomly or in any other way. Yet, a separate meta-analysis was conducted for RCTs only (Higgins & Green, 2011). A pre–post effect of SMC_RCTs = 0.50, 95% CI [0.38, 0.66] was found for k = 10 samples and o = 24 outcomes (p < .001; τ = 0.14). We did not conduct moderator analyses due to the low number of samples. For the same reason, we could not evaluate follow-up effects of RCT studies (k = 3).

For RCTs, publication bias was significant based on a univariate model (SMC.uni_RCTs = 0.57, 95% CI [0.38, 0.76], p < .001; τ = 0.35) as indicated by a significant Egger test (p < .001). The trim and fill method suggests to add k = 6 studies, which would lead to a decrease of the effect to SMC_RCTs = 0.39, 95% CI [0.14, 0.66]; p = .003; τ = 0.62. The results have to be treated with caution as they rely on the univariate and not the appropriate multilevel model. An effect smaller than SMC_RCTs = 0.50 should be assumed.

Sensitivity Analysis

We conducted a sensitivity analysis for the main effects sizes reported before. Results are shown in Table 1. Variations of correlations between outcome variables and reliabilities accounted for marginal differences in results, whereas the choice of correlations between pre-, post-, and follow-up measurements should not be neglected.

Table 1.

Results from sensitivity analysis.

	α	r _dim	SMC_pre–post			SMC_pre–fu			SMC_post–fu
			r_pre–post			r_pre–fu			r_post–fu
			.30	.50	.70	.30	.50	.70	.30	.50	.70
Main analysis	.70	.20	0.42	0.49	0.61	0.57	0.68	0.88	−0.05	−0.05	−0.07
		.40	0.41	0.48	0.60	0.46	0.55	0.72	−0.05	−0.06	−0.07
		.60	0.41	0.47	0.59	0.56	0.67	0.88	−0.05	−0.06	−0.07
	.80	.20	0.44	0.51	0.62	0.57	0.68	0.88	−0.04	−0.05	−0.07
		.40	0.44	0.51	0.61	0.46	0.55	0.72	−0.05	−0.05	−0.07
		.60	0.44	0.51	0.61	0.56	0.67	0.88	−0.05	−0.06	−0.07
	.90	.20	0.42	0.49	0.62	0.57	0.68	0.88	−0.04	−0.05	−0.07
		.40	0.41	0.48	0.60	0.46	0.55	0.72	−0.05	−0.06	−0.07
		.60	0.41	0.47	0.59	0.56	0.67	0.88	−0.05	−0.06	−0.07
RCTs	.70	.20	0.25	0.28	0.33	–	–	–	–	–	–
		.40	0.31	0.36	0.45	–	–	–	–	–	–
		.60	0.25	0.28	0.33	–	–	–	–	–	–
	.80	.20	0.62	0.70	0.84	–	–	–	–	–	–
		.40	0.43	0.50	0.62	–	–	–	–	–	–
		.60	0.62	0.70	0.85	–	–	–	–	–	–
	.90	.20	0.25	0.28	0.33	–	–	–	–	–	–
		.40	0.31	0.36	0.45	–	–	–	–	–	–
		.60	0.25	0.28	0.33	–	–	–	–	–	–

Note. r_pre–post, r_pre–fu, r_post–fu = correlations between pre-, post-, and follow-up measurements; α = Cronbach’s alpha; r_dim = correlations between EI dimensions; values reported in the text are shown in boldface.

Discussion

This meta-analysis aimed at expanding previous findings on the efficacy of EI interventions and at determining important moderators of the intervention effects. After a systematic review of the literature, 24 studies (containing 28 samples) that fulfilled the inclusion criteria were meta-analyzed. The results yielded a moderate overall effect size for the impact of EI training of SMC_pre–post = 0.51 and the effect did not differ when we considered only RCTs (SMC_RCTs = 0.50). This result supports the previous findings by Schutte et al. (2013) confirming that trainings increase EI. Nevertheless, trainings that are based on ability EI models showed significantly higher effects than mixed or trait EI models. In addition, training effects of Mayer and Salovey’s (Mayer et al., 2004) understanding emotions dimension were significantly higher that the training effects of the facilitating thought dimension. Finally, the analysis of the follow-up effects showed that the effect remained at SMC_pre–fu = 0.55, suggesting that the positive changes in EI due to interventions remain over time. Still, because of a significant publication bias, the true effects should be considered slightly lower than 0.51. Nevertheless, the obtained results contribute to the existing knowledge of EI and EI development, and provide suggestions and guidelines for future interventions.

From a theoretical standpoint, much has been discussed about plasticity of EI and whether it can be learned and developed. The obtained results show that specific interventions improve EI. However, the intensity of this improvement depends on the theoretical background of the interventions. The results confirm that it is easier to develop ability EI and related explicit knowledge than trait EI. The possibility for increasing EI is inherent to the ability-based EI models. What the present study’s results imply is the idea that different operating levels of EI exist. We consider that the trainings analyzed in this study tap into what is denominated as declarative knowledge, factual information about emotions, and emotional abilities, not the actual skills to use this information (Anderson & Schunn, 2000). The idea of EI as a “multilevel” construct is not new. Fiori (2009) proposed to differentiate automatic from conscientious processes in emotional abilities. Mikolajczak et al. (2009) suggested distinguishing between emotional knowledge, abilities, and dispositions. Therefore, both, individual differences in EI and the differences in training effects might be best understood by considering the different operating levels of EI.

Moreover, the obtained results showed that trainings influence understanding emotions more than facilitating thought. Understanding emotions has been shown to be highly related with crystallized intelligence (MacCann et al., 2014), implying again that these trainings focused on increasing explicit knowledge, enhancing awareness about different emotional abilities or aspects of EI, not on how they actually are used in every-day situations. In Joseph and Newman’s (2010) cascading model of EI, the ability to understand emotions is conceptualized as accumulated knowledge structures, and positioned as a precondition for the ability to regulate emotions. This means that managing and maintaining the desired affective states requires a high level of emotional understanding, or in other words, enough accumulated (declarative) knowledge about how emotions change over time, how they differ, and which emotions are the most appropriate ones depending on the situation. Hence, in order to translate this knowledge into practice (to enhance the procedural knowledge) and in order for it to have observable benefits, repetitive and longer trainings are needed.

Future Research

Numerous interventions did not include control groups and we found ten studies that were RCTs. Follow-up measurements were conducted on seven samples only, and three RCTs conducted follow-up measurements. Here, moderator analyses were not possible due to the low number of samples. Results from RCTs provide valuable information about intervention efficacy and allow rigorous empirical comparison of the results, which is why this experimental design is strongly recommended for future research. The lack of follow-up measurements significantly limits the examination of the intervention effects. Follow-up measurement is one of the important criteria for evaluating trainings (Kirkpatrick, 1996), and we would certainly encourage researchers in the field to consider including it in their future interventions.

Furthermore, the obtained results indicated a strong influence of the type of publication on the effects of intervention and a publication bias was found as well. These findings shed a different light on the moderate training effect of SMC_pre–post = 0.51, because unpublished studies and non-peer-reviewed articles might show lower effects than published ones. As there were no differences between the quality of theses and peer-reviewed articles, the reason for this finding could be that big effects are published more easily. This might contribute to conveying a biased picture regarding the efficacy of EI interventions and we believe that nonsignificant results should also be published or at least be available more easily.

Finally, the most effective interventions were those that focused on enhancing specific emotional abilities, as conceptualized in Mayer and Salovey’s (Mayer et al., 2004) four branch model (Crombie, Lombard, & Noakes, 2011; Kidwell, Hasford, & Hardesty, 2015; Nélis et al., 2011). These interventions were carried out in different target groups and had different formats and durations, but all used a workshop approach with group discussions and interactive participation. This is useful for those who plan to develop future EI interventions.

Besides the suggestions derived from the results of the current meta-analysis, we propose some additional guidelines for EI interventions. One goal should be identifying specific individual differences and situational factors that might determine the effects of the interventions. It is most probable that not everyone will benefit from EI trainings in the same way. Schutte et al. (2013) suggest exploring cognitive styles as a possible determinant of the intervention effects. In addition, some previous studies mention positive relations between openness to experience and training proficiency (Barrick & Mount, 1991; Dean, Conte, & Blankenhorn, 2006). As Barrick and Mount (1991) argue, being curious, imaginative, and having a wider range of interests has been associated with positive attitudes towards learning and higher motivation for the training process.

Detecting the specific groups or populations that can benefit from the interventions should be another aim. Exploring the factors in work, family, or social contexts that can help people benefit the most from interventions should be the focus of future research. Schutte et al. (2013) mention family relationships, social network context, or the nature of the population as potentially important situational factors for optimizing the training effects. Moreover, entire professions or groups might suffer more from the negative effects of stress or increased emotional labour (Daus & Ashkanasy, 2005). Some attempts at enhancing EI among particularly vulnerable professions such as teachers or social workers, and vulnerable populations, such as the unemployed, have been made (Grant, Kinman, & Alexander, 2014; Hodzic, Ripoll, Bernal, & Zenasni, 2015; Vesely et al., 2014). Still, future interventions should try to detect more vulnerable groups and adapt the trainings considering the specific situational factors and needs of those groups.

Footnotes

Appendix

Table A1.

Coded variables and their explanations.

Category	Variable	Explanation
	Authors	Names of authors
	n.EG	Sample size experimental group
	n.CG	Sample size control group
Training characteristics	Time pre–post	Time between pre- and post-measurement in hours
	Time pre–fu	Time between pre and follow-up measurement in hours
	Number	Total number of sessions administered
	Duration	Duration of single session in hours
	Schedule	0 = training followed a fixed schedule; 1 = schedule differed between participants
	Model	Model that is used to define EI
	Mode	0 = experience-based; 1 = theory-based
	Techniques	Techniques to convey theory, e.g., lectures; or to practice experience, e.g., role plays
	Group vs. individual	0 = trainings were conducted in a group; 1 = trainings were conducted individually
	Feedback	0 = no feedback; 1 = feedback at least once
	Diary	0 = no diary; 1 = participants had to write a diary
	Goals	0 = no individual goals; 1 = individual goals were formulated for participants
	Instrument	Name of the EI instrument
	Reliability	Cronbach’s alpha of instrument
Moderators	Ability vs. trait model	Model that training is based on: 0 = ability model; 1 = trait model; 2 = mixed model
	Dimensions	O = overall EI; P = perceiving; F = facilitating thought; U = understanding; R = regulating; EQIra = intrapersonal; EQIer = interpersonal; EQS = stress; EQA = awareness; EQM = mood
	Length	Intervention hours per week
Control variables	Gender	Percentage of females in the original sample
	Age	Average age of the original sample
	Occupation	0 = student; 1 = part-time workers; 2 = full-time workers; 3 = unemployed; 4 = mixed
	Motivation	0 = none; 1 = course credit/money; 2 = special reward
	Country	Country of investigation
	Year	Year of publication
	Type publication	0 = published in peer-reviewed journal; 1 = thesis; 2 = other
	Rating	0 = self-rated EI; 1 = peer-rated EI
	CG activities	0 = passive; 1 = waiting list; 2 = placebo
	Randomization	0 = fully randomized; 1 = quasi-randomized; 2 = not randomized
	Dropout	Number of people that did not take part in post-measurement in relation to pre-measurement
Risk of bias (Higgins & Green, 2011)	Sequence	Method to generate allocation sequence
	Concealment	Method to conceal allocation sequence
	Blinding	Methods to blind participants
	Incomplete	Completeness of the reported outcome data
	Other	Any other concerns about bias
	Bias	Risk of bias index: 1 point for every of the 5 categories without concerns (these are added)

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Adetunji

S. O.

Soezin

Margaret

N. N.

(2015). Effects of emotional intelligence and creativity thinking training on improving the emotional intelligence of recidivists in Lagos State, Nigeria. Participatory Educational Research (PER), 2, 11–23. doi:10.17275/per.14.15.2.1

Anderson

J. R.

Schunn

C. D.

(2000). Implications of the ACT-R learning theory: No magic bullets. In Glaser

(Ed.), Advances in instructional psychology (Vol. 5, pp. 1–34). Mahwah, NJ: Erlbaum.

Bar-On

(2006). The Bar-On model of emotional-social intelligence (ESI). Psicothema, 18, 13–25.

Barrick

M. R.

Mount

M. K.

(1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26. doi:10.1111/j.1744-6570.1991.tb00688.x

Bisquerra

Pérez-Escoda

(2007). Las competencias emocionales. Educación XXI, 10, 61–82.

Boyatzis

R. E.

Goleman

Rhee

(2000). Clustering competence in emotional intelligence: Insights from the Emotional Competence Inventory (ECI). In Bar-On

Parker

J. D. A.

(Eds.), The handbook of emotional intelligence: Theory, development, assessment, and application at home, school, and in the workplace (pp. 343–362). San Francisco, CA: Jossey-Bass.

Campo

Laborde

Weckemann

(2015). Emotional intelligence training: Implications for performance and health. In Colombus

A. M.

(Ed.), Advances in psychology research (pp. 75–92). New York, NY: Nova.

*Cejudo

Latorre

(2015). Effects of the Spock videogame on improving emotional intelligence in adolescents. Electronic Journal of Research in Educational Psychology, 13, 319–341. doi:10.14204/ejrep.36.15060

Cherniss

Grimm

L. G.

Liautaud

J. P.

(2010). Process-designed training: A new approach for helping leaders develop emotional and social competence. Journal of Management Development, 29, 413–431. doi:10.1108/02621711011039196

10.

*Clark

K. A.

(2014). The evaluation of an emotions education training on early childhood preservice teachers’ emotion-related awareness and behaviors (Unpublished doctoral dissertation). Auburn University, Auburn, AL.

11.

Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.

12.

*Crombie

D. T.

Lombard

Noakes

T. D.

(2011). Increasing emotional intelligence in cricketers: An intervention study. International Journal of Sports Science and Coaching, 6, 69–86. doi:10.1260/1747-9541.6.1.69

13.

*Dacre Pool

Qualter

. (2012). Improving emotional intelligence and emotional self-efficacy through a teaching intervention for university students. Learning and Individual Differences, 22, 306–312. doi:10.1016/j.lindif.2012.01.010

14.

Daus

C. S.

Ashkanasy

N. M.

(2005). The case for the ability-based model of emotional intelligence in organizational behavior. Journal of Organizational Behavior, 26, 453–466. doi:10.1002/job.321

15.

*Davis

C. M.

(2014). The impact of intervention methods on emotional intelligence (Doctoral dissertation). Retrieved from PsycINFO. (Order No. AAI3563147)

16.

Dean

A. M.

Conte

J. M.

Blankenhorn

T. R.

(2006). Examination of the predictive validity of Big Five personality dimensions across training performance criteria. Personality and Individual Differences, 41, 1229–1239. doi:10.1016/j.paid.2006.04.020

17.

*Di Fabio

Kenny

M. E

. (2011). Promoting emotional intelligence and career decision making among Italian high school students. Journal of Career Assessment, 19, 21–34. doi:10.1177/1069072710382530

18.

Durlak

J. A.

Weissberg

R. P.

Dymnicki

A. B.

Taylor

R. D.

Schellinger

K. B.

(2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432. doi:10.1111/j.1467-8624.2010.01564.x

19.

Egger

Smith

G. D.

Schneider

Minder

(1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.

20.

Fiori

(2009). A new look at emotional intelligence: A dual-process framework. Personality and Social Psychology Review, 13, 21–44. doi:10.1177/1088868308326909

21.

Gignac

G. E.

(2010). Genos emotional intelligence technical manual (2nd ed.). Sydney, Australia: Genos Press.

22.

*Görgens-Ekermans

Delport

Du Preez

(2015). Developing emotional intelligence as a key psychological resource reservoir for sustained student success. SA Journal of Industrial Psychology, 41. doi:10.4102/sajip.v41i1.1251

23.

Grant

Kinman

Alexander

(2014). What’s all this talk about emotion? Developing emotional intelligence in social work students. Social Work Education, 33, 874–889. doi:10.1080/02615479.2014.891012

24.

*Groves

K. S.

McEnrue

M. P.

Shen

(2008). Developing and measuring the emotional intelligence of leaders. Journal of Management Development, 27, 225–250. doi:10.1108/02621710810849353

25.

*Hansenne

Nélis

Feyers

Salmon

Majerus

(2014). Better neuronal efficiency after emotional competences training: An fMRI study. Psychologica Belgica, 54, 328–349. doi:10.5334/pb.av

26.

*Herpertz

Schütz

Nezlek

(2016). Enhancing emotion perception, a fundamental component of emotional intelligence: Using multiple-group SEM to evaluate a training program. Personality and Individual Differences, 95, 11–19. doi:10.1016/j.paid.2016.02.015

27.

Higgins

J. P. T.

Green

(2011). Cochrane handbook for systematic reviews of interventions Version 5.1.0. Retrieved from http://handbook.cochrane.org/

28.

*Hodzic

Ripoll

Bernal

Zenasni

(2015). The effects of emotional competences training on unemployed adults. A longitudinal study. Applied Psychology: Health and Well-Being, 7, 275–292. doi:10.1111/aphw.12048

29.

Hunter

J. E.

Schmidt

F. L.

(2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: SAGE.

30.

Jennings

P. A.

Greenberg

M. T.

(2009). The prosocial classroom: Teacher social and emotional competence in relation to student and classroom outcomes. Review of Educational Research, 79, 491–525.

31.

Joseph

D. L.

Jin

Newman

D. A.

O’Boyle

E. H.

(2015). Why does self-reported emotional intelligence predict job performance? A meta-analytic investigation of mixed EI. Journal of Applied Psychology, 100, 298–342. doi:10.1037/a0037681

32.

Joseph

D. L.

Newman

D. A.

(2010). Emotional intelligence: An integrative meta-analysis and cascading model. Journal of Applied Psychology, 95, 54–78. doi:10.1037/a0017286

33.

*Kasler

Hen

Nov

(2013). Building emotional competence in educators. International Journal of Higher Education, 2, 31–41. doi:10.5430/ijhe.v2n4p31

34.

*Kidwell

Hasford

Hardesty

D. M.

(2015). Emotional ability training and mindful eating. Journal of Marketing Research, 52, 105–119. doi:10.1509/jmr.13.0188

35.

Kirk

B. A.

Schutte

N. S.

Hine

D. W.

(2011). The effect of an expressive-writing intervention for employees on emotional self-efficacy, emotional intelligence, affect, and workplace incivility. Journal of Applied Social Psychology, 41, 179–195. doi:10.1111/j.1559-1816.2010.00708.x

36.

Kirkpatrick

(1996). Great ideas revisited: Techniques for evaluating training programs. Training and Development, 50, 54–59.

37.

*Kotsou

Nélis

Gregoire

Mikolajczak

(2011). Emotional plasticity: Conditions and effects of improving emotional competence in adulthood. Journal of Applied Psychology, 96, 827–839. doi:10.1037/a0023047

38.

*Lange

(2014). The evaluation of a development program directed at the trait emotional intelligence of employees of the Ministry of Justice in northern Namibia (Unpublished doctoral dissertation). University of Namibia, Windhoek, Namibia.

39.

Lindquist

K. A.

Gendron

Feldman Barrett

Dickerson

B. C.

(2014). Emotion perception, but not affect perception, is impaired with semantic memory loss. Emotion, 14, 375–387. doi:10.1037/a0035293

40.

MacCann

Joseph

D. L.

Newman

D. A.

Roberts

R. D.

(2014). Emotional intelligence is a second-stratum factor of intelligence: Evidence from hierarchical and bifactor models. Emotion, 14, 358–374. doi:10.1037/a0034755

41.

Martins

Ramalho

Morin

(2010). A comprehensive meta-analysis of the relationship between emotional intelligence and health. Personality and Individual Differences, 49, 554–564. doi:10.1016/j.paid.2010.05.029

42.

Mayer

J. D.

Caruso

D. R.

Salovey

(2016). The ability model of emotional intelligence: Principles and updates. Emotion Review, 8, 290–300. doi:10.1177/1754073916639667

43.

Mayer

J. D.

Salovey

Caruso

D. R.

(2004). Emotional intelligence: Theory, findings, and implications. Psychological Inquiry, 15, 197–215. doi:10.1207/s15327965pli1503_02

44.

Mayer

J. D.

Salovey

Caruso

D. R.

Sitarenios

(2001). Emotional intelligence as a standard intelligence. Emotion, 1, 232–242. doi:10.1037/1528-3542.1.3.232

45.

Mayer

J. D.

Salovey

Caruso

D. R.

Sitarenios

(2003). Measuring emotional intelligence with the MSCEIT V2.0. Emotions, 3, 97–105.

46.

Mikolajczak

Quoidbach

Kotsou

Nélis

(2009). Les competences emotionnelles [Emotional competencies]. Paris, France: Dunod.

47.

*Nélis

Kotsou

Quoidbach

Hansenne

Weytens

Dupuis

Mikolajczak

(2011). Increasing emotional competence improves psychological and physical well-being, social relationships, and employability. Emotion, 11, 354–366. doi:10.1037/a0021554

48.

*Nélis

Quoidbach

Mikolajczak

Hansenne

(2009). Increasing emotional intelligence: (How) is it possible? Personality and Individual Differences, 47, 36–41. doi:10.1016/j.paid.2009.01.046

49.

Perera

H. N.

DiGiacomo

(2013). The relationship of trait emotional intelligence with academic performance: A meta-analytic review. Learning and Individual Differences, 28, 20–33.

50.

*Pérez-Escoda

Filella

Alegre

Bisquerra

(2012). Developing the emotional competence of teachers and pupils in school contexts. Electronic Journal of Research in Educational Psychology, 10, 1183–1207.

51.

Petrides

K. V.

(2009). Psychometric properties of the Trait Emotional Intelligence Questionnaire (TEIQue). In Stough

Saklofske

D. H.

Parker

J. D. A.

(Eds.), Assessing emotional intelligence: Theory, research and applications (pp. 85–101). New York, NY: Springer.

52.

Petrides

K. V.

Furnham

Frederickson

(2004). Emotional intelligence. Psychologist, 17, 574–577.

53.

Petrides

K. V.

Pita

Kokkinaki

(2007). The location of trait emotional intelligence in personality factor space. British Journal of Psychology, 98, 273–289. doi:10.1348/000712606X120618

54.

R (Version 3.2.5) [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.

55.

*Rene

(2015). Effects of emotional intelligence training on emerging staff and student leaders in a collegiate setting (Unpublished doctoral dissertation). Nova Southeastern University, Fort Lauderdale, FL.

56.

*Reuben

Sapienza

Zingales

(2009). Can we teach emotional intelligence? Unpublished manuscript.

57.

Roberts

R. D.

Zeidner

Matthews

(2001). Does emotional intelligence meet traditional standards for an intelligence? Some new data and conclusions. Emotion, 1, 196–231. doi:10.1037/1528-3542.1.3.196

58.

Ruini

Ottolini

Tomba

Belaise

Albieri

Visani

. . . Fava

G. A.

(2009). School intervention for promoting psychological well-being in adolescence. Journal of Behavior Therapy and Experimental Psychiatry, 40, 522–532. doi:10.1016/j.jbtep.2009.07.002

59.

Ruiz-Aranda

Salguero

J. M.

Cabello

Palomera

Fernández-Berrocal

(2012). Can an emotional intelligence program improve adolescents’ psychosocial adjustment? Results from the INTEMO Project. Social Behavior and Personality, 40, 1373–1380. doi:10.2224/sbp.2012.40.8.1373

60.

Salovey

Mayer

J. D.

(1990). Emotional intelligence. Imagination, Cognition, and Personality, 9, 185–211. doi:10.2190/DUGG-P24E-52WK-6CDG

61.

Sánchez-Álvarez

Extremera

Fernández-Berrocal

(2016). The relation between emotional intelligence and subjective well-being: A meta-analytic investigation. The Journal of Positive Psychology, 11, 276–285. doi:10.1080/17439760.2015.1058968

62.

Schutte

N. S.

Malouff

J. M.

Thorsteinsson

E. B.

(2013). Increasing emotional intelligence through training: Current status and future directions. The International Journal of Emotional Education, 5, 56–72.

63.

Schutte

N. S.

Malouff

J. M.

Thorsteinsson

E. B.

Bhullar

Rooke

S. E.

(2007). A meta-analytic investigation of the relationship between emotional intelligence and health. Personality and Individual Differences, 42, 921–933. doi:10.1016/j.paid.2006.09.003

64.

Sin

N. L.

Lyubomirsky

(2009). Enhancing well-being and alleviating depressive symptoms with positive psychology interventions: A practice-friendly meta-analysis. Journal of Clinical Psychology, 65, 467–487. doi:10.1002/jclp.20593

65.

Slaski

Cartwright

(2003). Emotional intelligence training and its implications for stress, health and performance. Stress and Health: Journal of the International Society for the Investigation of Stress, 19, 233–239. doi:10.1002/smi.979

66.

*Vesely

A. K.

Saklofske

D. H.

Nordstokke

D. W.

(2014). EI training and pre-service teacher wellbeing. Personality and Individual Differences, 65, 81–85. doi:10.1016/j.paid.2014.01.052

67.

Viechtbauer

(2010). Conducting meta-analysis in R with the metafor package. Journal of Statistical Software, 36, 1–48. doi:10.18637/jss.v036.i03

68.

Widen

S. C.

(2013). Children’s interpretation of facial expressions: The long path from valence-based to specific discrete categories. Emotion Review, 5, 72–77. doi:10.1177/1754073912451492

69.

*Wing

J. F.

Schutte

N. S.

Byrne

(2006). The effect of positive writing on emotional intelligence and life satisfaction. Journal of Clinical Psychology, 62, 1291–1302. doi:10.1002/jclp.20292

70.

Yalcin

B. M.

Karahan

T. F.

Ozcelik

Igde

F. A.

(2008). The effects of an emotional intelligence program on the quality of life and well-being of patients with Type 2 diabetes mellitus. The Diabetes Educator, 34, 1013–1024. doi:10.1177/0145721708327303

71.

*Zijlmans

L. J. M.

Embregts

P. J. C. M.

Gerits

Bosman

A. M. T.

Derksen

J. J. L.

(2011). Training emotional intelligence related to treatment skills of staff working with clients with intellectual disabilities and challenging behaviour. Journal of Intellectual Disability Research, 55, 219–230. doi:10.1111/j.1365-2788.2010.01367.x

72.

*Zijlmans

L. J. M.

Embregts

P. J. C. M.

Gerits

Bosman

A. M. T.

Derksen

J. J. L.

(2015). The effectiveness of staff training focused on increasing emotional intelligence and improving interaction between support staff and clients. Journal of Intellectual Disability Research, 59, 599–612. doi:10.1111/jir.12164