Psychometric Properties of the Environmental Reward Observation Scale: Study on Its Internal Structure,Factor Invariance,and Method Effect Associated With Its Negative Items

Abstract

The aim of this study was to assess the factorial structure of the scale, the method's effect associated with its negative items, its temporal invariance, and factorial invariance according to sex. For this purpose, three samples were collected, an initial sample of 200 participants, a second sample of 461 participants and a third sample of 107 participants; making a total of 768 Peruvian university students. Other instruments were applied together with the EROS scale in order to measure satisfaction with life, anxiety, stress and depression. Regarding the results, in the initial sample it was found that the original scale containing positive and negative items does adequately fit the data (RMSEA = .19; CFI = .77; TLI = .71) and also evidence was found supporting the existence of a methodological effect associated with the negative items. It was also found that version B of the scale which only has positive items data fits the data (RMSEA = .13; CFI = .96; TLI = .95). In the second sample it was found that version B still had a good fit to the data in a larger sample (RMSEA = .07; CFI = .98; TLI = .98). In addition, it was found that the scale can be considered invariant according to sex and presents validity based on other constructs. In the third sample it was found that the test-retest reliability of the scale was adequate (.70 [CI95% .593–.788]) and also evidence was found in favor of the temporal invariance of the scale. It is concluded that the scale formed only by positive items presents more robust psychometric properties and constitutes a better alternative to measure the level of reward provided by the environment.

Keywords

Reverse worded items construct validity factorial invariance test-retest reliability Environmental Reward Observation Scale

Introduction

Behavioral Activation is a therapy set out in the third generation of cognitive-behavioral therapy whose main objective is to carry out activities that increase the contact with the contingencies of the environment, and thus feel reinforced by the environment (Hayes, 2016), which is based on the assumption that the behavioral change causes a change in state of life, in perceived rewards, in solution of problems (Pérez-Alvares, 2014) and in emotions (Barraca & Pérez-Álvarez, 2015). In addition, it is a therapy that uses a version of functional analysis and a philosophy based on functional contextualism to explain human behavior (Kanter et al., 2011). Its effectiveness has been demonstrated in several meta-analysis studies (Barth et al., 2013; Mazzucchelli et al., 2009, 2010; Zabihi et al., 2020) and in randomized clinical studies (Dimidjian et al., 2006; Gawrysiak et al., 2009; Hopko et al., 2011). Furthermore, it is as effective as pharmacological treatment (paroxetine) with less treatment abandonment or refusal (Dimidjian et al., 2006) and is more effective than Cognitive Behavioral Therapy (Richards et al., 2016). Also, several studies have demonstrated its efficacy in group interventions (Mahen et al., 2019; Simmonds-Buckley et al., 2019) and through mobile applications (Ly et al., 2014).

Since the development of this treatment protocol, one of the main needs was the development of original evaluation instruments that would allow for an effective assessment of the main objectives of the therapy: to measure the increase in behavior and to measure the access to positive reinforcement (Martell et al., 2013). However, the absence of instruments with robust psychometric properties may make it difficult to evaluate these objectives and to objectively assess the effectiveness of therapy.

Responding to this need, Armento and Hopko (2007) developed the Environmental Reward Observation Scale (EROS), which is a brief ten-item instrument to evaluate increased behavior and access to positive environmental reinforcement. The instrument has been widely used in randomized clinical studies (Fernández-Rodríguez et al., 2020; Mira et al., 2019; Vázquez et al., 2019) and explanatory studies (Aoki et al., 2019; Kern et al., 2019; Maitland et al., 2019; Martínez-Vispo et al., 2019; Otero et al., 2020).

Regarding the psychometric properties of the EROS scale, the original study conducted with university students (Armento & Hopko, 2007) showed that a one-dimensional model fits well with the data (RMSEA = .06; GFI = .92; NFI = .90) with good reliability in its scores (α = .88). Furthermore, EROS has several adaptations. Its French version (Wagener & Blairy, 2015) showed good evidence based for internal structure (RMSEA = .06; GFI = .99; NFI = .99) and reliability of scores (α = .89). In the Spanish version (Barraca & Pérez-Álvarez, 2010), its internal structure was reviewed by means of an Exploratory Factor Analysis (EFA) showing that the items formed two factors, although the first factor explained most of the variance (45.7%). Regarding its reliability, it showed an adequate level of internal consistency (α = .86). In Colombia (Valderrama-Díaz et al., 2016) the EFA showed that the items formed two factors, where the first factor explained most of the variance (46.8%) and the reliability of the scores was good (α = .87). Additionally, there is a Spanish version only for breast cancer survivors (Fernández-Rodríguez et al., 2020), where the evidence suggests a non-adjustment of the one-dimensional model (RMSEA = .116 [IC90% .096–.137); in spite of that, its reliability was excellent (α = 91).

With respect to the psychometric studies carried out in Spanish-speaking populations, there are several methodological limitations, such as the use of the principal components method as an EFA estimator, which only takes into account the total variance and does not differentiate between the common variance and the error variance to form the factors (Watkins, 2018). In addition, the use of Kaiser's rule overestimates the number of factors as a function of the number of items, that is, a greater number of items increases the number of factors estimated (Watson, 2017). Additionally, the use of varimax rotation is not appropriate since it assumes that the factors are not related to each other (Lloret et al., 2017). Furthermore, it is worth noting that only one study (Fernández-Rodríguez et al., 2020) carried out a Confirmatory Factorial Analysis (CFA) to evaluate the internal structure of the scale. However, it was carried out on a very specific sample of women, which limits the generalizability of the results. Therefore, it can be seen that there is a lack of conclusive psychometric studies to confirm the internal structure of the scale in the Spanish-speaking population.

It is important to mention that all the psychometric studies carried out use Cronbach's Alpha coefficient (α) as a reliability indicator and there is no evidence of the use of compound coefficients that are typically used in factorial models and that are resistant to tau-equivalence non-compliance (Viladrich et al., 2017). Nor has psychometric evidence been found on the test-retest reliability of the scale, which is important for evaluating the stability of scores over time.

Furthermore, no psychometric studies have been found that assure the factorial invariance of the scale according to the sex of those evaluated; several studies have found that women are more sensitive to reinforcements and rewards, because they obtain a higher score in behavioral events and participate in more behavioral domains than men (Ryba & Hopko, 2012; Tull et al., 2010). Nor has psychometric evidence been found on the temporal invariance of the scale that guarantees that the changes observed over time are the product of a real change in the level of the construct.

Another important aspect is the lack of psychometric studies that evaluate the possible methodological effect associated with the presence of negative items in the scale, since these types of items make it difficult to adequately understand the items (van Sonderen et al., 2013), cannot control for acquiescence bias in the factor structure (Savalei & Falk, 2014), and generally create an additional factor in the structure of the scale (Lindwall et al., 2012).

Finally, the review of previous psychometric studies shows that SEM has not been used to assess validity based on the relationship to other constructs. This is even though it is a very useful tool that provides information about the relationship between the observed variables (items) and the constructs they represent and the relationship between the measured constructs themselves (McCoach et al., 2013). For this study, according to the literature review, EROS was expected to have a positive relationship with life satisfaction (Hopko et al., 2011) and a negative relationship with depression and anxiety (Safra et al., 2019; Vázquez et al., 2019).

Faced with these methodological and psychometric limitations, this study has the following objectives: (a) evaluate the internal structure of the scale through CFA, (b) evaluate the method’s effect associated with negative items, (c) evaluate reliability using more robust indicators, (d) evaluate temporal reliability, (e) evaluate temporal invariance, (f) evaluate factor invariance according to sex, and (g) evaluate validity related to other constructs.

Method

Participants

For this study, three samples were collected, starting with an initial sample of 200 participants in order to compare two versions of the test. Version A involved 100 students (37% male and 63% female) aged 17–29 (M = 20.8, SD = 2.5) and version B involved 100 students (62% male and 38% female) aged 17–28 (M = 20.4, SD = 2.1). In both cases they were university students attending a private university in Lima, Peru. The sample size used for both versions (A and B) is sufficient considering that it is a one-dimensional model with ten indicators whose factor weights are equal to or greater than .50 (Wolf et al., 2013).

To confirm the psychometric properties of version B, a second sample of 461 university students (58.4% male and 41.6% female) between 17 to 28 years old (M = 20.7, SD = 2.4) was collected, where no significant differences were found between the ages of males (M = 20.2, SD = 2.4) and females (M = 19.9, SD = 2.3) and the effect size was low (t₍₄₅₉₎ = 1.06, p = .288, d = .24, CI95% −.20–.68). Finally, to evaluate the temporal invariance of version B, a third sample was collected which consisted of 107 university students of both sexes (21.5% male and 78.5% female) from 18 to 29 years old (M = 21.2, SD = 2.5). The time between the first and second application was 36 days.

The following inclusion criteria were used for the collection of the three samples: (a) informed consent of the participants, (b) ability to read and write in Spanish, (c) not older than 40 years of age, and (d) being enrolled in a university degree program. The following exclusion criteria were also used: (a) failure to complete all tests, and (b) failure to complete the second assessment in the temporal invariance.

Measures

Environmental reward observation scale (EROS)

This 10-item instrument was developed by Armento and Hopko (2007) to measure the degree of reward provided by the environment. The version adapted to Spanish by Barraca and Pérez-Álvarez (2010) was used for this study, where it showed a high reliability (α = .86). The items have four response categories that range from “totally disagree” (1) to “totally agree” (4), where a higher score shows greater experience of the environment.

Depression, anxiety and stress scale (DASS-21)

The short version of 21 items adapted to Spanish by Bados et al. (2005) was used, where it showed adequate reliability and validity indexes. Also, other studies have corroborated the existence of a model of three related factors (Patias et al., 2016; Wang et al., 2016). The scale has three dimensions that evaluate the presence and intensity of the negative emotional states of depression, anxiety and stress. In addition, each dimension is made up of 7 items which have four categories of response ranging from: “Does not apply to me” (0) to “Applies a lot or most of the time” (3). The present study also evaluated its psychometric properties, where it showed adequate reliability indices for the depression (α = . 93; ω = .91), anxiety (α = . 91; ω = .88) and stress components (α = . 89; ω = .87). In addition, it showed that the model of three related factors presented adequate indexes of adjustment (CFI = .98; TLI = .98; RMSEA = .067, SRMR = .043).

Diener's satisfaction with life scale (SWLS)

The short version of five items was used for the study, adapted to Spanish by Atienza et al. (2000) where it showed adequate reliability (α = .84) and validity indexes based on the internal structure (GFI = .98; NFI = .99; NNFI = .99; RMR = .02). Also, different studies in Latin America evaluated the metric quality of the instrument in university students, evidencing the existence of a unifactorial model (Oliver et al., 2018; Padrós et al., 2015). Regarding the structure of the scale, the five items have five response categories ranging from: “Strongly disagree” (1) to “Strongly agree” (5), where a higher score on the scale evidences greater satisfaction with life. In the present study, it showed adequate indices of reliability (α = . 90; ω = 91) and validity based on internal structure (CFI = .99; TLI = .99; RMSEA = .088; SRMR = .018).

Procedure and statistical analysis

The study was approved by an ethics committee of the Center for Health Research and Innovation (CIISA) of the Faculty of Health Sciences at a private university in Lima, Peru and following the rules of the Declaration of Helsinki (World Medical Association, 2013) did not represent a risk for participants. A cross-sectional design was used for data collection and the instruments were applied collectively in the classrooms. In each case, three fourth-year psychology students were trained in data collection before applying the instruments. During the application process, anonymity and confidentiality of the results were ensured, where the university students signed an informed consent, the objectives of the study were explained and doubts regarding the procedure were resolved. The average time for answering the instruments was 20 minutes.

The process followed to study the psychometric properties of the scale is made up of three stages: (a) content-based validity, (b) study of method effect, and (c) confirmation of psychometric properties (see Figure 1).

Figure 1.

Stages of the EROS scale adaptation process.

Content based validity

For version A of the scale, five judges participated: four psychologists with the academic degree of Master in Psychology and one Doctor in Psychology. Similarly, for version B, five judges participated: four psychologists with the academic degree of Master in Psychology and one Doctor in Psychology. All the judges had at least 15 years of professional experience. To evaluate the structure and content of the items, all the judges used four criteria: (a) relevance, (b) representativeness, (c) clarity, and (d) context of the items. Aiken's V coefficient (Aiken, 1980) was used for its quantification and an ad hoc program in MS Excel© format (Ventura-León, 2019) was used for its calculation. Then thirteen psychology students in their last year of studies evaluated the clarity of the items in both versions (A and B).

Study on the method’s effect

As shown in Figure 2, four models were proposed, the A₁ model maintains the original approach with positive and negative items, the A₂ model adds a specific factor for negative items and the A₃ model adds a specific factor for positive items. A B model of the scale was also tested, where the 10 items are all written in a positive way.

Figure 2.

Models of the A and B versions of the scale.

The Diagonally Weighted Least Squares with Mean and Variance corrected (WLSMV) estimator was used to evaluate these models since the items are all ordinal (Brown, 2015). The chi-square test (χ2), the RMSEA (Root Mean Square Error of Approximation) and the SRMR (Standardized Root Mean Square Residual) were used to evaluate model fit, in which case values less than .05 indicate good fit, and between .05 and .08 are considered acceptable (Kline, 2015). In addition, the CFI (Comparative Fit Index), TLI (Tucker-Lewis Index) was used for these cases, values higher than .95 indicate a good fit and higher than .90 an acceptable fit (Schumacker & Lomax, 2015). The WRMR (Weighted Root Mean Square Residual) index was also used where values below 1.0 are adequate (DiStefano et al., 2018). For bi-factor models, the hierarchical omega coefficient (ω_h) and the total omega (ω_t) were used to calculate reliability (Zinbarg et al., 2005). For the one-dimensional model, the omega coefficient was used (McDonald, 1999). All statistical analyses were performed with the “lavaan” package (Rosseel, 2012) using the RStudio environment (RStudio Team, 2018) for R (R Core Team, 2019).

Confirmation of psychometric properties

For the descriptive analyses of the items (mean [M], standard deviation [SD], asymmetry [g1] and kurtosis [g2]) the program SPSS 22.0 for Windows was used. For Confirmatory Factor Analysis (CFA) the package “lavaan” was used (Rosseel, 2012), for factor invariance models of the scale the package “semTools” was used (Jorgensen et al., 2018) and for evaluating test-retest reliability the package “irr” was used (Gamer et al., 2019). In all cases the RStudio environment (RStudio Team, 2018) was used for R (R Core Team, 2019).

As part of the confirmatory factor analysis, the WLSMV estimator (Weighted Least Squares with Mean and Variance corrected) was used and the same adjustment indicators made in the pilot test were taken into account. In addition, to evaluate the relevance of the modification indexes (MI) in the model, the Saris et al. (2009) method was used, where “M” represents that the parameter is wrongly specified, “NM” represents that the parameter is not wrongly specified, “EPC: M” represents a misspecification using the expected change in the parameter and “EPC: NM” represents that there is no misspecification using the expected change in the parameter.

To evaluate the internal consistency of the scale, Cronbach's alpha coefficient (Cronbach, 1951) and omega coefficient (McDonald, 1999) were used, where a value of ω > .80 is adequate (Raykov & Hancock, 2005). The H coefficient was also used (Mueller & Hancock, 2001) because it is robust for correlated errors. A value greater than .70 is acceptable (Geldhof et al., 2014). In addition, to evaluate the test-retest reliability of the scale, the Intraclass Correlation Coefficient (ICC) was used, for which a mixed effects bidirectional model (ICC3) was used, for single measurement and the type of relationship was in absolute agreement (Koo & Li, 2016). A value greater than .60 is considered acceptable (Fleiss et al., 2013). Finally, in order to evaluate the invariance of the scale according to sex and time invariance, a sequence of hierarchical variance models was proposed, which were increasingly restrictive. To this end, first the configural invariance (reference model) was tested, followed by the metric invariance (equality of factorial loads), the scalar invariance (equality of factorial loads and intercepts) and finally the strict invariance was tested (equality of factorial loads, intercepts and residuals). Two strategies were used to compare the sequence of models: first a formal statistical test was used, for which the chi-square difference (Δχ²) was used where non-significant values (p > .05) suggest invariance between groups. Second, a modeling strategy was employed, using differences in the CFI (ΔCFI) where values less than <.010 evidence model invariance between groups (Chen, 2007).

Finally, to evaluate the relationship of the EROS scale with other variables, a structural equation model was proposed, where the degree of reward provided by the environment (EROS) is related to the level of stress, anxiety, depression and satisfaction with life. To estimate the model, the WLSMV estimator (Weighted Least Squares with Mean and Variance corrected) was used and the same adjustment indicators made in the confirmatory factor analysis were taken into account.

Results

Content based validity

Table 1 shows the modifications made following the suggestions of the judges and students in the items of version A and version B. In item 2 the expression “experiencias que vivo” (“experiences I live”) was used in both versions and in version B the item was written as a positive. In item 4, the expression “encontrar motivos” (“find reasons”) was added in both versions. In item 5, the expression “A comparación de mi” (“Compared to me”) was added at the beginning of the item and “vidas más gratificantes” (“more rewarding lives”) at the end of it for version A, in version B the item was written as a positive, keeping the content of the item. In item 6, the wording was kept in version A and in version B it was written as a positive. In item 7, the term “aficiones” (“hobbies”) was changed to “actividades” (“activities”) in both versions, and in version B the item was written as a positive. In item 9, the wording in version A was kept and in version B it was written as a positive. Finally, in item 10 the expression “actividades que realizo” (“activities I carry out”) was added in both versions.

Table 1.

Content validity of both scale versions.

Spanish adaptation	Version A	V_Rel	V_Rep	V_Cla	V_Con	Version B	V_Rel	V_Coh	V_Cla	V_Con
1. Muchas actividades de mi vida son agradables	Muchas actividades de mi vida son agradables	.93	.93	.93	.93	Muchas actividades de mi vida son agradables	.93	.93	.93	.93
2. Últimamente, me he dado cuenta de que las cosas que vivo me hacen infeliz	Últimamente, me he dado cuenta que las experiencias que vivo, me hacen infeliz	.93	.93	.87	.80	Últimamente, me he dado cuenta que las experiencias que vivo, me hacen feliz	.93	.87	.87	.80
3. En general, estoy muy satisfecho con la forma en que empleo mi tiempo	En general, estoy muy satisfecho con la forma en que utilizo mi tiempo	1.00	.87	.87	.93	En general, estoy muy satisfecho con la forma en que utilizo mi tiempo	.93	.87	.80	.93
4. Me resulta fácil encontrar con qué disfrutar en la vida	Me resulta fácil encontrar motivos para disfrutar de la vida	.93	.93	.73	.73	Me resulta fácil encontrar motivos para disfrutar de la vida	.93	.93	.73	.73
5. Otras personas parecen tener vidas más plenas	A comparación de mí, otras personas parecen tener vidas más gratificantes	.87	.87	.73	.73	Mi vida es tan gratificante como la de otras personas	.80	.87	.80	.73
6. Ya no me resultan gratificantes actividades que antes me agradaban	Ya no me resultan gratificantes actividades que antes me agradaban	1.00	.87	.93	.93	Las actividades que antes hacía me siguen resultando gratificantes	.93	.87	.93	.93
7. Desearía encontrar más aficiones que me divirtiesen	Desearía encontrar más actividades que me diviertan	.87	.87	.73	.73	Encuentro actividades que me divierten	.80	.87	.80	.73
8. Estoy satisfecho con mis logros	Estoy satisfecho con mis logros	.93	.80	.73	.80	Estoy satisfecho con mis logros	.93	.80	.73	.80
9. Mi vida es aburrida	Mi vida es aburrida	1.00	1.00	.87	.93	Mi vida es interesante	.93	1.00	.87	.93
10. Las actividades en que tomo parte normalmente salen bien	Las actividades que realizo normalmente salen bien	.80	.93	.80	.73	Las actividades que realizo normalmente salen bien	.73	.93	.87	.80

Nota. V_Rel = Relevance; V_Rep = Representativity; V_Cla = Clarity; V_Cont = Context.

Study on the method’s effect

Table 2 shows that the one-dimensional structure in the A₁ model presents inadequate indexes of adjustment to the data (RMSEA = .19; CFI = .77; TLI = .71; WRMR = 1.51), where negative loads associated with the inverse items are observed. On the other hand, the A₂ model, where a specific factor was added for negative items, presents better adjustment indexes (RMSEA = .10; CFI = .95; TLI = .93; WRMR = .73). In addition, it can be seen that the reliability of the A₂ model, considering only the general factor as true variance, presents a very low level (ωH = .17). On the other hand, when the specific factor is included as part of the true variance, the reliability of the model increases notably (ω_t = .72). This change could be attributed to the method’s effect associated with the negative items. Following this line of thinking, the A₃ model was evaluated, where a specific factor was added for the positive items. As shown in Table 2, this model does not present adequate indexes of adjustment to the data (RMSEA = .14; CFI = .89; TLI = .84; WRMR = .90), evidencing that the presence of positive items in the scale does not impact the model's adjustment. These results show that there is a methodological effect associated with the presence of negative items in the scale.

Table 2.

Comparison of adjustment indicators of both scale versions.

Models	Adjustment rates
Models	X²	gl	p	X²/gl	RMSEA	SRMR	CFI	TLI	WRMR
Model A₁	166.77	35	.00	4.76	.19	.14	.77	.71	1.51
Model A₂	59.23	30	.00	1.97	.10	.06	.95	.93	.73
Model A₃	91.76	30	.00	3.05	.14	.08	.89	.84	.90
Model B	99.93	35	.00	2.85	.13	.06	.96	.95	.86
	Standardized estimates of parameters
Items	Model A₁			Model A₂		Model A₃		Model B
Item 2 (−)	−.58			−.37	.48	−.63		.82
Item 5 (−)	−.59			−.20	.75	−.69		.80
Item 6 (−)	−.74			−.41	.72	−.83		.83
Item 7 (−)	−.54			−.18	.75	−.66		.78
Item 9 (−)	−.65			−.55	.36	−.72		.88
Item 1 (+)	.66			.73		.49	.49	.78
Item 3 (+)	.55			.69		.23	.75	.67
Item 4 (+)	.33			.40		.14	.42	.69
Item 8 (+)	.73			.82		.48	.63	.78
Item 10 (+)	.67			.76		.32	.73	.86

Note: Reliability level of the A₂ model: ω_H = .17; ω_t = .72 Reliability level of model B (ω) = .91.

In light of this, a B model was proposed, where all the items are written in positive way. As shown in Table 2, this model has adequate adjustment indexes (RMSEA = .13; CFI = .96; TLI = .95; WRMR = .86) and has a high level of reliability (ω = .91) which is higher than the A₂ model. It is also noteworthy that the items in model B show a higher factorial weight than in the other models. Taking into account these findings, the B model was chosen for the final version in the process of adaptation of the scale.

Confirmation of psychometric properties

Descriptive analysis of the scale items

Table 3 shows that item 7 has the highest average score in the total sample (M = 3.17) and similarly in the group of men (M = 3.15) and women (M = 3.20). It is also observed that item 3 presents the lowest average score in the total sample (M = 2.78) and in the different groups of men (M = 2.80) and women (M = 2.77). In addition, it can be seen that the items present adequate indices of asymmetry and kurtosis (±1.5) in the total sample and in all the specific groups.

Table 3.

Descriptive analysis of the items.

Items	Total sample (n = 461)				Males (n = 269)				Women (192)
Items	M	SD	g1	g2	M	SD	g1	g2	M	SD	g1	g2
E1	3.13	.76	−.92	1.04	3.11	.80	−.98	1.02	3.17	.71	−.77	.91
E2	3.12	.72	−.76	.88	3.09	.76	−.82	.87	3.15	.68	−.60	.72
E3	2.78	.82	−.28	−.42	2.80	.85	−.26	−.58	2.77	.77	−.32	−.15
E4	3.01	.73	−.66	.66	2.98	.78	−.66	.37	3.05	.66	−.59	1.08
E5	2.95	.75	−.43	−.00	2.91	.77	−.38	−.16	3.01	.72	−.50	.31
E6	2.99	.74	−.57	.34	2.97	.76	−.61	.36	3.01	.72	−.50	.31
E7	3.17	.71	−.76	.81	3.15	.74	−.75	.65	3.20	.69	−.77	1.11
E8	2.99	.81	−.51	−.21	2.95	.84	−.45	−.42	3.06	.76	−.59	.17
E9	3.05	.78	−.64	.20	3.05	.80	−.65	.07	3.06	.74	−.62	.44
E10	2.99	.66	−.66	1.33	2.96	.70	−.65	.92	3.03	.59	−.60	2.03

Note: M=Mean; SD=Standard Deviation; g1= Skewness; g2= Kurtosis.

Validity based on internal structure

It can be seen in Table 4 that the proposed one-dimensional model presents adequate adjustment indexes (CFI = .98; TLI = .98; RMSEA = .075). It can also be seen that the factorial weight of the latent variable with each one of its observed variables is high and significant (see Figure 3). Furthermore, following Saris et al. (2009) method to evaluate the relevance of the modification indexes (MI) in the model and according to the item content analysis, a correlation was specified between the errors of items 1 and 2 (.34), 4 and 6 (.25), and 9 and 10 (.33).

Table 4.

One-dimensional model fit rates and invariance models by sex.

One-dimensional model	χ²	df	p	TLI	IFC	SRMR	RMSEA	Δχ²	Δdf	p	ΔCFI
Total sample	114.24	32	.000	.98	.99	.028	.075	—	—	—	—
According to sex
Men	81.27	32	.000	.98	.99	.031	.076	—	—	—	—
Women	90.39	32	.000	.97	.98	.042	.089	—	—	—	—
Configural	171.23	64	.000	.98	.98	.039	.085	—	—	—	—
Metric	172.74	73	.000	.98	.98	.039	.077	7.17	9	.618	.001
Scalar	170.30	92	.000	.98	.98	.039	.061	13.36	19	.819	.007
Strict	144.02	93	.000	.99	.99	.039	.049	.34	1	.557	.004
According to time
Test	56.13	32	.005	.96	.97	.062	.084	—	—	—	—
Retest	57.98	32	.003	.96	.97	.068	.977	—	—	—	—
Configural	113.20	64	.000	.96	.97	.070	.085	—	—	—	—
Metric	108.35	73	.005	.97	.98	.071	.068	4.82	9	.849	.007
Scalar	118.87	89	.019	.98	.98	.073	.056	13.47	16	.637	.003
Strict	112.05	90	.058	.98	.98	.073	.048	.29	1	.586	.004

Figure 3.

Confirmatory factor analysis of the scale.

Factorial invariance according to sex and time invariance of the scale

Table 4 shows the adjustment indexes for the sequence of invariance models proposed and its difference with the least restrictive model in the sequence.

Regarding factor invariance by sex, the configural model was evaluated first, which fit the data well (TLI = .98; CFI = .98; RMSEA = .085; SRMR = .039) and can therefore be used as a reference model for the evaluation of the following models. Next, the metric invariance was evaluated, where it can be seen that only the RMSEA index improved slightly (TLI = .98; CFI = .98; RMSEA = .077; SRMR = .039). Also, it can be seen that the differences in chi-square were statistically not significant (Δχ² = 7.17; p > .05) and there were no important changes when comparing both models (ΔCFI = .001). Therefore, the metric invariance of the scale is affirmed. After evaluating the scalar invariance, where only the RMSEA index improved again (TLI = .98; CFI = .98; RMSEA = .061; SRMR = .039), no significant differences were observed between the two models (Δχ² = 13.36; p > .05) and no important changes are seen in regards to the metric model either (ΔCFI = .003). Finally, strict invariance was calculated, where the adjustment indices improved slightly (TLI = .99; CFI = .99; RMSEA = .049; SRMR = .039), no significant differences are observed between the two models (Δχ² = .34; p > .05), and no important changes are seen with respect to the scalar model (ΔCFI = .003). Therefore, the scale has proven to be strictly invariant for both groups (men and women). Similarly, the scale also showed evidence of being invariant over time, in the invariance models proposed: metric (ΔCFI = .007), scalar (ΔCFI = .003) and strict invariance (ΔCFI = .004).

Scale reliability

The scale evidences adequate levels of reliability since it presents a high omega coefficient (ω = .93 [IC95% = .92–.94]), a Cronbach’s alpha coefficient that can be considered excellent (α = .93 [IC95% = .92–.94]) and an adequate H coefficient (.94). Furthermore, the ICC for test-retest reliability was adequate (.70 [CI95% .593–.788]).

Validity based on relationship to other variables

Based on the literature review, a SEM model was proposed to evaluate the latent relationship between the EROS scale and the levels of stress, anxiety, depression and life satisfaction. As shown in Table 5, the structural model presents adequate adjustment indexes (RMSEA = .03; CFI = .96; TLI = .96) and the measurement models are adequately represented by their items.

Table 5.

Explanatory model.

Structural model
X²	gl	p	X²/gl	RMSEA	IQ 90%.	SRMR	IFC	TLI
828.14	581	.00	1.43	.03	.026–.035	.04	.96	.96
Measurement models
EROS		Stress		Anxiety	Depression		Satisfaction
λ (item)		λ (item)		λ (item)	λ (item)		λ (item)
.62 (E1)		.42 (D1)		.48 (D2)	.71 (D3)		.88 (S1)
.66 (E2)		.67 (D6)		.64 (D4)	.70 (D5)		.80 (S2)
.70 (E3)		.71 (D8)		.74 (D7)	.76 (D10)		.87 (S3)
.75 (E4)		.77 (D11)		.73 (D9)	.79 (D13)		.72 (S4)
.78 (E5)		.74 (D12)		.79 (D15)	.77 (D16)		.64 (S5)
.69 (E6)		.74 (D14)		.78 (D19)	.76 (D17)
.74 (E7)		.75 (D18)		.74 (D20)	.76 (D21)
.73 (E8)
.75 (E9)
.68 (E10)

Note: λ: Factorial load.

Figure 4 shows that the degree of reward provided by the environment is negatively related to the level of depression (.34; p < .01), anxiety (.22; p < .01) and stress (.21; p < .01). Furthermore, it has a positive relationship with the degree of satisfaction with life (.51; p < .01). Taking into account these results, it can be concluded that the scale presents validity related to other constructs.

Figure 4.

Model of latent relationships.

Discussion

The aim of this study was to adapt and evaluate the psychometric properties of the Spanish version of the EROS scale in Peruvian university students, because it is one of the most used instruments in the framework of behavioral activation (BA) as a treatment for depression (Barth et al., 2013; Richards et al., 2016) and other mental health problems (Fernández & Mairal, 2017; Hirayama et al., 2019).

In the initial sample, four models were tested, three models (A₁, A₂, A₃) that maintain the original approach of positive and negative items; and a B model where all items were written in a positive way. It was found that the original A₁ model presents inadequate adjustment indexes (RMSEA = .19; CFI = .77; TLI = .71; WRMR = 1.51), although this result does not coincide with what was found in the original study by Armento and Hopko (2007) where they evidenced that the one-dimensional model did present adequate adjustment indexes (RMSEA = .06; GFI = .92; NFI = .90) and neither does it coincide with the psychometric study carried out in France (Wagener & Blairy, 2015) where it also showed adequate fit indexes (RMSEA = .06; GFI = .99; NFI = .99).

This difference in model fit could be attributed to cultural differences between countries. People living in highly individualistic countries prioritize autonomy and self-realization while people living in less individualistic countries prioritize social interaction and cohesion (Hofstede et al., 2010). In addition, less individualistic countries show greater empathic concern, kindness, life satisfaction, pro-social behavior (Chopik et al., 2017) and lower levels of loneliness (Heu et al., 2019). All of this can influence the way in which people perceive and value the degree of reward that their environment provides; that is, the cultural aspect can condition the response to the items and affect the factor structure of the scale between countries with marked individualism, such as the United States (IC = 91) and France (IC = 71) and more collectivist countries such as Peru (IC = 16). The classification of countries according to their level of individualism was based on the information provided by (Hofstede Insights, 2020).

Regarding the adaptation of the instrument in less individualistic countries such as Spain (compared with the rest of the European countries) and Latin America, it was found that in the Spanish adaptation (Barraca & Pérez-Álvarez, 2010) the Exploratory Factor Analysis (EFA), showed the possible existence of two factors, where items 5 (.31), 6 (.28), 7 (.65) and 10 (-.34) entered in both factors, most of which are written in a negative way. Similarly, in the adaptation to Colombia (Valderrama-Díaz et al., 2016) the EFA showed the existence of two factors that explained 56.9% of the accumulated variance, and again item 7 (.47) showed a high factorial weight in both factors. In another study of adaptation for women survivors of breast cancer in Spain (Fernández-Rodríguez et al., 2020), the one-dimensional model presented adjustment problems (RMSEA = .116) and the values of the Unidimensional Congruence (UniCo = .89) and Explained Common Variance (ECV = .81) did not support the presence of a one-dimensional model in the data (UniCo > .95; ECV > .85).

These adjustment problems observed in the present and previous studies could be due to the method’s effect caused by the presence of inverse items, since these can form additional factors not associated with the construct (Brown, 2015; Tomás et al., 2013; Zhang et al., 2016). Taking this into account, the A₂ model was tested where a second specific factor was added only for negative items (2, 5, 6, 7 and 9). This model improved the fit indexes (RMSEA = .10; CFI = .95; TLI = .93; WRMR = .73) and the total omega coefficient drastically and showed that including the specific factor as part of the true variance notably improves the reliability of the model (ω_t = .72). This was not the case in the A₃ model where including a specific factor for positive items did not contribute to the fit indexes of the model (RMSEA = .14; CFI = .89; TLI = .84; WRMR = .90). So these findings confirm the existence of method bias associated with negative items.

To overcome this limitation, the B model of the scale was proposed with all positively phrased items, where the results showed adequate adjustment indexes in their majority (RMSEA = .13; CFI = .96; TLI = .95; WRMR = .86) and a better level of reliability (ω = .91) than the previous models. In addition, the items in model B show a higher factorial weight than in the other models.

In the large sample, the B model of the EROS scale continued to show adequate adjustment indicators (RMSEA = .07; SRMR = .02; CFI = .98; TLI = .98; WRMR = .78) and all items showed a high factorial weight (λ>.70) evidencing that negative items transformed to positive continue to adequately measure the degree of reward provided by the environment. However, some correlated errors were found, which can be attributed to similar conceptual content among the items (Brown, 2015). In addition, the analysis of the modification indexes (MI) following the method of Saris et al. (2009) showed that it is pertinent to include these correlated errors in the model.

Regarding items 1 (“Many activities in my life are pleasant”) and 2 (“Lately, I have realized that the experiences I live, make me happy”), both items refer to activities or experiences that produce some degree of satisfaction. In the case of item 4 (“I find it easy to find reasons to enjoy life”) and item 6 (“The activities I used to do are still rewarding”), both items refer to some degree of satisfaction with life. Finally, items 8 (“I am satisfied with my achievements”) and 9 (“My life is interesting”) have in common that both refer to a positive affectivity of their personal experience.

Furthermore, these results constitute the first empirical evidence of the existence of a one-dimensional model for this scale in Spanish for university students, since in previous studies (Barraca & Pérez-Álvarez, 2010; Valderrama-Díaz et al., 2016) only exploratory analyses were carried out, where the existence of a one-dimensional model was not clear. It is important to note that no other psychometric studies of the scale have been found in Latin America or in other countries of the world besides the United States, France and Spain.

Regarding the reliability of the scale, the α coefficient shows an excellent level of reliability (α = .93) which is similar to that found in previous studies, where they also used the same coefficient (Armento & Hopko, 2007; Barraca & Pérez-Álvarez, 2010; Fernández-Rodríguez et al., 2020; Valderrama-Díaz et al., 2016; Wagener & Blairy, 2015) although this indicator can lead to biased estimates of reliability since it is difficult to guarantee the tau-equivalence of items and it is sensitive to the presence of correlated errors in the model (Dunn et al., 2014; Yang & Green, 2010). Furthermore, its use is not recommended for ordinal items (Elosua & Zumbo, 2008) . Therefore, to overcome these limitations, the omega coefficient (ω = .91) was used, which showed adequate reliability indices. The H coefficient was also used since it is robust regarding the presence of correlated errors (Dominguez-Lara, 2016). In this indicator the scale also showed an adequate level of reliability (.94). In addition, the test-retest method shows evidence that the scale scores are reliable over time (ICC = .60). This result allows for the objective contrast of a participant’s changes through time and presents objective data on the effectiveness of the therapy. All the coefficients used show an adequate level of reliability, thus ensuring less measurement error and greater accuracy of the scores obtained. These results also constitute the first empirical evidence on the reliability of the scale, through other indicators more robust than the traditional Cronbach’s α.

Additionally, the sequence of hierarchical variance models proposed in the invariance study show that the scale can be considered invariant according to sex and over time, for this purpose the ΔCFI index was used as the main criterion since ΔRMSEA can be affected by the sample size (Chen, 2007). This allows for the comparison of the scores between men and women, as well as the changes that can occur over time, asuring that the differences found are due to differences in the actual level of the construct. These results are particularly important when evaluating the effectiveness of intervention therapies. Furthermore, it is important to note that these results constitute the first empirical evidence on factorial invariance of the scale.

Regarding the validity in relation to other variables, it was found that the degree of reward provided by the environment is positively related to the degree of satisfaction with life (ρ = .51; p < .01). This result is coherent since if people positively value each aspect of their lives then that would allow them to feel reinforced by their environment. In addition, it was found that the EROS scale has a negative relationship with the level of depression (ρ = −.34; p < .01), anxiety (ρ = −.22; p < .01) and stress (ρ = −.21; p < .01). This relationship can be explained because depressive symptoms are strongly associated with reduced availability and sensitivity to environmental reward (Eshel & Roiser, 2010; Huys et al., 2013; Safra et al., 2019). Similarly, escape and avoidance behaviors associated with anxiety and stress reduce a person's exposure to sources of environmental reward (Grant & White, 2016; Harlé et al., 2017).

Regarding the limitations of the study, a non-probabilistic sampling was used for convenience which limits the generalization of the results. The metric quality of the scale was not evaluated in a clinical sample. Despite these limitations, the results of the study are important not only for psychometric research but also for therapies based on behavioral activation.

In conclusion, this study represents a significant contribution to measurement in the context of behavioral activation therapy as it will facilitate a rapid and efficient assessment of the degree of environmental reward perceived by the person, thus allowing for a systematic monitoring of progress in therapy. In addition, psychometric evidence has been shown for the existence of a method’s effect regarding negative items in the adaptations to Spanish. Finally, the results of the study provide a statistical and methodological basis for the development of the EROS scale in future psychometric studies. Further studies could use the scale to evaluate the effectiveness of therapy in different mental health problems.

Footnotes

Article Notes

Declaration of Conflicting Interests

The author(s) declare(s) that there is no conflict of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Lindsey W. Vilca

Robert I. Echebaudes-Ilizarbe

Michael White

References

Aiken

L. R.

(1980). Content validity and reliability of single items or questionnaires. Educational and Psychological Measurement, 40(4), 955–959. https://doi.org/10.1177/001316448004000419

Aoki

Doi

Horiuchi

Takagaki

Kawamura

Umeno

Fujita

Kitagawa

Sakano

(2019). Mediating effect of environmental rewards on the relation between goal-directed behaviour and anhedonia [Mediating effect of environmental rewards on the relation between goal-directed behaviour and anhedonia]. Current Psychology. https://doi.org/10.1007/s12144-019-00312-y

Armento

M. E. A.

Hopko

D. R.

(2007). The environmental reward observation scale (EROS): Development, validity, and reliability. Behavior Therapy, 38(2), 107–119. https://doi.org/10.1016/j.beth.2006.05.003

Atienza

F. L.

Pons

Balaguer

García-Merita

(2000). Propiedades psicométricas de la escala de satisfacción con la vida en adolescents [Psychometric properties of the life satisfaction scale in adolescents]. Psicothema, 12(2), 314–319. https://www.unioviedo.es/reunido/index.php/PST/article/view/7597

Bados

Solanas

Andrés

(2005). Psychometric properties of the Spanish version of depression, anxiety and stress scales (DASS). Psicothema, 17(4), 679–683. http://www.psicothema.com/psicothema.asp?id = 3165

Barraca

Pérez-Álvarez

(2010). Adaptación española del environmental reward observation (EROS) [Translation results Spanish adaptation of the environmental reward observation (EROS)]. Ansiedad y Estrés, 16(1), 95–107. https://dialnet.unirioja.es/servlet/articulo?codigo = 3191048

Barraca

Pérez-Álvarez

(2015). Activación Conductual para el tratamiento de la depresión [Translation results behavioral activation for the treatment of depression]. Editorial Sintesis S.A.

Barth

Munder

Gerger

Nüesch

Trelle

Znoj

Jüni

Cuijpers

(2013). Comparative efficacy of seven psychotherapeutic interventions for patients with depression: A network. PLoS Medicine, 10(5), e1001454. https://doi.org/10.1371/journal.pmed.1001454

Brown

T. A.

(2015). Confirmatory factor analysis for applied research (Vol. 43). The Guilford Press.

10.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

11.

Chopik

W. J.

Brien

E. O.

Konrath

S. H.

(2017). Differences in empathic concern and perspective taking across 63 countries. Journal of Cross-Cultural Psychology, 48(1), 23–38. https://doi.org/10.1177/0022022116673910

12.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

13.

Dimidjian

Hollon

S. D.

Dobson

K. S.

Schmaling

K. B.

Kohlenberg

R. J.

Addis

M. E.

Gallop

McGlinchey

J. B.

Markley

D. K.

Gollan

J. K.

Atkins

D. C.

Dunner

D. L.

Jacobson

N. S.

(2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology, 74(4), 658–670. https://doi.org/10.1037/0022-006X.74.4.658

14.

DiStefano

Liu

Jiang

Shi

(2018). Examination of the weighted root mean square residual: Evidence for trustworthiness? Structural Equation Modeling: A Multidisciplinary Journal, 25(3), 453–466. https://doi.org/10.1080/10705511.2017.1390394

15.

Dominguez-Lara

(2016). Evaluación de la confiabilidad del constructo mediante el coeficiente H: Breve revisión conceptual y aplicaciones [Evaluation of the reliability of the construct using the H coefficient: Brief conceptual review and applications]. Psychologia: Avances de La Disciplina, 10(2), 87–94. https://doi.org/10.21500/19002386.2134

16.

Dunn

T. J.

Baguley

Brunsden

(2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology (London, England: 1953), 105(3), 399–412. https://doi.org/10.1111/bjop.12046

17.

Elosua, P., & Zumbo, B. D. (2008). Coeficientes de fiabilidad para escalas de respuesta categórica ordenada. Psicothema, 20(4), 896–901. Retrieved from http://www.psicothema.com/psicothema.asp?id=3572

18.

Eshel

Roiser

J. P.

(2010). Reward and punishment processing in depression. Biological Psychiatry, 68(2), 118–124. https://doi.org/10.1016/j.biopsych.2010.01.027

19.

Fernández

N. E.

Mairal

B. J.

(2017). Behavioral activation versus cognitive restructuring to reduce automatic negative thoughts in anxiety generating situations. Psicothema, 29(2), 172–177. https://doi.org/10.7334/psicothema2016.325

20.

Fernández-Rodríguez

González-Fernández

Coto-Lesmes

Pedrosa

(2020). Behavioral activation and acceptance and commitment therapy in the treatment of anxiety and depression in cancer survivors: A randomized clinical trial. Behavior Modification. https://doi.org/10.1177/0145445520916441

21.

Fleiss

J. L.

Levin

Paik

M. C.

(2013). Statistical methods for rates and proportions (3rd ed.). Wiley.

22.

Gamer

Fellows

Singh

(2019). Various coefficients of interrater reliability and agreement . https://cran.r-project.org/web/packages/irr/irr.pdf

23.

Gawrysiak

Nicholas

Hopko

(2009). Behavioral activation for moderately depressed university students: Randomized controlled trial. Journal of Counseling Psychology, 56(3), 468–475. https://doi.org/10.1037/a0016383

24.

Geldhof

G. J.

Preacher

K. J.

Zyphur

M. J.

(2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138

25.

Grant

D. M.

White

E. J.

(2016). The influence of anxiety on cognitive control processes. Oxford Research Encyclopedia of Psychology. https://doi.org/10.1093/acrefore/9780190236557.013.74

26.

Harlé

K. M.

Guo

Zhang

Paulus

M. P.

A. J.

(2017). Anhedonia and anxiety underlying depressive symptomatology have distinct effects on reward-based decision-making. PLoS One, 12(10), e0186473. https://doi.org/10.1371/journal.pone.0186473

27.

Hayes

S. C.

(2016). Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and cognitive therapies. Behavior Therapy, 47(6), 869–885. https://doi.org/10.1016/j.beth.2016.11.006

28.

Heu

L. C.

Zomeren

Van ., & Hansen

(2019). Lonely alone or lonely together ? A cultural-psychological examination of individualism—Collectivism and loneliness in five European countries. Personality & Social Psychology Bulletin, 45(5), 780–793. https://doi.org/10.1177/0146167218796793

29.

Hirayama

Ogawa

Yanai

Suzuki

Shimizu

(2019). Behavioral activation therapy for depression and anxiety in cancer patients: A case series study. BioPsychoSocial Medicine, 13, 9. https://doi.org/10.1186/s13030-019-0151-6 (2019)

30.

Hofstede Insights. (2020). Country comparison. https://www.hofstede-insights.com/country-comparison/

31.

Hofstede

G. J.

Minkov

(2010). Cultures and organizations: Software of the mind (3rd ed.). McGraw-Hill Education. https://books.google.com.pe/books?id = o4OqTgV3V00C

32.

Hopko

D. R.

Armento

M. E. A.

Robertson

S. M. C.

Ryba

M. M.

Carvalho

J. P.

Colman

L. K.

Mullane

Gawrysiak

Bell

J. L.

McNulty

J. K.

Lejuez

C. W.

(2011). Brief behavioral activation and problem-solving therapy for depressed breast cancer patients: Randomized trial. Journal of Consulting and Clinical Psychology, 79(6), 834–849. https://doi.org/10.1037/a0025450

33.

Huys

Q. J.

Pizzagalli

D. A.

Bogdan

Dayan

(2013). Mapping anhedonia onto reinforcement learning: A behavioural meta-analysis. Biology of Mood & Anxiety Disorders, 3(1), 12–16. https://doi.org/10.1186/2045-5380-3-12

34.

Jorgensen

T. D.

Pornprasertmanit

Schoemann

A. M.

Rosseel

(2018). semTools: Useful tools for structural equation modeling. R package version 0.5–1. https://cran.r-project.org/package = semTools

35.

Kanter

J. W.

Busch

A. M.

Rusch

L. C.

(2011). Activación conductual: Refuerzos positivos ante la depresión [Behavioral activation: Positive reinforcements for depression]. Alianza Editorial. https://books.google.com.pe/books?id = zKj0ZwEACAAJ

36.

Kern

Busch

Schneider

K. L.

Miller

S. A.

Appelhans

B. M.

Waring

M. E.

Whited

M. C.

Pagoto

(2019). Psychosocial factors associated with treatment outcomes in women with obesity and major depressive disorder who received behavioral activation for depression. Journal of Behavioral Medicine, 42(3), 522–533. https://doi.org/10.1007/s10865-018-9993-9

37.

Kline

R. B.

(2015). Principles and practice of structural equation modeling (4th ed.). The Guilford Press.

38.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

39.

Lindwall

Barkoukis

Grano

Lucidi

Raudsepp

Liukkonen

Thøgersen-Ntoumani

(2012). Method effects: The problem with negatively versus positively keyed items. Journal of Personality Assessment, 94(2), 196–204. https://doi.org/10.1080/00223891.2011.645936

40.

Lloret

Ferreres

Hernández

Tomás

(2017). The exploratory factor analysis of items: Guided analysis based on empirical data and software. Anales de Psicología, 33(2), 417–432. https://doi.org/10.6018/analesps.33.2.270211

41.

K. H.

Trüschel

Jarl

Magnusson

Windahl

Johansson

Carlbring

Andersson

(2014). Behavioural activation versus mindfulness-based guided self-help treatment administered through a smartphone application: A randomised controlled trial. BMJ Open, 4(1), e003440. https://doi.org/10.1136/bmjopen-2013-003440

42.

Mahen

H. A. O.

Moberly

N. J.

Wright

K. A.

(2019). Trajectories of change in a group behavioral activation treatment for severe, recurrent depression. Behavior Therapy, 50(3), 504–514. https://doi.org/10.1016/j.beth.2018.08.007

43.

Maitland

D. W. M.

Neilson

E. C.

Munoz

E. A.

Ybanez

Murray

A. L.

(2019). The impact of an enriched environment on the relationship between activation and depression in Latinx and Non-Latinx students. The Psychological Record, 69(4), 541–550. https://doi.org/10.1007/s40732-019-00351-8

44.

Martell

C. R.

Dimidjian

Herman-Dunn

(2013). Activación conductual Para la depresión: Una guía clínica [Behavioral activation for depression: A clinical guide]. Desclée De Brouwer.

45.

Martínez-Vispo

Senra

López-Durán

Fernández del Río

Becoña

(2019). Boredom susceptibility as predictor of smoking cessation outcomes: Sex differences. Personality and Individual Differences, 146(March), 130–135. https://doi.org/10.1016/j.paid.2019.03.026

46.

Mazzucchelli

T. G.

Kane

R. T.

Rees

C. S.

(2010). Behavioral activation interventions for well-being: A meta-analysis. The Journal of Positive Psychology, 5(2), 105–121. https://doi.org/10.1080/17439760903569154

47.

Mazzucchelli

T. G.

Kane

Rees

(2009). Behavioral activation treatments for depression in adults: A meta-analysis and review. Clinical Psychology: Science and Practice, 16(4), 383–411. https://doi.org/10.1111/j.1468-2850.2009.01178.x

48.

McCoach

D. B.

Gable

R. K.

Madura

J. P.

(2013). Evidence based on relations to other variables: Bolstering the empirical validity arguments for constructs. In D. B. McCoach, R. K. Gable, & J. P. Madura (Eds.), Instrument development in the affective domain (pp. 209–248). Springer. https://doi.org/10.1007/978-1-4614-7135-6

49.

McDonald

R. P.

(1999). Test theory: A unified treatment. Taylor & Francis.

50.

Mira

Díaz-García

Castilla

Campos

Romero

Bretón-López

Botella

(2019). An internet-based intervention for depressive symptoms: Preliminary data on the contribution of behavioral activation and positive psychotherapy strategies. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, 19(133), 129–146. https://doi.org/10.1007/978-3-030-25872-6_10

51.

Mueller

R. O.

Hancock

G. R.

(2001). Rethinking construct reliability within latent variable systems. In Structural equation modeling: Past and present. A festschrift in honor of Karl G. Jöreskog (pp. 195–261). Scientific Software International.

52.

Oliver

Galiana

Bustos

(2018). Validación de la escala de satisfacción con la vida y su relación con las dimensiones de autoconcepto en universitarios peruanos [Translation results Validation of the life satisfaction scale and its relationship with the dimensions of self-concept in Peruvian university students]. Persona, 2(021), 29–44. https://doi.org/10.26439/persona2018.n021.3018

53.

Otero

Hita

Torres

Á. J.

Vázquez

F. L.

(2020). Brief psychological intervention through mobile app and conference calls for the prevention of depression in non-professional caregivers: A pilot study. International Journal of Environmental Research and Public Health, 17(12), 4578–4515. https://doi.org/10.3390/ijerph17124578

54.

Padrós, F

Gutiérrez, C. Y

C. Y.

Medina, M. A

M. A.

(2015). Propiedades psicométricas de la escala de satisfacción con la vida (SWLS) de diener en población de michoacán (méxico) [Psychometric properties of Diener’s life satisfaction scale (SWLS) in a population of Michoacán (Mexico)]. Avances En Psicología Latinoamericana, 33(2), 223–230. https://doi.org/10.12804/apl33.02.2015.04

55.

Patias

N. D.

Machado

W. L.

Bandeira

D. R.

Dell´Aglio

D. D.

(2016). Depression anxiety and stress scale (DASS-21)—Short form: Adaptação e validação Para adolescentes brasileiros [Depression anxiety and stress scale (DASS-21)—Short form: Adaptation and validation For Brazilian adolescents]. Psico-USF, 21(3), 459–469. https://doi.org/10.1590/1413-82712016210302

56.

Pérez-Alvares

(2014). Las terapias de tercera generación como terapias contextuales [Third-generation therapies as contextual therapies]. Editorial Sintesis S.A.

57.

R Core Team. (2019). A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.r-project.org/

58.

Raykov

Hancock

G. R.

(2005). Examining change in maximal reliability for multiple-component measuring instruments. The British Journal of Mathematical and Statistical Psychology, 58(Pt 1), 65–82. https://doi.org/10.1348/000711005X38753

59.

Richards

D. A.

Ekers

McMillan

Taylor

R. S.

Byford

Warren

F. C.

Barrett

Farrand

P. A.

Gilbody

Kuyken

O'Mahen

Watkins

E. R.

Wright

K. A.

Hollon

S. D.

Reed

Rhodes

Fletcher

Finning

(2016). Cost and outcome of behavioural activation versus cognitive behavioural therapy for depression (COBRA): A randomised, controlled, non-inferiority trial. The Lancet, 388(10047), 871–880. https://doi.org/10.1016/S0140-6736(16)31140-0

60.

Rosseel

(2012). Lavaan : An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

61.

RStudio Team. (2018). RStudio: Integrated Development Environment for R. RStudio, Inc. http://www.rstudio.com/

62.

Ryba

M. M.

Hopko

D. R.

(2012). Gender differences in depression : Assessing mediational effects of overt behaviors and environmental reward through daily diary monitoring. Depression Research and Treatment, 2012, 865679. https://doi.org/10.1155/2012/865679

63.

Safra

Chevallier

Palminteri

(2019). Depressive symptoms are associated with blunted reward learning in social contexts. PLoS Computational Biology, 15(7), e1007224. https://doi.org/10.1371/journal.pcbi.1007224

64.

Saris

W. E.

Satorra

van der Veld

W. M.

(2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 561–582. https://doi.org/10.1080/10705510903203433

65.

Savalei

Falk

C. F.

(2014). Recovering substantive factor loadings in the presence of acquiescence bias: A comparison of three approaches. Multivariate Behavioral Research, 49(5), 407–424. https://doi.org/10.1080/00273171.2014.931800

66.

Schumacker

R. E.

Lomax

R. G.

(2015). A beginner’s guide to structural equation modeling: Fourth edition (4th ed.). Routledge.

67.

Simmonds-Buckley

Kellett

Waller

(2019). Acceptability and efficacy of group behavioral activation for depression among adults: A meta-analysis. Behavior Therapy, 50(5), 864–885. https://doi.org/10.1016/j.beth.2019.01.003

68.

Tomás

J. M.

Oliver

Galiana

Sancho

Lila

(2013). Explaining method effects associated with negatively worded items in trait and state global and domain-specific self-esteem scales. Structural Equation Modeling: A Multidisciplinary Journal, 20(2), 299–313. https://doi.org/10.1080/10705511.2013.769394

69.

Tull

M. T.

Gratz

K. L.

Latzman

R. D.

Kimbrel

N. A.

Lejuez

C. W.

(2010). Reinforcement sensitivity theory and emotion regulation difficulties : A multimodal investigation. Personality and Individual Differences, 49(8), 989–994. https://doi.org/10.1016/j.paid.2010.08.010

70.

Valderrama-Díaz

M. A.

Bianchi-Salguero

J. M.

Villalba-Garzón

J. A.

(2016). Validación de la environmental reward observation scale (EROS) en población colombiana [Validation of the environmental reward observation scale (EROS) in Colombian population]. Universitas Psychologica, 15(4). https://doi.org/10.11144/Javeriana.upsy15-4.vero

71.

van Sonderen

Sanderman

Coyne

J. C.

(2013). Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. PLoS One, 8(7), e68967. https://doi.org/10.1371/journal.pone.0068967

72.

Vázquez

F. L.

López

Blanco

Otero

Torres

Á. J.

Ferraces

M. J.

(2019). The impact of decreased environmental reward in predicting depression severity in caregivers. Annals of Psychology, 35(3), 357–363. https://doi.org/10.6018/analesps.35.3.329131

73.

Vázquez

F. L.

Torres

Á. J.

Otero

Blanco

López

García-Casal

Arrojo

(2019). Cognitive-behavioral intervention via interactive multimedia online video game for active aging: Study protocol for a randomized controlled trial. Trials, 20(1), 1–15. https://doi.org/10.1186/s13063-019-3859-5

74.

Ventura-León

(2019). Back to content-based validity. Adicciones, 0(0). https://doi.org/10.20882/adicciones.1213

75.

Viladrich

Angulo-Brunet

Doval

(2017). A journey around alpha and omega to estimate internal consistency reliability. Anales de Psicología, 33(3), 755–782. https://doi.org/10.6018/analesps.33.3.268401

76.

Wagener

Blairy

(2015). Validation and psychometric properties of the French versions of the environmental reward observation scale and of the reward probability index. Psychologica Belgica, 55(2), 71–86. https://doi.org/10.5334/pb.bg

77.

Wang

Shi

H.-S.

Geng

F.-L.

Zou

L.-Q.

Tan

S.-P.

Wang

Neumann

D. L.

Shum

D. H. K.

Chan

R. C. K.

(2016). Cross-cultural validation of the depression anxiety stress. Psychological Assessment, 28(5), e88–e100. https://doi.org/10.1037/pas0000207

78.

Watkins

M. W.

(2018). Exploratory factor analysis: A guide to best practice. Journal of Black Psychology, 44(3), 219–246. https://doi.org/10.1177/0095798418771807

79.

Watson

J. C.

(2017). Development establishing evidence for internal structure using exploratory factor analysis factor analysis. Measurement and Evaluation in Counseling and Development, 50(4), 232–238. https://doi.org/10.1080/07481756.2017.1336931

80.

Wolf

E. j.

Harrington

K. M.

Clark

S. L.

Miller

M. W.

(2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. Educational and Psychological Measurement, 76(6), 913–934. https://doi.org/10.1177/0013164413495237

81.

World Medical Association. (2013). World medical association declaration of Helsinki ethical principles for medical research involving human subjects. Clinical Review & Education Special, 27(20), 2191–2194. https://doi.org/10.1001/jama.292.11.1359

82.

Yang

Green

S. B.

(2010). A note on structural equation modeling estimates of reliability. Structural Equation Modeling: A Multidisciplinary Journal, 17(1), 66–81. https://doi.org/10.1080/10705510903438963

83.

Zabihi

Lemmel

F. K.

Orgeta

(2020). Behavioural activation for depression in informal caregivers: A systematic review and meta-analysis of randomised controlled clinical trials. Journal of Affective Disorders, 274, 1173–1183. https://doi.org/10.1016/j.jad.2020.03.124

84.

Zhang

Noor

Savalei

(2016). Examining the effect of reverse worded items on the factor structure of the need for cognition scale. PLoS One, 11(6), e0157795. https://doi.org/10.1371/journal.pone.0157795

85.

Zinbarg

R. E.

Revelle

Yovel

(2005). Cronbach’s, α revelle’s β and McDonald’s ω H: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. https://doi.org/10.1007/s11336-003-0974-7