The Socio-Emotional Guidance Questionnaire (SEG-Q)

Abstract

Since facilitating the fullest development of each student in terms of enriching their intellectual development as well as their personal, social, and emotional development has become an important objective, social and emotional guidance has become an integral part of education. This recent acknowledgment has stimulated research interest into how to measure, evaluate, and optimize guidance activities in schools. This interest, however, is contingent on a valid assessment of integrated socioemotional guidance, which despite the growing attention still remains a problem. This study therefore aims to investigate the validity and generalizability of the Socio-Emotional Guidance Questionnaire (SEG-Q). Measurement invariance across three groups of teachers teaching in different stages of secondary education (Total n = 3,336) was tested, by means of multigroup confirmatory factor analyses (MGCFA) in Mplus. The results show partial invariance of the SEG-Q across the teacher groups, confirming that the SEG-Q is a psychometrically sound self-report instrument for secondary education teachers which can be used by researchers and practitioners to measure, map, describe, or evaluate integrated socioemotional guidance.

Keywords

socioemotional guidance secondary education teachers measurement invariance validity factor analyses

Introduction

Facilitating the fullest development of each student in terms of enriching their intellectual development as well as their personal, social, and emotional development has become an important objective of education (Aluede, Imonikhe, & Afen-Akpaida, 2007; Elias, 2003). Social and emotional guidance of students (or SEG) has therefore moved from the margins to the mainstream of education (Lapan, Gysbers, Multon, & Pike, 1997). In line with Best et al. (1995), we define SEG as the whole of activities integrated in the curriculum that aim to stimulate the social and emotional development of every student. It can be attained trough the quality of teaching and learning; through the nature of the relationships among pupils, teachers, and other adults; through monitoring pupils’ overall progress and social and emotional development; through specific support systems; and through extracurricular activities (Best, 1999). This sort of guidance, is referred to as integrated guidance (Detrez, Bosman, & Gombeir, 2000), comprehensive guidance (e.g., Gysbers & Henderson, 2001; Lapan, 2001), pastoral care (e.g., Best, et al., 1995; Watkins, 1999), or a whole school approach to guidance (Lam & Hui, 2010).

All emphasize the central role of teachers in guidance. This is not surprising given that teachers are by far the key figures in the daily school life of students and, consequently, a potential significant other for adolescents who have the need or desire for such a relationship (Tatar, 1998). Teachers are thus the vital link for the implementation of SEG into the curriculum and no school guidance program can be successful without their support and active involvement (Aluede & Egbochuku, 2007; Hui, 2002). As such, teachers’ perception of whether SEG is part of their task and their guiding competences are crucial elements of integrated SEG at school.

Although teachers are considered as key figures, it is not their sole responsibility. Because providing SEG results into additional responsibilities and possibly more stress and a higher workload, a supportive network at the school level becomes a necessity (Chan & Hui, 1998). This is reflected by the concept of a whole-school approach in which the involvement of all teachers is regarded as essential, though demanding the recognition, clarification, organization and planning of guidance at the school level (Aluede, et al., 2007; Best, et al., 1995; Hui, 2002). Factors that have been identified as supporting guidance by the teacher in the classroom and thus the effectiveness of a SEG program at school are a caring school climate, a clear vision, the principal’s support, supporting systems and caring structures, cooperation, communication and coordination of the school team, cooperation with external partners, and the professionalization of the school team (Aluede, et al., 2007; Best, et al., 1995; Galassi & Akos, 2004; Hui, 2000; Luk-Fong & Lung, 2003; Triliva & Poulou, 2006). These characteristics are considered to have a facilitating and supporting role for SEG in the classroom. Together with teachers’ task perception and guiding competences, these school characteristics form the concept of integrated SEG.

The recent acknowledgment of the importance of SEG has also stimulated research interest into how to measure, evaluate, and optimize SEG activities in schools. This is contingent however on a valid assessment of integrated SEG, which still remains a problem. To our knowledge there was no existing instrument, which measured integrated SEG both at the level of the school and the teacher. Therefore, a previous study aimed to develop and validate the Socio-Emotional Guidance Questionnaire (SEG-Q; Authors, 2010). This first exploratory validation study on the SEG-Q yielded good results showing that the SEG-Q measures both task perceptions and guiding competence at the level of the teacher as well as the characteristics of a supportive school network as mentioned in the literature. Two factors at the level of the teacher and nine factors at the level of the school were identified which reliably measure teachers’ perceptions on integrated SEG (for more information see Authors [2010] or the method section). Since construct validation is an ongoing process and evidence should be collected from a number of sources and samples to be able to more adequately evaluate the psychometric merit of an instrument (Messick, 1995), the focus of this study is to further investigate the psychometric properties of the SEG-Q for different groups of teachers—as opposed to one, smaller group of first grade teachers as in the previous study (Authors, 2010).

Measurement Invariance: The Problem of Nonequivalent Measures

Researchers often assume that an instrument measures the same constructs in different groups or that meeting the criteria for reliability and construct validity is sufficient to make valid comparisons (Sass, 2011; Steinmetz, Schmidt, Tina-Booh, Wiezczorek, & Schwartz, 2009). However, often, this assumption is not justified and, therefore, needs to be evaluated by testing for factorial invariance (Milfont & Fischer, 2010). Factorial invariance focuses on the correspondence of factors across different groups and centers around two issues: measurement invariance and structural invariance (Byrne, Shavelson, & Muthén, 1989). Measurement invariance assesses the invariance of the basic model structure while structural invariance assesses the invariance of means, variances, and covariances of the latent variables. Whereas structural invariance is optional and for the researcher to decide, measurement invariance needs to be tested when comparing groups and, especially, group means (Milfont & Fischer, 2010). It is necessary to investigate to what extent manifest variables’ measurement properties are transportable and generalizable across populations. After all, violations of measurement invariance can be as threatening as the inability to demonstrate reliability and validity (Vandenberg & Lance, 2000) since the observed differences in means or other statistics, might then reflect differences in systematic biases of response across groups or different understanding of the concepts, rather than substantive differences.

The Present Study

The purpose of this study was to extend the validity of the SEG-Q by assessing to what extent the factor structure holds across different groups of teachers. Evidence of invariance across these groups of teachers would support further not only the validity but, also, the generalizability of the SEG-Q. In addition, this study can be seen as an example or application of the use of the CFA framework in checking for measurement invariance, of which the importance in psychoeducational assessment has recently been emphasized by Sass (2011).

Method

Sample

Data was collected from three different groups of Flemish secondary education teachers (Total N = 3,336). The first group consisted of 1,420 teachers who taught the first 2 years or first stage of secondary education (12-14–year-old students; 7th and 8th grade in the United States); the second group of 1,126 teachers taught the 3rd and 4th year or second stage (14-16–year-old students; 9th and 10th grade in the United States); and the last group of 790 teachers the 5th and 6th year or third stage of secondary education (16-18–year-old students; 11th and 12th grade in the United States).¹ As in the total population of Flemish secondary education teachers, there were more female than male teachers; in the first, second and third group respectively 71.7%, 65.8%, and 63.7% was female. The average age of the first group was 38.2 years (SD ± 10.8 years) with their ages ranging from 20 to 74 years. For the second group the average age was 38.2 years (SD ± 10.3 years) with ages ranging from 21 to 64 years. For the third group the average age was 41.2 years (SD ± 9.8 years) with ages ranging from 21 to 61 years.

Instrument

The SEG-Q was developed based on a literature review identifying several aspects of social and emotional guidance and possible items to measure them, and based on focus group research with several key-participants (e.g., teachers, guidance teachers, principals, trainers; n = 26) in order to check for content validity. A pilot study was conducted (n = 115), to check for difficulty, clarity, and feasibility of the items (Jacobs & Struyf, 2010, 2012). The SEG-Q is a self-report instrument for teachers that consists of 57 items (see appendix 1), answered on a 5-point likert-type scale (1 = I totally disagree and 5 = I totally agree). It consists of three parts. Part one—coordination and organization at the school—contains 22 items, which measure the coordination and organization of SEG at the level of the school. Four scales measure some of the earlier mentioned school characteristics. The first scale, climate (n = 7), measures to which extent a school maintains a borne climate that both facilitates learning and caring. The second scale, vision (n = 3), measures the school’s explicit and shared vision with regard to SEG. The third scale, principal’s support (n = 6), measures the extent to which the school’s principal takes responsibility and supports the teachers at school with regard to SEG of pupils. The fourth and final scale of part one measures the presence of structures and procedures (n = 6) regarding SEG at school.

The other school characteristics mentioned in the literature are measured by part two of the SEG-Q—support of the teachers at school. This part consists of 20 items and 5 scales that measure the extent to which teachers are supported at school when it comes to integrating SEG. The first scale, internal cooperation (n = 4), measures to what extent teachers and other school personnel cooperate in school with regard to students’ SEG by means of making information accessible to teachers, coordination between the in-house counselors and teachers, and by deliberating on the desired approach. The second scale, professionalization (n = 5), measures the extent to which teachers develop themselves with regard to SEG as well as the support they get to do so. The third scale, teacher communication (n = 3), measures the extent to which teachers at school communicate with each other with respect to the SEG of students. The fourth scale, external cooperation (n = 5), measures to what extent teachers and other school personnel work together with professionals and care providers from outside the school, like for example external student counseling services or mental health centers. The fifth and final scale of part two, communication with parents (n = 3), measures the cooperation between teachers and the parents with respect to SEG and students’ development.

The final and third part of the SEG-Q—guidance by the teachers—contains 15 items regarding the SEG by the individual teacher. Teachers’ task perception is operationalized using two scales: narrow task perception (n = 6) and broad task perception (n = 3). The scale narrow task perception measures to what extent teachers perceive SEG as best provided by specialists (e.g., counselors or a mental health center), and not as the teacher’s main responsibility. The scale broad task perception measures the teacher’s perception of education as being more than knowledge transition or the intellectual stimulation of students. The third and final scale of part three, guiding competence (n = 6), measures to what extent teachers guide their students in their social and emotional development.

Analytic Approach

Firstly, the original factor structure (Figure 1), as identified by the Authors (2010), was tested with the sample data for each group of teachers. This is referred to as testing for configural invariance (Vandenberg & Lance, 2000) and was performed using three different confirmatory factor analyses (CFAs’) per group; one CFA per group for each part of the questionnaire. In order to evaluate the model fit, we used multiple fit indices as the CFI, TLI, and (RMSEA) with a minimum value of 0.95 for the CFI and TLI and 0.06 for the RMSEA (Hu & Bentler, 1995). In case of insufficient model fit, modification indices (MI) were used to identify how to improve model fit. Items with a high MI across groups for factors, other than the one they are intended to measure, were deleted. These items might have measured different constructs. Furthermore, to obtain a more realistic and better fitting model the MIs’ were used, also, to add error covariances between items belonging to the same factor, though always based on the content of the items. Unless the MI were nonsignificant (p > 3.84) for one or more groups, items were always deleted and error covariances were always added simultaneously to the three different groups of teachers.

Figure 1.

Initial Factor structure of the Socio-Emotional Guidance Questionnaire (SEG-Q) as obtained by Jacobs and Struyf (2010).

When we found that configural invariance was in place, the second step was to proceed with testing for measurement invariance to determine whether or not the scores on each construct had the same meaning across groups. Measurement invariance was tested by the means of multigroup confirmatory factor analyses (MGCFA) in Mplus 5 (Muthén & Muthén, 1998-2007). Because the data was ordered-categorical (items answered on a likert-type scale), a Weighted Least Squares Means-Variance Adjusted (WLSMV) estimator with delta parameterization was used. A series of nested models were tested, following the recommendations of Muthén and Muthén (1998-2007) when using this estimator. In the first unconstrained model, the factor loadings and thresholds were allowed to differ across groups with the scale factors fixed at one and the factor means fixed at zero in all groups. This allowed us to establish a baseline model which could be used to further test the other, more restrictive models. In the second model, both factor loadings and thresholds were constrained to be equal with the scale factors fixed at one in group one and free in the other groups and the factor means fixed at zero in one group and free in the others. The factor loadings and thresholds were not constrained separately with first testing metric invariance and, subsequently, scalar invariance. Instead, they were constrained in tandem because, for categorical outcomes the item probability curve is influence by both parameters (Muthén & Muthén, 1998-2007).

As recommended by Vandenberg and Lance’s review study (2000), we used the χ² difference test to evaluate model fit but we examined, also, the differences in comparative fit indices. For the latter, a change of −0.01 or more in CFI or TLI and a change of 0.015 or more in RMSEA indicates noninvariance (Chen, 2007). For the χ² difference test, a nonsignificant difference in χ² supports measurement invariance whereas a significant difference in χ² between the more and less restrictive model shows that the measures are noninvariant. For categorical data, using the WLSMV estimator, the conventional approach of just taking the difference in χ² value and degrees of freedom between two models is inappropriate because the χ² difference is not distributed as χ² (Muthén & Muthén, 1998-2007). Therefore the χ² DIFFTEST function in Mplus was used, which applies a correction of the χ² value. When the difference of this test was significant (p < .05), only partial measurement invariance could be established. In order to determine the degree of partial measurement invariance, the model was modified by setting free some parameters. Which parameters to start freeing depends on the MI which gives the expected drop in the model χ² value when this parameter is freely estimated. The MI is statistically significant when it exceeds 3.84 (Dimitrov, 2006). Firstly, the parameters for the item with the largest significant MI was set free. Since for categorical outcomes the item probability curve is influenced by both parameters (Muthén & Muthén, 1998-2007), thresholds and factor loadings were relaxed in tandem. After freeing the factor loadings and thresholds of the first item, the model was tested again. If the χ² difference test was still significant, other parameters, according to the highest MI, had to be relaxed. When there was no difference in χ² there was no need to free additional parameters.

Results

Confirmatory Factor Models

As presented in Table 1, the original model for part one—coordination and organization at school—consisting of four factors and 22 items, provided a relatively good but still insufficient fit for the three groups. The MI for item 7 with factors other than factor 3 were very high, ranging from 23.417 to 104.324, which might indicate that this item measured several constructs. The same applied to item 5 with MI ranging from 69.354 to 98.324. Therefore, item 5 and 7 were deleted. The goodness-of-fit indices for the model without items 5 and 7 were still insufficient. According to the modification indices, error covariances between items of the same scale needed to be incorporated to enhance model fit. In total, seven different error covariances (see Table 1) were added which led to a very good model fit for the three different groups of teachers. These error covariances were mainly added for the items of the factor structures and procedures. As the name of the scale suggests, these items measure two things, namely the presence of structures as well as procedures. This is also reflected by the error covariances that were added to the model.

Table 1.

Fit Indices for the Confirmatory Factor Analyses.

	Teacher Group 1			Teacher Group 2			Teacher Group 3
Tested Model	CFI	TLI	RMSEA	CFI	TLI	RMSEA	CFI	TLI	RMSEA
Part 1 coordination and organization
Hypothesized model	.919	.973	.076	.893	.966	.089	.874	.953	.100
Hypothesized model, without 7 and 5	.937	.976	.072	.913	.967	.072	.909	.962	.092
Hypothesized model, without 7 and 5, 7 error covariance (error covariance: 1-2, 29-30, 25-26, 15-17, 26-28, 27-28, 26-27)	.970	.989	.050	.960	.985	.059	.958	.982	.063
Part 2 support of the teachers at school
Hypothesized model	.928	.972	.091	.920	.972	.090	.921	.961	.110
Hypothesized model, without 31 and 46	.950	.979	.080	.949	.981	.076	.954	.977	.089
Hypothesized model, without 31 and 46, 5 error covariance (error covariance: 42-43, 54-55, 33-35, 55-38, 33-37)	.973	.989	.059	.975	.991	.054	.976	.988	.064
Part 3 guidance by the teachers
Hypothesized model	.867	.906	.120	.904	.949	.103	.891	.942	.109
Hypothesized model, without 59	.885	.915	.116	.920	.954	.099	.905	.949	.106
Hypothesized model, without 59, 5 error covariance (error covariance: 64-65, 56-58, 69-70, 67-68, 66-67)	.972	.978	.059	.971	.984	.059	.969	.984	.060

The fit indices of part two of the SEG-Q—support of the teachers at school—also show that the hypothesized, original model with five factors and 20 items insufficiently fitted the data. The MI of item 31 along with factors other than factor 2 were extremely high ranging from 92.895 to 182.072. The same accounted for item 46 (MI’s from 50.182 to 129.977). These items were therefore deleted. The new model without items 31 and 46 resulted in a better but still unsatisfactory fit. The MI showed that five error covariances between items of the same scale needed to be incorporated. By adding these five error covariances, the model produced a good fit to the data. Two of these error covariances were added to the scale professionalization. When looking at the content of the items, it appears that item 34 and 36 are more about the school level, and item 33, 35, and 37 more about the individual level. For external cooperation, the error covariances show that the cooperation can both mean alignment with external caregivers (items 49 and 53) as well as actual support from external caregivers (items 38, 54, and 55).

For the original model of part three—guidance by the teachers—the fit indices showed that the structure with three factors and 15 items was not a good match. Item 59, which was supposed to measure narrow task perception, seemed, also, to be an indicator of guiding competence since the MIs’ of item 59 by factor 1 ranged from 36.462 to 110.610 across the three different groups of teachers. Therefore, item 59 was deleted from the questionnaire. However, without this item, the model fit remained unsatisfactory with the MIs’ indicating that five error covariances were needed. Adding these led to a well-fitting model for the three groups of teachers. These error covariances can also be justified based on the content of the items. For the scale narrow task perception, item 56 and 58 are more about the role of external caregivers and items 64 and 65 about the role of the individual teacher. For guiding competence, items 66, 67, and 68 describe actions and knowledge with respect to guiding students in their social and emotional development, while items 69 and 70 refer to handling problems.

We conclude that after considerable adaptations, we found good fitting models across groups for the three different parts of the questionnaire and established configural invariance. In total, five items had to be deleted and seven, five, and five error covariances respectively had to be added.

Measurement Invariance

In a second phase, MGCFA was used to compare the factor loadings and thresholds of the three different groups of teachers. The results can be found in Table 2. For every part of the SEG-Q, firstly, a measurement noninvariance model without constraints was tested and used as a baseline or comparison model for the other tested models. For the first part of the questionnaire—coordination and organization—the results in Table 2 show that measurement invariance could not be supported when comparing the measurement invariance model with this baseline model. The χ² difference test produced a significant result. Freeing the parameters of item 3—the item with the highest MI—across groups seemed insufficient. Therefore, the factor loading and thresholds of item 9 and, subsequently those of item 10 were unconstrained. This led to a good fitting model across groups and a nonsignificant difference in χ². The changes in the other fit indices were also acceptable, namely less than 0.01.

Table 2.

Fit Indices for the Multigroup Confirmatory Factor Analyses.

	Overall Fit Indices			Comparative Fit Indices
Tested Model	CFI	TLI	RMSEA	p-value Δχ²	\|ΔCFI\|	\|ΔTLI\|	\|ΔRMSEA\|
Part 1: Coordination and organization
Measurement noninvariance	.964	.986	.056	—	—	—	—
Measurement invariance (factor loadings and thresholds)	.970	.991	.048	.0000	.006	.005	.008
Measurement invariance, freeing item 3	.971	.990	.048	.0019	.007	.004	.008
Measurement invariance, freeing item 3 and 9	.971	.990	.048	.0094	.007	.004	.008
Measurement invariance, freeing item 3, 9 and 10	.971	.990	.048	.0962	.007	.004	.008
Part 2: Support of the teachers at school
Measurement noninvariance	.975	.991	.053	—	—	—	—
Measurement invariance (factor loadings and thresholds)	.975	.989	.059	.0000	.000	.002	.006
Measurement invariance, freeing item 38	.977	.992	.051	.0000	.002	.001	.002
Measurement invariance, freeing item 38 and 32	.976	.992	.052	.0000	.001	.001	.001
Measurement invariance, freeing item 38, 32 and 43	.977	.992	.052	.0000	.002	.001	.001
Measurement invariance, freeing item 38, 32, 43 and 42	.977	.992	.052	.0011	.002	.001	.001
Measurement invariance, freeing item 38, 32, 43, 42 and 49	.976	.991	.052	.0020	.001	.000	.001
Measurement invariance, freeing item 38, 32, 43, 42, 49 and 55	.977	.991	.052	.0701	.002	.000	.001
Part 3: Guidance by the teachers
Measurement noninvariance	.970	.982	.060	—	—	—	—
Measurement invariance (factor loadings and thresholds)	.955	.981	.061	.0000	.015	.001	.001
Measurement invariance, freeing item 65	.961	.983	.057	.0000	.009	.001	.003
Measurement invariance, freeing item 65 and 64	.972	.988	.048	.1107	.002	.006	.012

Note. The comparison model is always the measurement noninvariance model since factor loadings and thresholds are constrained together. As such, only for the measurement invariance models comparative fit indices could be calculated.

Measurement invariance of the factor loadings and thresholds across teacher groups for the second part of the questionnaire—support of the teachers at school—also, was not supported. Both the unconstrained and constrained model showed good fit but, as can be seen in Table 2, the χ² difference between these two models was significant. For this reason, we tested for partial measurement invariance by freeing the factor loading and thresholds of item 38, and subsequently item 32, 42, and 43. After also freeing the parameters of items 49 and 55—items with relatively high and significant MI’s—the result of the χ² difference test was no longer significant. This invariant model with six unconstrained items across groups produced a good fit, with no great differences in other fit indices between this model and the measurement noninvariant model.

Lastly, the results, also, did not support measurement invariance of the factor loadings and thresholds for part three—guidance by the teachers. Since the MI were high for item 65’s parameters, this item’s factor loading and thresholds were freed. However, freeing this item, was insufficient with results still showing a significant difference in χ² and a high MI for item 64. Therefore, item 64’s factor loading and thresholds were freed, also, which led to a nonsignificant difference with the baseline model.

The final model and corresponding parameters for the invariant items for the three parts of the SEG-Q are presented in Figure 2. The noninvariant items which had different factor loadings and/or threshold across groups, were omitted (see also discussion). The correlations between the factors of the different parts of the SEG-Q are also incorporated in Figure 2. The results show that there are moderate to high correlations between the factors. Internal consistency measures of the final invariant scales as (see Table 3) show that all scales are reliable with α ranging from 0.7 to 0.9.

Figure 2.

Final Invariant Model With Standardized Parameter Estimates.

Table 3.

Cronbach α for the Scales of the SEG-Q.

Scale	α
Climate	.75
Vision	.81
Principal’s support	.85
Structures and procedures	.81
Professionalization	.72
Communication teachers	.89
Communication parents	.85
Narrow task perception	.81
Broad task perception	.73
Guiding competences	.70

Discussion

An important goal of every educational institution nowadays is to stimulate the total development of every student, which means, also, enriching their personal, social, and emotional development. As a consequence, integrated SEG and research on integrated socioemotional guidance (SEG) has become important. With respect to the latter, an instrument to measure integrated SEG is needed. Therefore, this study aimed to extend the SEG-Q’s validation by investigating its factor structure. The assumption of measurement invariance or population homogeneity across teachers teaching in different stages of secondary education was assessed by means of CFA and MGCFA.

The Factor Structure of the SEG-Q

In general, the CFA’s results supported the original factor structure of the SEG-Q across teacher groups by confirming that the questionnaire measures the different school and teacher characteristics as postulated in the literature. Only minor modifications to the measurement model were needed: five of the total 57 items were deleted because they all measured more than one underlying construct, and 17 error covariances were added between items measuring the same factor. After these minor modifications, the model proved to be a good fit to the data of the three different groups, thus establishing configural invariance. This result affirms that the specific school and teacher characteristics as mentioned in the literature and measured by the SEG-Q are separate and identifiable aspects of integrated SEG to which teachers attach the same meaning (Vandenberg & Lance, 2000).

Measurement Invariance Across Teacher Groups

Full measurement invariance of the factor loadings and the thresholds could not be confirmed. The χ² test indicated a significant difference in model fit compared to the noninvariant baseline model due to respectively three items of part one, six items of part two, and two items of part three of the SEG-Q. This raises the question whether the teachers of different stages of secondary education perceived the content of the items in a different light. At first glance, the items’ contents did not reveal any obvious explanation for the noninvariance. There might be a problem that the items were formulated ambiguously and contained certain presumptions. For example, item 3 “The school’s principal is aware of the support I provide for the socioemotional development of pupils” already presumed that the teacher was providing support. Therefore, differences in the respondents’ answers to this item might reflect both a different interpretation of this item (with some teachers reacting to the presumption and others not), or a genuine difference in the support provided by teachers of the different stages. However, as is generally the case with scientific research and construct validity, in particular, more research is needed to further confirm and explain these preliminary findings and possible hypotheses.

Another question that arises is what the failure of establishing full measurement invariance in the present study means for the further use of the SEG-Q. As stated in the literature, full measurement invariance is rather unlikely to hold in practice (Hoyle & Smith, 1994). The outstanding question then is: how many invariant items are needed to make valid comparisons? And what should we do with the noninvariant items? As to the first question, although no definite answer or rule exists, at least one item other than the one constrained to one for identification purposes needs to be invariant across groups (Byrne, et al., 1989). For this study’s findings, this means that the factor “internal cooperation” cannot be used when comparing or using measures of different groups of teachers as only three items could be withheld after the CFA, and two of them seem to be noninvariant across the groups. On the question of what we should do with the noninvariant items when using the SEG-Q, three different options are mentioned in the literature (Gregorich, 2006; Millsap & Kwok, 2004). The first option is to allow group comparisons on all items regardless of any evidence of lack of measurement invariance. The rationale behind this approach is the belief that the population differences in factor structure are small and will not obscure inferences from the scale (Millsap & Kwok, 2004). According to Steinmetz and his colleagues (2009) using a scale with only partial invariance of the underlying model may suffice in a structural equation model. However, when using manifest composite scores, partial invariance is probably insufficient because both invariant and noninvariant items are aggregated to form the composite (Steinmetz, et al., 2009). In contrast to the first option, the second option would be to abandon the use of the scales with noninvariant items altogether. The reasoning behind this is that the lack of invariance means that the scale is measuring different latent variables in different groups (Gregorich, 2006). For the SEG-Q, this would mean that the only usable scales are vision, structures and procedures, communication with teachers, communication with parents, guiding competence, narrow task perception, and broad task perception. Finally, the third and, in our opinion, most reasonable option is a compromise between option one and two in which noninvariant items are treated as nuisance and, therefore, excluded from appropriate group measures (Millsap & Kwok, 2004). For the SEG-Q, this option would mean excluding items 3, 9, and 10 in part one of the questionnaire, items 32, 38, 42, 43, 49, and 55 in part two and items 64 and 65 in part three (see also appendix). As a consequence, the factors “internal cooperation” and “external cooperation” are left with two or less items and thus no longer useable. However, apart from losing these two scales, this option seems a defensible approach and, consequently, recommended by the authors when using the SEG-Q to measure integrated SEG in schools.

As stated, we opted to investigate measurement invariance in this study by mainly using the χ² approach as suggested by Vandenberg and Lance (2000) complemented with Chen’s criterion (Chen, 2007). Although no consensus exists, it should be noted that the results and conclusions of this study would have been different if we had decided to mainly use Chen’s criterion. In that case the initial SEG-Q should be regarded an invariant measure of integrated SEG.

Limitations

While the present findings are encouraging and indicate that overall the SEG-Q provides consistent measures of integrated SEG, important questions still remain. One such important issue is the equivalence or stability of the SEG-Q scores over time. Therefore, the use of the SEG-Q in a longitudinal study could prove very useful. Furthermore, there was no examination of the invariance across other potentially interesting groups. For example, a test to establish invariance across gender can be conducted and, also, measurement invariance testing for other groups of teachers (e.g., primary education teachers) and teachers in other European countries and abroad is needed. That being the case, research interested in comparing means, variances or effects between groups, other than the one examined here, is desirable to further establish measurement invariance and the generalizability of the SEG-Q.

Conclusion

In general, our findings demonstrated partial measurement invariance of the SEG-Q scores across secondary education teachers. Psychometrically, this means that the instrument is eliciting responses to questions, which are being perceived in the same way by different subpopulations of teachers. We can thus conclude that the SEG-Q is a psychometrically sound self-report instrument for teachers. The SEG-Q measures the in the literature considered important school characteristics and central role of the teacher regarding SEG by means of three parts, 10 scales and 38 items. The SEG-Q can be used by researchers and practitioners (e.g., academics, policymakers, principals, counselors, school psychologists . . . ) to evaluate integrated SEG at school, to measure the current state of affair or to enhance dialogue on this matter between different parties at school. This use can be very useful given the changing roles in an integrated model of SEG. In an integrated model of guidance, the teacher becomes the first line helper, and counselors are set with coordinating and supporting roles. This involves professional guidance and training of the regular class teachers and other school personnel, curriculum development, maintenance of records, and doing research as a reflective practitioner. The SEG-Q can enable the school counselor to fulfill this new role of supporting, coordinating, and evaluating guidance at school.

Footnotes

Appendix

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The research presented in this article was funded by the Research Council of the University of Antwerp through a PhD grant awarded to Karen Jacobs.

Notes

References

Aluede

Egbochuku

(2007). The influence of personal characteristics on secondary school teachers’ beliefs about guidance and school counselling programs. Education, 127, 440-446.

Aluede

Imonikhe

Afen-Akpaida

(2007). Towards a conceptual basis for understanding developmental guidance and counselling model. Education, 128, 189-201.

Best

(1999). The impact on pastoral care of structural, organisational and statutory changes in schooling: Some empirical evidence and a discussion. British Journal of Guidance & Counselling, 27, 55-70.

Best

Lang

Lodge

Watkins

(1995). Pastoral care and personal-social education: Entitlement and provision. London, UK: Cassell.

Byrne

B. M.

Shavelson

R. J.

Muthén

(1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456-466.

Chan

D. W.

Hui

E. K. P.

(1998). Stress, support and psychological symptoms among guidance and non-guidance secondary school teachers in Hong Kong. School Psychology International, 19, 169-178.

Chen

F. F.

(2007). Sensitivity of goodness of fit indices to lack of measurement invariance. Structural Equation Modeling, 14, 464-504.

Detrez

Bosman

Gombeir

(2000). Een leer-krachtige concretisiering van het concept ’geïntegreerde jongerenbegeleiding’ (GJB) [A powerful realization of the concept of integrated guidance]. Handboek Leerlingenbegeleiding, 31, 25-61.

Dimitrov

D. M.

(2006). Comparing groups on latent variables: A structural equation modeling approach. Work, 26, 429-436.

10.

Elias

M. J.

(2003). Academic and social-emotional learning (Vol. 11): Brussels, Belgium: International Bureau of Education.

11.

Galassi

J. P.

Akos

(2004). Developmental advocacy: Twenty-first century school counseling. Journal of Counseling & Development, 82, 146-157.

12.

Gregorich

S. E.

(2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical Care, 44, S78-S94.

13.

Gysbers

N. C.

Henderson

(2001). Comprehensive guidance and counseling programs: A rich history and a bright future. Journal of Professional School Counseling, 4, 246-256.

14.

Hoyle

R. H.

Smith

G. T.

(1994). Formulating clinical research hypotheses as structural equation models: A conceptual overview. Journal of Consulting and Clinical Psychology, 62, 429-440.

15.

L. T.

Bentler

P. M.

(1995). Evaluating model fit. In Hoyle

R. H.

(Ed.), Structural equation modeling: Concepts, issues and applications (pp. 77-99). Thousand Oaks, CA: SAGE.

16.

Hui

E. K. P.

(2000). Guidance as a whole school approach in Hong Kong: From remediation to student development. International Journal for the Advancement of Counselling, 22, 69-82.

17.

Hui

E. K. P.

(2002). A whole-school approach to guidance: Hong Kong teachers’ perceptions. British Journal of Guidance & Counseling, 30, 63-80.

18.

Jacobs

Struyf

(2010). Socio-emotionele geïntegreerde leerlingenbegeleiding op school: de constructvaliditeit en betrouwbaarheid van de SEG-vragenlijst (Integrated socio-emotional guidance: construct validity and reliability of the socio-emotional guidance questionnaire). Pedagogische Studiën: Tijdschrift voor Onderwijskunde en Opvoedkunde, 87(6), 394-411.

19.

Jacobs

Struyf

(2012). Measuring Integrated Socio-Emotional Guidance at School: Factor Structure and Reliability of the Socio-Emotional Guidance Questionnaire (SEG-Q). Manuscript accepted for publication in Measurement and Evaluation in Counseling and Development.

20.

Lam

S. K. Y.

Hui

E. K. P.

(2010). Factors affecting the involvement of teachers in guidance and counselling as a whole-school approach. British Journal of Guidance & Counselling, 38, 219-234.

21.

Lapan

R. T.

(2001). Results-based comprehensive guidance and counseling programs: A framework for planning and evaluation. Journal of Professional School Counseling, 4, 289-299.

22.

Lapan

R. T.

Gysbers

N. C.

Multon

K. D.

Pike

G. R.

(1997). Developing guidance competency self-efficacy scales for high school and middle school students. Measurement and Evaluation in Counseling and Development, 30, 4-16.

23.

Luk-Fong

Y. Y. P.

Lung

C. L.

(2003). The initial development of an instrument for the evaluation of guidance and counseling services in schools. School Psychology International, 24, 292-312.

24.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

25.

Milfont

T. L.

Fischer

(2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3, 111-121.

26.

Millsap

R. E.

Kwok

O. M.

(2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93-115.

27.

Muthén

L. K.

Muthén

B. O.

(1998-2007). Mplus user’s guide (5th ed.). Los Angeles, CA: Author.

28.

Sass

D. A.

(2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29, 347-363.

29.

Steinmetz

Schmidt

Tina-Booh

Wiezczorek

Schwartz

S. H.

(2009). Testing measurement invariance using multigroup CFA: Differences between educational groups in human values measurement. Quality and Quantity, 43, 599-616.

30.

Tatar

(1998). Primary and secondary school teachers’ perceptions and actions regarding their pupils’ emotional lives. School Psychology International, 19, 151-168.

31.

Triliva

Poulou

(2006). Greek teachers’ understandings and constructions of what constitutes social and emotional learning. School Psychology International, 27, 315-338.

32.

Vandenberg

R. J.

Lance

C. E.

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70.

33.

Watkins

(1999). Personal-social education: Beyond the national curriculum. British Journal of Guidance & Counselling, 27, 71-84.