Measuring Students’ Perceptions of the Medical School Learning Environment: Translation,Transcultural Adaptation,and Validation of 2 Instruments to the Brazilian Portuguese Language

Abstract

Background:

Although learning environment (LE) is an important component of medical training, there are few instruments to investigate LE in Latin American and Brazilian medical schools. Therefore, this study aims to translate, adapt transculturally, and validate the Medical School Learning Environment Scale (MSLES) and the Johns Hopkins Learning Environment Scale (JHLES) to the Brazilian Portuguese language.

Method:

This study was carried out between June 2016 and October 2017. Both scales have been translated and cross-culturally adapted to Brazilian Portuguese Language and then back translated and approved by the original authors. A principal components analysis (PCA) was performed for both the MSLES and the JHLES. Test–retest reliability was assessed by comparing the first administration of the MSLES and the JHLES with a second administration 45 days later. Validity was assessed by comparing the MSLES and the JHLES with 2 overall LE perception questions; a sociodemographic questionnaire; and the Depression, Anxiety, and Stress Scale (DASS-21).

Results:

A total of 248 out of 334 (74.2%) first- to third-year medical students from a Brazilian public university were included. Principal component analysis generated 4 factors for MSLES and 7 factors for JHLES. Both showed good reliability for the total scale (MSLES α = .809; JHLES α = .901), as well as for each subdomain. Concurrent and convergent validity were observed by the strong correlations found between both scale totals (r = 0.749), as well as with both general LE questions: recommend the school to a friend (MSLES: r = 0.321; JHLES: r = 0.457) and overall LE rating (MSLES: r = 0.505; JHLES: r = 0.579). The 45-day test–retest comparison resulted in a Pearson correlation coefficient of 0.697 for the JHLES and 0.757 for the MSLES.

Conclusions:

Reliability and validity have been demonstrated for both the MSLES and the JHLES. Thus, both represent feasible options for measuring LE in Brazilian medical students.

Keywords

Medical education medical students mental health learning environment

Introduction

In recent decades, medical education has paid greater attention to medical schools’ learning environment (LE) and medical students’ perception of their LE.¹ In this context, developing valid and reliable instruments intending to measure and comprehend the most important aspects of the LE² is of major importance, allowing comparisons with many aspects of the environment and targeting more precise interventions to help students avoid or cope with negative environments.

The need for valid instruments to evaluate the LE in the Brazilian context may be especially important. A recent systematic review and meta-analysis of Brazilian medical students³ found high rates of mental health problems, such as generalized anxiety disorder, depressive symptoms, and burnout, even higher than US medical students.⁴ According to Tempski et al,⁵ in a qualitative study with Brazilian medical students, medical schools’ LEs were felt to contribute to poor student well-being. Furthermore, medical education in Brazil is a growing business as compared with its international counterparts and is receiving greater attention from medical education leaders around the world.⁶ According to recent data, Brazil has 1 of the largest medical education systems in the world,⁶ with private medical schools being responsible for more than 50% of student enrollment.⁷ Accurate measurement of LE quality is necessary to determine how LE quality varies across this large system and how LE factors could enhance student learning or contribute to poor student well-being.

To our knowledge, there is only 1 instrument translated to Brazilian Portuguese language to assess LE, the Dundee Ready Education Environment Measure (DREEM).^8,9 It was developed by Roff et al¹⁰ and is the most published measure of LE perceptions,² being the “gold standard” for measuring LE for many years. However, DREEM has drawn criticism for its 50-item length and concerns that some DREEM items were subject to misinterpretation and low factor loadings.¹¹ As an example, in the Swedish validation, students failed to understand questions about cheating (broad question not specifying in which situation cheating occurs) and students’ accommodation (not appropriate for all institutions and students, as some universities do not provide accommodation for their students).¹¹ Other problems were noted in the Greek validation, in which the items concerning cheating, feedback, and course objectives were misunderstood.¹² All these problems using DREEM resulted in a limited score interpretation validity evidence,² such that 1 group of authors^2(p1692) concluded that “researchers have been more likely to consider using DREEM because they knew of it, rather than conducting a literature review to find a different tool.”

Thus, scales other than DREEM are needed to measure LE perceptions in the Brazilian and Latin America context and all over the world. Two out of several different tools are noteworthy due to its large use, adequate length, and good psychometric proprieties shown by previous studies,² the Medical School Learning Environment Scale (MSLES) and the Johns Hopkins Learning Environment Scale (JHLES), which was more recently developed and its use is growing worldwide.^13-15 Hence, the development or the translation to the Brazilian Portuguese language of new instruments that evaluate medical schools’ LE is timely and important.

Thus, the objective of this study is to translate, adapt, and determine if interpretation of scores from 2 LE scales (JHLES and MSLES) have validity. The choice for these 2 instruments rely on the fact that these are possibly better options than DREEM due to their length, and better psychometric proprieties based on previous studies. Furthermore, having other options of scales with different perspectives and variables is important to contribute to a more solid evidence in this specific field. Also, we discuss the benefits and limitations of using either JHLES or MSLES based on several perspectives, such as their psychometric and structural characteristics (presented here and in past studies).

Method

This study was carried out between June 2016 and October 2017. Institutional review board approval was obtained from the Federal University of Juiz de Fora (UFJF) Ethical Committee and students signed a consent term.

Participants and criteria for eligibility

All 334 undergraduate students regularly enrolled in the second (end of first year) through fifth (beginning of third year) semesters of the School of Medicine of the UFJF and present during the classes when scales were administered were invited to participate in this study. Brazilian medical schools have a 6-year curriculum: the first 2 years are preclinical years (mostly classroom activities), followed by 2 years of clinical activity (classroom and hospital activities) and the last 2 years are the clerkship (mostly hospital activities). Students who did not agree to participate or who did not fully complete the questionnaires were excluded from the study.

Procedures

From August to September 2017, 2 authors (A.O.F. and B.N.S.) visited all medical students (n = 334) twice at the end of their classes to distribute all questionnaires. Professors agreed to let a researcher use class time to go over the research process, and the study objectives were explained to all students to have their informed consent. Students who voluntarily agreed to participate were given all 5 questionnaires in press for the first round of testing, and they were informed that 45 days later they would have only the JHLES and MSLES retested.

No compensation was provided to students who agreed to participate. The first round of testing lasted approximately 30 minutes, and the second round took about 15 minutes. To compare both test and retest data, students were asked to fill out the sociodemographic scale with each student’s national registration number (CPF, in Portuguese). Students who did not correctly fill out the CPF in either 1 of the applications were excluded from the study.

Instruments

Sociodemographic data were collected from all participants including age, course semester, sex, and ethnicity.

Overall LE perception questions were developed by Shochet et al¹³ and replicated in ensuing research.^14,15 The first question asks students to rank their overall perception of the LE as terrible, poor, fair, good, or exceptional. The second item asks students if they strongly disagree, disagree, are neutral, agree, or strongly agree with the statement “Based on my sense of the learning environment at the School of Medicine, I would recommend it to a close friend as a great school to attend.”

The Depression, Anxiety, and Stress Scale (DASS), shortened 21-item version (DASS 21), developed by Lovibond and Lovibond,¹⁶ is a 21-question scale intended to differentiate and grade the symptoms of depression, stress, and anxiety.¹⁷ Each domain (depression, stress, and anxiety) is made up of 7 questions answered with a 4-point Likert-type scale and has been widely used in different populations, including medical students,^18,19 and translated to the Brazilian Portuguese by Vignola and Tucci,²⁰ showing good internal consistency. We chose to measure depression, anxiety, and stress as variables to correlate with LE because many studies have shown a significant and inverse correlation between psychological distress and poor levels of students’ perception of their LE.^21-23

The JHLES is a 28-item instrument that evaluates medical students’ perceptions of their LE. Developed by Shochet et al¹³ at the Johns Hopkins School of Medicine, the JHLES development followed a systematic review of previous instruments² and a series of open-ended questions to investigate several events or relationships that medical students might face during their medical training and can affect their LE perception.²⁴ The most remarkable ones were then cataloged and compared with previous scales. A factor analysis resulted in a 28-item scale composed of 7 domains: community of peers, faculty relationships, academic climate, meaningful engagement, mentoring, inclusion and safety, and physical space. Each item has 5-point response options, with high scores indicating a better perception of the LE, so the JHLES total score can range from 28 to 140. The JHLES has been translated, adapted, assessed its psychometric properties, and used in several countries, including Taiwan,¹⁴ India,²⁵ China, Israel, and Malaysia.^15,26

Medical School Learning Environment Scale–Short Version (MSLES) is a 17-item scale adapted by Rosenbaum et al.²⁷ The original 55-item version was developed by Marshall²⁸ and was based on the pre-existing Learning Environment Questionnaire (LEQ).²⁹ The LEQ was comprised of 50 items and was made up of 7 domains: breadth of interest, student interaction, organization (goal direction), flexibility (authoritarianism), meaningful learning experience, emotional climate, and nurturance. Due to the length of the original scale, other authors have modified it into a shorter format.^27,30 Rosenbaum et al²⁷ condensed the original MSLES in a 16-item scale and further added a vertical integration question, resulting in a 17-item scale with 5-point Likert-type responses (ranging from “never” to “very often”), comprised of 6 different domains: breadth of interest, student–student interactions, supportiveness, meaningful learning experiences, organization, and vertical integration. The original 55-item version has shown good psychometric features,³¹ as does the 17-item version.²⁷

Validity analyses

The validity was separated into 4 phases presented below: (1) the translation validity, (2) the factor analysis, (3) concurrent validity, and (4) the criterion-related validity.

(a) Translation validity and Transcultural Adaptation

In June through September 2016, we carried out a translation and cross-cultural adaptation of both scales, simultaneously. There is no general consensus concerning cross-cultural adaptation; therefore, the translation and adaptation processes that we used were adapted from previous studies and guidelines and are described below.^32-34

First, approval was obtained from the original authors to translate both scales. Next, 3 authors (R.F.D., O.S.E., and G.L.), all native Portuguese speakers and fluent in the English language, independently translated both scales from the original English to the Brazilian Portuguese, after which a meeting was held to compare the versions; assess the initial conceptual, semantic, and content equivalence; and achieve consensus on translated scale items for each instrument.

Some slight changes were necessary to make each scale appropriate for the Brazilian culture. As an example, in the original JHLES, question 12 states: “The faculty advisors . . .”; as advisory is not a widespread practice in Brazilian universities, we decided to modify this question to become, “Professors and Directors . . .” Other changes were minor such as JHLES questions 22 and 23, where “mentor” was translated into “research mentor” (question 22) and “professor” (question 23).

After the translation to the Brazilian Portuguese language, both scales were sent to 2 native English speakers to perform back-translations independently, resulting in a total of 4 back-translation instruments (2 for MSLES and 2 for JHLES). Then, the 2 back-translations were sent to each scales’ original authors to create back-translation final versions. The English back-translation final versions were then translated into Portuguese by authors (R.F.D., O.S.E., and G.L.) and both scales were given in a discussion group to 12 Brazilian medical students for pilot testing and response process validity. Minor suggestions in punctuation and scale layout made by students were corrected by the Brazilian authors.

To reconcile the data with previous research^13,15 and to create a comparison question for concurrent validity purposes, we also carried out the same adaptation process described above for the 2 overall LE perception questions, which resulted in no major changes.

(b) Factor Analysis

Factor analysis was carried out using SPSS version 21, IBM for Microsoft Windows. Principal component analyses (PCAs) for each scale were performed using Varimax rotation with Kaiser normalization. Eigenvalues greater than 1.0 and Scree plots were used to evaluate the component factors, and factor loadings of 0.40 or greater were considered satisfactory. This was used to identify the factor structure of the instruments.

Concurrent validity refers to the degree to which a construct correlates with other measures of the same construct that are measured at about the same time. In our study, concurrent validity was assessed correlating the JHLES and the MSLES using Pearson correlation coefficients.

(d) Criterion-Related Validity

Descriptive statistics were reported for the sociodemographic features, LE and DASS21 scales, and the 2 general LE questions. Convergent validity³⁵ was assessed by calculating Pearson correlation coefficients for JHLES and MSLES totals to the 2 overall LE perception questions and to the DASS 21 scales.¹⁶ Validity was also assessed through contrasted groups by comparing the differences among the means of JHLES and MSLES scores between sex, ethnicity, semester of graduation, overall perception of LE, and school endorsement using independent t-tests or ANOVAs for independent measures.

Finally, as there were slight changes between the original JHLES and the revised Brazilian factorial JHLES in the PCA, we decided to explore this scale further, aiming to verify if the use of the original version of the scale would be appropriate to the Brazilian context as well. For that matter, we carried out several analyses using both factorial analyses of the instrument. These analyses were the correlation of the Brazilian and Original American JHLES with sex, ethnicity, semester of graduation, overall perception of the LE, school endorsement, and DASS. The same approach was not possible for the MSLES as the PCA resulted in different dimensions and, therefore, the use of the American version in this context would not be appropriate.

Reliability

Reliability was assessed in the following way. First, we assessed the internal consistency³⁶ of the instrument using Cronbach alpha coefficient (95% confidence interval), which varies from 0 to 1. An instrument is considered to have good reliability if Cronbach alpha is greater than 0.7.³⁷ Then, stability was tested using a test–retest reliability (45 days period) through Pearson correlation and intraclass correlation coefficients (ICCs). Although there is no gold standard period for the test–retest reliability, we decided to use this period to verify if the stability was still high after a medium-term period, which supports even further this stability. Our data collection happened predominantly in the middle of the semester and this is a period with a low quantity of examinations and the LE has a low likelihood of changing in this period.

All analyses were carried out using SPSS 21 (SPSS Inc.).

Results

From a sample of 334 undergraduate students, 248 students (74.2%) completed the questionnaire and 65 students (19.5%) filled out the retest questionnaires. Table 1 summarizes the main findings concerning sample description and general LE questions.

Table 1.

Sample characteristics and overall learning environment perception questions.

	Population		Overall JHLES score		Overall MSLES score
	n	%	Mean	SD	Mean	SD
Whole sample	248	100.0%	90.70	13.49	49.37	7.30
Sex
Male	119	48.0%	91.71	13.08	49.56	7.20
Female	129	52.0%	89.75	13.85	49.20	7.41
Ethnicity
White	170	68.5%	91.64	13.41	49.66	7.28
Non-White	78	31.5%	88.28	13.46	48.56	7.45
Semester of graduation*
Second	76	30.6%	98.06	1.31	54.08	6.22
Third	60	24.2%	86.93	1.68	48.76	6.98
Fourth	77	31.0%	87.69	1.52	47.09	7.24
Fifth	35	14.2%	87.85	2.24	45.59	4.76
Overall perception of the LE**
Exceptional	40	16.1%	103.72	8.00	55.31	5.59
Good	159	64.1%	91.00	11.63	49.59	6.44
Fair	37	14.9%	82.18	11.30	45.62	6.17
Poor/terrible	12	4.9%	68.66	9.40	37.81	6.98
School endorsement***
Would recommend this school	230	92.7%	91.91	12.83	49.96	6.96
Neutral	15	6.0%	76.86	12.21	41.73	7.86
Would not recommend this school	3	1.3%	68.66	14.43	41.50	3.53
	Mean	SD
Age	20.07	1.97

Abbreviations: JHLES, Johns Hopkins Learning Environment Scale; LE, learning environment; MSLES, Medical School Learning Environment Scale.

Significant differences between second versus third (P ⩽ .001), second versus fourth (P ⩽ .001), and second versus fifth (P < .001) for both JHLES and MSLES. **Significant differences: poor terrible versus fair (P = .002) for JHLES and MSLES, fair versus good (P = .004) for the MSLES and all other combinations of options for both JHLES and MSLES (P < .001). ***Significant differences: would recommend versus neutral (P < .001) and versus not recommend (P = .006) in the JHLES and between would recommend versus not recommend in the MSLES (P < .001).

Medical School Learning Environment Scale

Principal component analysis

The data were appropriate for the PCA procedure, demonstrating a Kaiser–Meyer of 0.818 and Bartlett index of 922.359 (P < .001). Comparing MSLES structure after carrying out factor analysis on the Brazilian MSLES, we found substantial differences between it and the original version.²⁷ Although the original MSLES had 7 domains, the PCA of the Brazilian MSLES resulted in only 4 factor scales, which we named (Table 2): (1) Students, (2) Exams, (3) External Problems, and (4) School. These 4 dimensions accounted for 51.64% of variance, with each domain accounting for (1) 15.32%, (2) 12.93%, (3) 12.82%, and (4) 10.56%. Items 14, 10, and 15 were factorially complex (loaded highly on more than 1 factor). The complete rotated component matrix is presented in Table 2.

Table 2.

Principal components analysis of the Medical School Learning Environment Scale (MSLES).

	Component
	1Students	2Exams	3External Problems	4School
MSLES5 Students spend time assisting each other	.745
MSLES4 Students in the school get to know each other well	.725
MSLES2 Upper-level students provide informal guidance to lower level students	.684
MSLES3 Students gather together for informal activities	.662
MSLES14 Students in the school are distant from each other	−.577		.540
MSLES12 Courses emphasize the interdependence of facts, concepts, and principles		.780
MSLES13 Exams provide a fair measure of student achievement		.724
MSLES11 Exams emphasize understanding of concepts		.673
MSLES10 The relationship between basic science and clinical material is unclear		−.560	.407
MSLES17 Students are reluctant to share with each other problems they are having			.753
MSLES8 Students have difficulty finding time for family and friends			.516
MSLES9 Competition for grades is intense			.502
MSLES6 Students hesitate to express their opinions and ideas to faculty			.486
MSLES16 Faculty, administrators, and staff give personal help to students having academic difficulty				.759
MSLES7 Students’ complaints are responded to with meaningful action				.624
MSLES1 The environment of the school allows for interests outside of medicine				.527
MSLES15 Faculty are reserved and distant with students			.498	−.504
Cronbach alphas—Global = 0.809	0.781	0.688	0.560	0.625

Abbreviation: MSLES, Medical School Learning Environment Scale.

Internal consistency

Cronbach coefficient was 0.809 for the total scale and was 0.781 for the subscale Students, 0.688 for Exams, 0.560 for External Problems, and 0.625 for School (Table 2).

Test–retest

The 45-day test–retest comparison resulted in a Pearson correlation coefficient of 0.757 (P < .001) and a “single measures” ICC of 0.757 (P < .001).

Concurrent validity

The MSLES Global Score showed a significant (P < .05) and positive correlation with JHLES Total Score (r = 0.749).

Convergent validity

The MSLES Global Score showed a significant (P < .05) and positive correlation with LE school endorsement (r = 0.321), and with Overall LE Perception (r = 0.505). Moreover, global MSLES showed a significant negative correlation with all DASS 21 domains: Depression (r = –0.256), Anxiety (r = –0.304), and Stress (r = –0.222). Finally, contributing to convergent validity, a significant (P < .05) difference was found between MSLES global score and students’ overall perception of their LE, with greater MSLES scores meaning a greater LE overall perception, such as exceptional (mean = 55.31; SD = 5.59), good (mean = 49.59; SD = 6.44), fair (mean = 45.62; SD = 6.17), and poor/terrible (mean = 37.81; SD = 6.98). Also, MSLES means were also higher (P < .05) for those who recommended their school (mean = 49.96, SD = 6.96) in comparison with those who did not (mean = 41.50; SD = 3.53).

Johns Hopkins Learning Environment Scale (JHLES)

Principal components analysis

Table 3 summarizes the factor analysis for the JHLES. A Kaiser–Meyer of 0.887 and Barlett index of 2694.631 (P < .001) suggest that this data set is appropriate for carrying out the PCA procedure. Similar to the American factor analysis,¹³ our PCA analysis yielded 7 domains: (1) Faculty Relationships (2) Community of Peers, (3) Academic Climate, (4) Meaningful Engagement, (5) Physical Space, (6) Inclusion and Safety, and (7) Mentoring. All 7 dimensions accounted for 61.36% of variance, with each domain accounting for 13.17%, 11.76%, 10.02%, 9.00%, 6.38%, 6.29%, and 4.71% of the variance, respectively. Items 6, 15, and 19 were factorially complex (loaded highly on more than 1 factor).

Table 3.

Principal components analysis of the JHLES.

	Component
	1Faculty Relationships	2Community of peers	3Academic Climate	4Meaningful Engagement	5Physical Space	6Inclusion and Safety	7Mentoring
JHLES8 I feel that my medical school faculty members have taken the time to get to know me.	.746
JHLES9 I feel that my medical school faculty I encounter genuinely care about my well-being.	.715
JHLES12 The faculty advisors in the Colleges Advisory Program are readily accessible and interested in students.	.689
JHLES11 There are faculty members that I feel comfortable confiding in when important concerns come up.	.588
JHLES19 My medical school is flexible and responsive to my needs as a student.	.553		.511
JHLES7 I feel that my medical school faculty I encounter are supportive of my professional goals.	.545
JHLES20 I feel that I have a say in decision making about courses and curricular changes.	.537
JHLES1 How connected do you feel to other students?		.833
JHLES5 To what extent have you felt a sense of belonging during your time as a student in your school?		.764
JHLES2 How supported do you feel in your personal and professional pursuits by other students?		.743
JHLES3 It has been easy to make friends in my medical school.		.723
JHLES4 I feel a sense of community in my medical school.		.631
JHLES17 The workload during medical school is manageable.			.753
JHLES14 I feel that course exams and assessments test my knowledge and abilities fairly.			.705
JHLES13 Our medical school’s curriculum allows me to use my preferred learning style.			.621
JHLES16 To what extent do you trust that the institution has fulfilled your needs as a medical student?			.515
JHLES15 I understand the goals and objectives of my medical school curriculum.			.485	.456
JHLES21 My medical school encourages scholarship and innovation.				.709
JHLES10 I’ve encountered an abundance of positive, inspiring faculty role models at my medical school.				.665
JHLES18 My medical school engages students as meaningful participants.				.525
JHLES6 I’ve encountered an abundance of positive, inspiring role models among fellow students in my medical school.		.412		.503
JHLES27 The preclinical building has a significant effect on my perception of the learning environment.					.810
JHLES28 The work spaces where clinical teaching occurs contributes positively to my sense of the learning environment.					.800
JHLES25 I sense there is discrimination based on gender, race, ethnicity, or sexual identity at my medical school						−.757
JHLES26 I feel concerned at times for my personal safety at my medical school.						−.678
JHLES24 I am concerned that students are mistreated in my medical school.						−.524
JHLES22 I’ve found a mentor in a research field that interests me.							.746
JHLES23 I’ve found a mentor in a clinical specialty or discipline that I am passionate about.							.653
Cronbach alphas—Global = 0.901	0.838	0.832	0.795	0.724	0.735	0.515	0.370

Abbreviation: JHLES, Johns Hopkins Learning Environment Scale.

Internal consistency

According to the cutoff previously mentioned, the Cronbach alpha was above 0.70 for the total scale (0.901), and for most subscales, as follows: (1) Faculty Relationships (0.838), (2) Community of Peers (0.832), (3) Academic Climate (0.795), (4) Meaningful Engagement (0.724), (5) Physical Space (0.735), with exception of the subscales (6) Inclusion and Safety (0.515), and (7) Mentoring (0.370) (see Table 3).

Test–retest

The 45-day test–retest comparison resulted in a Pearson correlation coefficient of 0.697 (P < .001) and a “single measures” ICC of 0.697 (P < .001).

Concurrent Validity

The MSLES Global Score showed a significant (P < .05) and positive correlation with JHLES Total Score (r = 0.749).

Convergent validity

The JHLES also showed convergent validity (see Table 4). A positive and significant (lowest P < .01) correlation was found with LE perception (r = 0.579) and school recommendation (r = 0.457). Moreover, JHLES means were higher (P < .05) for those who endorsed their school (mean = 91.91, SD = 12.83) in contrast to those who did not (mean = 68.66; SD = 14.43) and those who were neutral (mean = 76.86; SD = 12.21). Also, a significant (P < .05) difference was found between the JHLES global score and all options of their overall perception of their LE: exceptional (mean = 103.72; SD = 8.0), good (mean = 91.0; SD = 11.63), fair (mean = 82.18; SD = 11.30), and poor/terrible (mean = 68.66; SD = 9.40). Finally, negative and significant (P < .01) correlations were found for all 3 DASS subdomains: Depression (r = –0.241), Anxiety (r = –0.277), and Stress (r = –0.181).

Table 4.

Correlations between JHLES/MSLES and DASS 21, LE perception, and school recommendation.

	JHLES Global Score	MSLES Global Score
JHLES Global Score	1.00	0.749**
JHLES Faculty	0.861**	0.665**
JHLES Community	0.656**	0.459**
JHLES Academic	0.784**	0.639**
JHLES Meaningful	0.790**	0.580**
JHLES Physical	0.519**	0.396**
JHLES Safety (Reverse coded)	0.526**	0.372**
JHLES Mentoring	0.390**	0.163*
MSLES Global Score	0.749**	1.00
MSLES Students	0.510**	0.711**
MSLES Exams	0.576**	0.705**
MSLES External Problems	0.439**	0.688**
MSLES School	0.665**	0.755**
LE Perception	0.579**	0.505**
School Recommendation	0.457**	0.321**
DASS Depression	−0.241**	−0.256**
DASS Anxiety	−0.277**	−0.304**
DASS Stress	−0.181**	−0.222**

Abbreviations: DASS, Depression, Anxiety, and Stress Scale; JHLES, Johns Hopkins Learning Environment Scale; LE, learning environment; MSLES, Medical School Learning Environment Scale.

P < .05; **P < .01.

Johns Hopkins Learning Environment Scale comparison: original versus Brazilian revised factor analyses

The comparisons between the original¹³ and the revised Brazilian factorial JHLES are presented in Supplementary Material 1 to 6. First, similar to the original analysis, all 28 items loaded significantly on the final factor and were included in the Brazilian Portuguese Version. However, we found small changes on some of the subdomain components (see Supplementary Material 1). Factors 3 (Academic Climate), 5 (Physical Space), 6 (Inclusion and Safety), and 7 (Mentoring) maintained the same structure as the original version. However, subscale Community of Peers lost question 6 (“I’ve encountered an abundance of positive, inspiring role models among fellow students in my medical school”). The subscale Meaningful Engagement acquired this question and also question 10 (“I’ve encountered an abundance of positive, inspiring faculty role models at my medical school”), and subscale Faculty Relationships gained questions 19 (“My medical school is flexible and responsive to my needs as a student”) and 20 (“I feel that I have a say in decision making about courses and curricular changes”) from Meaningful Engagement. Despite these changes, all subdomains of the Brazilian JHLES and the original versions had similar Cronbach alpha coefficients (Supplementary Material 2). Supplementary Materials 3 to 6 show comparisons among both original and factor revised JHLES and its correlations with each JHLES subdomain, MSLES, DASS, School Recommendation, LE Overall Perception, sex, ethnicity, semester of graduation, overall perception of the LE, and school endorsement. Generally, there is a slight nonsignificant difference between both scales, given that the magnitude and strength of associations were similar when compared with original and revised JHLES.

Discussion

In this study, MSLES and JHLES were transculturally adapted and the interpretation of scores from the instrument seemed to have appropriate validity (in content, internal structure, psychometric properties, and relation to other variables), supporting both scales’ use in educational settings. Therefore, we present them here as alternatives to other popular scales such as the DREEM, the only LE instrument translated to Brazilian Portuguese language so far.^8,9

For both newly translated scales, we found good test–retest reliability and convergent validity. Both were significantly correlated with each other, as well as with 2 questions about overall LE perception and school endorsement. The JHLES has already been shown to correlate significantly with the DREEM in previous studies,^14,25 as well as with both LE overall perception and school endorsement questions.¹³ To our knowledge, no study has compared the MSLES (short version) with any other instrument. Furthermore, JHLES and MSLES correlated significantly but negatively with the 3 DASS subdomains (depression, anxiety, and stress), as expected due to recent evidence pointing to a correlation between LE and several mental health indicators.^9,22

Although both MSLES and JHLES showed appropriate psychometric proprieties, they have differences that should be considered when using them to measure student LE perceptions. First, the stability of an instrument across cultures is important to allow reliable comparisons, in that students from different countries and backgrounds should interpret different questions in similar ways. The JHLES has been used in countries and cultures outside the United States^14,15,25,26 to make meaningful comparisons between LE perceptions, whereas the MSLES has been used mainly on US student populations.^38,39 After carrying out a PCA, the JHLES yielded the same domains in both Brazilian and US populations, whereas the MSLES did not, which suggests that the JHLES may be more appropriate to use internationally. We should highlight that, although the number of domains between the versions of the JHLES did not change, items still shifted to different domains, which also affects the internal structure. Nevertheless, the original American JHLES presented similar results as compared with the Brazilian JHLES in the analyses, and could also support a possible interchangeable use between both JHLES instruments. Second, the length of an instrument must be taken into account, especially when studying medical students, who often are overwhelmed with work and are loathed to spend time completing research questionnaires. In this regard, the MSLES is a short option, with 11 fewer questions than the JHLES.

Third, we should consider what each scale is trying to assess. Some domains of both tools seem to evaluate similar information and the answers did not differ among students. For instance, the domain Students (MSLES) is comparable with the domain Community of peers (JHLES), and the domain School (MSLES) is comparable with Faculty Relationship (JHLES). Nevertheless, the domain Exam (MSLES) seems to address more information related to the actual assessment and discipline integration whereas the Academic Climate (JHLES) is a broader domain, including workload, assessment, goals of the curriculum, and medical students’ needs. External problems (MSLES) is a mixed domain and includes sharing problems with students and faculty, having time for family/friends and competition for grades. The JHLES does not have a similar domain and these questions are assessed in other domains.

Despite some similarities between tools, JHLES seems to be more comprehensive than the MSLES. For instance, the Meaningful Engagement domain comprises several different concepts, such as faculty and peer role-modeling, and support of scholarship and innovation, both of which are not covered by the short MSLES.²⁷ In addition, the subdomains Inclusion and Safety and Mentoring are also included in JHLES and capture crucial LE aspects, such as discrimination, violence, safety, and mentoring. Finally, the assessment of the Physical Space of the medical school seems to be particularly important in developing countries such as Brazil. Some medical schools in these countries have limited physical space, a lack of simulation labs, and low patient exposure due to infrastructure problems. In sum, even though the MSLES is slightly shorter than the JHLES and has been used more often in US medical schools, the JHLES has been used across a greater variety of settings maintaining its factor structure on this population, and captured more variance in LE perceptions (ie, more comprehensive evaluation).

What do LE ratings tell us about our Brazilian students’ LE perceptions compared with other medical student populations? Our sample scores were slightly higher on the JHLES Global score (mean = 90.7), suggesting greater satisfaction with the LE, compared with India (mean = 86.2 and 82.8)²⁵ and lower compared with Taiwan (mean = 91.9),¹⁴ Malaysia (mean = 101-116),^15,26,40 Israel (mean = 100.9),²⁶ China (mean = 101.7),²⁶ and the United States (mean = 111).¹³ Curiously, when analyzing only the 2 overall LE perception questions compared with the same US/Johns Hopkins sample,¹³ Brazilian students tended to overall rank their school as having an exceptional or good LE more often than the Johns Hopkins students did (Brazil = 80.2%/United States = 77%), and Brazilian students also tended to recommend their school to a close friend more than US medical trainees (Brazil = 92.7%/United States = 81%). This brings us the question of why Brazilian medical students tend to rate their LE better when asked more globally than when asked specifically about each domain, more puzzling when we analyze a previous study that indicated lower levels of mental health among Brazilian versus US medical students.⁴ Is this difference explained due to a low criticism among Brazilian medical students toward their own LE? Or are they just less exposed than their American counterparts to the concept of LE? It seems possible that if Brazilian students are asked less often about their LE or less pushed to engage in LE criticisms by the faculty than their US peers, they may not be as exposed to the concept of LE and therefore might be more prone to desirability bias. Moreover, the mean age difference between both studies and the fact that US medical students attend a 4-year college before medical school might also play a role in explaining these differences. A single global rating may not be best for Brazilian students, whereas an LE instrument gets at more specific features of LEs and offers a more nuanced and likely valid assessment of LEs. Similar findings were presented by Ilgen et al⁴¹ in the context of simulation-based assessment of health professionals.

Our PCA might also indicate differences in LE concepts across cultures. Differently than the US study, 2 role-modeling questions (items 6 and 10) related closely to items from the JHLES Meaningful Engagement subscale which, in turn, lost 2 items (19 and 20) to the Faculty Relationship subscale. These changes might reflect different perceptions that Brazilian students hold about role-modeling and relationships with faculty. One hypothesis is that in Brazil, faculty members are considered more part of the institution, whereas at Hopkins they may be considered more as individuals. If this is true, then meaningful relationships with faculty at Hopkins/United States would be student–faculty relationships whereas in Brazil these same relationships would be student–School of Medicine/Directors relationships. Concerning role-modeling, on the contrary, it is plausible that to Brazilian students, the existence of a role model may be critical to perceiving a sense of meaning in the LE. Further cross-cultural studies are essential to understand these differences.

Finally, we found significant correlations between LE and depression, anxiety, and stress for both scales. These findings show the importance that LE has on the mental health, well-being, and burnout of students as noted by previous studies.^9,22 Strategies such as pass/fail grading systems, mental health programs, mind–body skills programs, modifying curriculum structure, multicomponent program reform, wellness programs, and advising/mentoring programs have been investigated with mixed results.⁴² The diagnosis of an institution’s LE using tools such as JHLES and MSLES can result in clinical and administrative implications, in a sense that health educators can map their medical school’s problem and act to minimize it, supporting medical students, training medical teachers, and modifying the curriculum as needed.⁴³

Limitations

First, we included only junior students (first 3 medical school years), so we cannot generalize these findings to more senior students, such as those in the clerkship years. More studies should be conducted using a large sample and including these students. Second, this is a single-institution study, and our results should be considered carefully when generalizing results to other institutions. Third, both scales and the overall LE perception items were completed at the same time. This can lead to a stronger correlation between instrument scores than if the surveys/items were completed at different times. However, it increases the response rate. Forth, our response rate was 75%. Although this response rate could be considered good, not all students have answered the questionnaire and the LE data could not be generalized for all students of our medical school. Finally, perceptions about the LE and knowledge about specific terms (such as advisory) might vary among institutions from the same country, justifying the need for a multi-institution study in Brazilian schools, such as has been done in the United States.^38,39

Conclusions

In this study, we provided good evidence for the reliability and validity of the Brazilian versions of the JHLES and MSLES, both of which appear to be useful alternatives to the DREEM. The JHLES also appeared to have advantages over the MSLES, but the choice of a scale to measure a particular construct should be based on individuals’ experiences as well as on psychometric proprieties and particular characteristics of each instrument. Furthermore, the use of the original JHLES might be preferred in contrast to the revised one created using data from our Brazilian population, when it is desirable to compare findings with previous studies from the United States,¹³ Taiwan,¹⁴ India,²⁵ China, Israel, and Malaysia.^15,26 Standardizing measurement across populations could allow for international benchmarking and cross-cultural research that would lead to a better understanding and improved quality of medical school LEs globally.

Supplemental Material

Supplementary_Material_2sep19_xyz257669895099c – Supplemental material for Measuring Students’ Perceptions of the Medical School Learning Environment: Translation, Transcultural Adaptation, and Validation of 2 Instruments to the Brazilian Portuguese Language

Supplemental material, Supplementary_Material_2sep19_xyz257669895099c for Measuring Students’ Perceptions of the Medical School Learning Environment: Translation, Transcultural Adaptation, and Validation of 2 Instruments to the Brazilian Portuguese Language by Rodolfo F Damiano, Aline O Furtado, Betina N da Silva, Oscarina da S Ezequiel, Alessandra LG Lucchetti, Lisabeth F DiLalla, Sean Tackett, Robert B Shochet and Giancarlo Lucchetti in Journal of Medical Education and Curricular Development

Footnotes

Funding:

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The paper received funding from the National Council for Scientific and Technological Developments (CNPq) from August 2017 to July 2018 and Giancarlo Lucchetti received a Research Productivity Scholarship at Level 2 (Medicine) from the Brazilian National Council for Scientific and Technological Development (CNPq).

Declaration of conflicting interests:

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author Contributions

RFD: Substantial contributions to the conception or design of the work, interpretation of data for the work, in drafting the work, final approval of the version to be published, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

AOF: Substantial contributions to the conception or design of the work; acquisition of data, revising the work critically for important intellectual content, final approval of the version to be published, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

BNS: Substantial contributions to the conception or design of the work, acquisition of data, revising the work critically for important intellectual content, and final approval of the version to be published, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

OSE: Substantial contributions to the conception or design of the work, interpretation of data for the work, revising the work critically for important intellectual content, final approval of the version to be published, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

ALGL: Substantial contributions to the conception or design of the work, interpretation of data for the work, revising the work critically for important intellectual content, final approval of the version to be published, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

LFD: Substantial contributions to the conception or design of the work; interpretation of data for the work, revising the work critically for important intellectual content, final approval of the version to be published, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

ST: Substantial contributions to the conception or design of the work; interpretation of data for the workrevising the work critically for important intellectual content, final approval of the version to be published and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

RBS: Substantial contributions to the conception or design of the work; interpretation of data for the work, revising the work critically for important intellectual content, final approval of the version to be published and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

GL: Substantial contributions to the conception or design of the work, analysis and interpretation of data for the work, and drafting the work. GL was also instrumental in final approval of the version to be published and Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Data Availability

The data used to support the findings of this study are available from the corresponding author on request.

Ethical Approval

The project has been approved by the Ethics Committee of the Federal University of Juiz de Fora.

ORCID iDs

Rodolfo F Damiano

Oscarina da S Ezequiel

Giancarlo Lucchetti

Supplemental Material

Supplemental material for this article is available online.

References

LCoME. Functions and Structure of a Medical School. https://medicine.vtc.vt.edu/content/dam/medicine_vtc_vt_edu/about/accreditation/2018-19_Functions-and-Structure.pdf. Updated 2018.

Colbert-Getz

Kim

Goode

Shochet

Wright

SM.

Assessing medical students’ and residents’ perceptions of the learning environment: exploring validity evidence for the interpretation of scores from existing tools. Acad Med. 2014;89:1687-1693.

Pacheco

Giacomin

Tam

, et al. Mental health problems among medical students in Brazil: a systematic review and meta-analysis. Braz J Psychiatry. 2017;39:369-378.

Lucchetti

Damiano

DiLalla

, et al. Cross-cultural differences in mental health, quality of life, empathy, and burnout between US and Brazilian medical students. Acad Psychiatry. 2018;42:62-67.

Tempski

Bellodi

Paro

Enns

Martins

Schraiber

LB.

What do medical students think about their quality of life? A qualitative study. BMC Med Educ. 2012;12:106.

Duvivier

Boulet

Opalek

van Zanten

Norcini

Overview of the world’s medical schools: an update. Med Educ. 2014;48:860-869.

Scheffer

Dal Poz

MR.

The privatization of medical education in Brazil: trends and challenges. Hum Resour Health. 2015;13:96.

de Oliveira Filho

Vieira

Schonhorst

Psychometric properties of the Dundee Ready Educational Environment Measure (DREEM) applied to medical residents. Med Teach. 2005;27:343-347.

Tempski

Santos

Mayer

, et al. Relationship among medical student resilience, educational environment and quality of life. PLoS ONE. 2015;10:e0131535.

10.

Roff

McAleer

Harden

, et al. Development and validation of the Dundee Ready Education Environment Measure (DREEM). Med Teach. 1997;19:295-299.

11.

Jakobsson

Danielsen

Edgren

Psychometric evaluation of the Dundee Ready Educational Environment Measure: Swedish version. Med Teach. 2011;33: e267-e274.

12.

Dimoliatis

Vasilaki

Anastassopoulos

Ioannidis

Roff

Validation of the Greek translation of the Dundee Ready Education Environment Measure (DREEM). Educ Health (Abingdon). 2010;23:348.

13.

Shochet

Colbert-Getz

Wright

SM.

The Johns Hopkins learning environment scale: measuring medical students’ perceptions of the processes supporting professional formation. Acad Med. 2015;90:810-818.

14.

Huang

H-S

Tackett

M-J.

Learning environment assessment of clinical students at National Taiwan University. J Med Educ. 2015;19:65-78.

15.

Tackett

Bakar

Shilkofski

Coady

Rampal

Wright

Profiling medical school learning environments in Malaysia: a validation study of the Johns Hopkins Learning Environment Scale. J Educ Eval Health Prof. 2015; 12:39.

16.

Lovibond

SH.

The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther. 1995;33:335-343.

17.

Apostolo

Mendes

Azeredo

ZA.

Adaptation to Portuguese of the Depression, Anxiety and Stress Scales (DASS). Rev Lat Am Enfermagem. 2006; 14:863-871.

18.

Beaufort

De Weert-Van Oene

Buwalda

VAJ

de Leeuw

JRJ

Goudriaan

AE.

The Depression, Anxiety and Stress Scale (DASS-21) as a screener for depression in substance use disorder inpatients: a pilot study. Eur Addict Res. 2017;23:260-268.

19.

Yusoff

Abdul Rahim

Baba

Ismail

Mat Pa

Esa

AR.

The impact of medical education on psychological health of students: a cohort study. Psychol Health Med. 2013;18:420-430.

20.

Vignola

Tucci

AM.

Adaptation and validation of the depression, anxiety and stress scale (DASS) to Brazilian Portuguese. J Affect Disord. 2014;155: 104-109.

21.

Benbassat

Undesirable features of the medical learning environment: a narrative review of the literature. Adv Health Sci Educ Theory Pract. 2013;18:527-536.

22.

Dyrbye

Thomas

Harper

, et al. The learning environment and medical student burnout: a multicentre study. Med Educ. 2009;43:274-282.

23.

Yusoff

MSB

Arifin

WN.

Educational environment and psychological distress of medical students: the role of a deep learning approach. J Taibah Univ Med Sci. 2015;10:411-418.

24.

Shochet

Colbert-Getz

Levine

Wright

SM.

Gauging events that influence students’ perceptions of the medical school learning environment: findings from one institution. Acad Med. 2013;88:246-252.

25.

Sengupta

Sharma

Das

Perception of learning environment among undergraduate medical students in two different medical schools through DREEM and JHLES Questionnaire. J Clin Diagn Res. 2017;11:JC01-JC04.

26.

Tackett

Wright

Lubin

Pan

International study of medical school learning environments and their relationship with student well-being and empathy. Med Educ. 2017;51:280-289.

27.

Rosenbaum

Schwabbauer

Kreiter

Ferguson

KJ.

Medical students’ perceptions of emerging learning communities at one medical school. Acad Med. 2007;82:508-515.

28.

Marshall

RE.

Measuring the medical school learning environment. J Med Educ. 1978;53:98-104.

29.

Rothman

Ayoade

The development of a learning environment: a questionnaire for use in curriculum evaluation. J Med Educ. 1970;45:754-799.

30.

Moore-West

Harrington

Mennin

Kaufman

Distress and attitudes toward the learning environment: effects of a curriculum innovation. Res Med Educ. 1986;25:293-300.

31.

Feletti

Clarke

RM.

Review of psychometric features of the Medical School Learning Environment Survey. Med Educ. 1981;15:92-96.

32.

Epstein

Santo

Guillemin

A review of guidelines for cross-cultural adaptation of questionnaires could not bring out a consensus. J Clin Epidemiol. 2015;68:435-441.

33.

Cangussu Silva

Ezequiel

ODS

Damiano

, et al. Translation, transcultural adaptation, and validation of the empathy, spirituality, and wellness in medicine scale to the Brazilian Portuguese Language. Teach Learn Med. 2018;30:404-414.

34.

Sousa

Rojjanasrirat

Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. J Evaluat Clin Prac. 2011;17:268-274.

35.

Downing

SM.

Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37:830-837.

36.

Tang

Cui

Babenko

Internal consistency: do we really know what it is and how to assess it?

J Psych Behav Sci. 2014;2:205-220.

37.

Cortina

JM.

What is coefficient alpha? An examination of theory and applications. J Appl Psychol. 1993;78:98-104.

38.

Skochelak

Stansfield

Dunham

, et al. Medical student perceptions of the learning environment at the end of the first year: a 28-medical school collaborative. Acad Med. 2016;91:1257-1262.

39.

Smith

Dunham

Dekhtyar

, et al. Medical student perceptions of the learning environment: learning communities are associated with a more positive learning environment in a multi-institutional medical school study. Acad Med. 2016;91:1263-1269.

40.

Tackett

Shochet

Shilkofski

, et al. Learning environment assessments of a single curriculum being taught at two medical schools 10,000 miles apart. BMC Med Educ. 2015;15:105.

41.

Ilgen

Hatala

Cook

DA.

A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 2015;49:161-173.

42.

Wasson

Cusmano

Meli

, et al. Association between learning environment interventions and medical student well-being: a systematic review. JAMA. 2016;316:2237-2252.

43.

Dyrbye

Lipscomb

Thibault

. Redesigning the learning environment to promote learner well-being and professional development [published online ahead of print November 26, 2019]. Acad Med. doi:10.1097/ACM.0000000000003094.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB