Surveying Why Students Opt In or Out of Honors Education—A Validity Study Using Cognitive Interviews

Abstract

Despite the utility of cognitive interviews, very few studies report on the practical application of this survey design procedure. Likewise, very few studies querying perceptions of honors education incorporate valid survey items that reflect the views of students of color who are eligible yet opt out of participation. We respond to these areas of inquiry by describing the development of survey items that explore why students would opt out of honors education in collegiate settings and may identify unintentional institutional barriers that prevent the participation of marginalized groups. We report on the process and outcomes of cognitive interviews used to refine our survey tool and the categorical principal components analysis (CATPCA) that not only provided additional refinements but also revealed the underlying two-component structure of the instrument.

Keywords

honors education cognitive interviews instrument development collegiate/university setting intersectionality

Get full access to this article

View all access options for this article.

References

Abes

E. S.

Jones

S. R.

McEwen

M. K.

(2007). Reconceptualizing the model of multiple dimensions of identity: The role of meaning-making capacity in the construction of multiple identities. Journal of College Student Development, 48(1), 1–22. https://doi.org/10.1353/csd.2007.0000

Bastedo

M. N.

Cumport

P. J.

(2003). Access to what? Mission differentiation and academic statrification in U.S. Public higher education. Higher Education, 46 (3), 341–359. https://doi.org/10.1023/a:1025374011204

Bernardi

R. A.

Nash

(2023). The importance and efficacy of controlling for social desirability response bias. Ethics & Behavior, 33(5), 413–429. https://doi.org/10.1080/10508422.2022.2093201

Boateng

G. O.

Neilands

T. B.

Frongillo

E. A.

Melgar-Quiñonez

H. R.

Young

S. L.

(2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 1-18. https://doi.org/10.3389/fpubh.2018.00149

Bowleg

(2008). When black + lesbian + woman ≠ black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research. Sex Roles, 59 (5-6), 312–325. https://doi.org/10.1007/s11199-008-9400-z

Bradbury-Jones

(2007). Enhancing rigour in qualitative health research: Exploring subjectivity through Peshkin’s I’s. Journal of Advanced Nursing, 59(3), 290–298. https://doi.org/10.1111/j.1365-2648.2007.04306.x

Brimeyer

T.M.

Schueths

A.M.

Smith

W.L

. (2014). Who benefits from honors: An empirical analysis of honors and non-honors students’ backgrounds, academic attitudes, and behaviors. Journal of the National Collegiate Honors Council, 15(1) 69–83. https://digitalcommons.unl.edu/nchcjournal/420/

Callahan

(2005). Identifying gifted students from underrepresented populations. Theory into Practice, 44(2), 98–104. https://doi.org/10.1207/s15430421tip4402_4

Choo

H. Y.

Ferree

M. M.

(2010). Practicing intersectionality in sociological research: A critical analysis of inclusions, interactions, and institutions in the study of inequalities. Sociological Theory, 28(2), 129–149. https://doi.org/10.1111/j.1467-9558.2010.01370.x

10.

Cognard-Black

A.J.

Spisak

A.L

. (2019). Creating a profile of an honors student: A comparison of honors and non-honors students at public research universities in the United States. Journal of the National Collegiate Honors Council, 20(1), 123–157. https://digitalcommons.unl.edu/nchcjournal/623/

11.

Cognard-Black

A.J.

Spisak

A.L

. (2021). Forging a more equitable path for honors education: Advancing racial, ethnic, and socioeconomic diversity. Journal of National Collegiate Honors Council, 22(1), 81–114. https://digitalcommons.unl.edu/nchcjournal/670/

12.

Collins

P. H

. (1993). Toward a new vision: Race, class, and gender as categories of analysis and connection. Race, Sex & Class, 1(1), 25–45. https://www-jstor-org-443.web.bisu.edu.cn/stable/41680038

13.

Collins

(2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12 (3), 229–238. https://doi.org/10.1023/a:1023254226592

14.

Crenshaw

K. W.

(1994). Mapping the margins: Intersectionality, identity politics, and violence against women of color. In Finemen

M. A.

Mykitiuk

(Eds.), The public nature of private violence (pp. 93–118). Routledge. https://doi.org/10.4324/9780203060902-12

15.

Cuádraz

G.H.

Uttal

. (1999). Intersectionality and in-depth interviews: Methodological strategies for analyzing race, class, and gender. Race, Gender, Class, 6(3), 156–186. https://www-jstor-org-443.web.bisu.edu.cn/stable/41674900

16.

Diaz

Farraggia

S. P.

Wellman

M. E.

Bottoms

B. L.

(2019). Honors education has a positive effect on college student success. In Cognard-Black

A. J.

Herron

Smith

P. J.

(Eds.), The demonstrable value of honors education: New research evidence (pp. 59–91). National Collegiate Honors Council. https://digitalcommons.unl.edu/nchcmonochap/55/

17.

Dietrich

Ehrlenspiel

(2010). Cognitive interviewing: A qualitative tool for improving questionnaires in sport science. Measurement in Physical Education and Exercise Science, 14(1), 61–60. https://doi.org/10.1080/10913670903455025

18.

Dillman

D. A.

Smyth

J. D.

Christian

L. H.

(2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley. https://doi.org/10.1002/9781394260645

19.

Fields

(2017). Discovering statistics using IBM SPSS Statistics: North American edition (5th ed.). Sage.

20.

Ford

Wright

B. L.

Trotman Scott

(2020). A matter of equity: Desegregating and integrating gifted and talented education for under-represented students of color. Multicultural Perspectives, 22(1), 28–36. https://doi.org/10.1080/15210960.2020.1728275

21.

Frasier

M. M.

(1991). Disadvantaged and culturally diverse gifted students. Journal for the Education of the Gifted, 14(3), 234–245. https://doi.org/10.1177/016235329101400305

22.

Griffin

K. A.

Museus

S. D.

(2011). Application of mixed-methods approaches to higher education and intersectional analyses. New Directions for Institutional Research, 2011(151), 15–26. https://doi.org/10.1002/ir.396

23.

Hart

C. M.

Ritchie

T. D.

Hepper

E. G.

Gebauer

J. E.

(2015). The balanced inventory of desireable responding short form (BIDR-16). Sage Open, 5(4), 1-9. https://doi.org/10.1177/2158244015621113

24.

Hébert

T. P.

McBee

M. T.

(2007). The impact of an undergraduate honors program on gifted university students. Gifted Child Quarterly, 51(2), 136–151. https://doi.org/10.1177/001698620729947

25.

Hill

J. P

. (2005). What honors students want (and expect): The views of top Michigan high school and college students. Journal of the National Collegiate Honors Council-Online Archive, 6(2), 95–107. https://digitalcommons.unl.edu/nchcjournal/177

26.

Hilton

C. E.

(2017). The importance of pretesting questionnaires: A field research example of cognitive pretesting the exercise referral quality of life scale (ER-QLS). International Journal of Social Research Methodology, 20(1), 21–34. https://doi.org/10.1080/13645579.2015.1091640

27.

Hilton

J.T.

Jordan

. (2021). The recruitment and retention of diverse students in honors: What the last twenty years of scholarship say. Journal of the National Collegiate Honors Council-Online Archive, 22(1), 115–133. https://digitalcommons.unl.edu/nchcjournal/673/

28.

Hofmeyer

Sheingold

B. H.

Taylor

. (2015). Do you understand what I mean? How cognitive interviewing can strengthen valid, reliable study instruments and dissemination products. Journal of International Education Research (JIER), 11(4), 261–268. https://doi.org/10.19030/jier.v11i4.9460

29.

Hussar

Zhang

Hein

Wang

Roberts

Cui

Smith

Bullock Mann

Barmer

Dilig

(2020). The condition of education 2020 (NCES 2020-144). U.S. Department of Education. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2020144

30.

Kool

Mainhard

Jaarsma

van Beukelen

Brekelmans

(2017). Effects of honours programme participation in higher education: A propensity score matching approach. Higher Education Research & Development, 36(6), 1222–1236. https://doi.org/10.1080/07294360.2017.1304362

31.

Lakin

(2016). Universal screening and the representation of historically underrepresented minority students in gifted education. Journal for Advanced Academics, 27(2), 139–149. https://doi.org/10.1177/1932202X16630348

32.

Larson

R. B.

(2019). Controlling social desirability bias. International Journal of Market Research, 61(5), 534–547. https://doi.org/10.1177/1470785318805305

33.

Linting

van der Kooij

(2012). Nonlinear principal components analysis with CATPCA: A tutorial. Journal of Personality Assessment, 94(1), 12–25. https://doi.org/10.1080/00223891.2011.627965

34.

Long

D. A.

McCoach

D. B.

Siegle

Callahan

C. M.

Gubbins

E. J.

(2023). Inequality at the starting line: Underrepresentation in gifted identification and disparities in early achievement. AERA Open, 9 (1), 1-25. https://doi.org/10.1177/23328584231171535

35.

Mariz

. (2016). An agenda for the future of research in honors. Journal of the National Collegiate Honors Council, 17(1), 3–8. https://digitalcommons.unl.edu/nchcjournal/513/

36.

McBee

(2006). A descriptive analysis of referral sources for gifted identification screening by race and socioeconomic status. Journal of Secondary Gifted Education, 17(2), 103–111. https://doi.org/10.4219/jsge-2006-686

37.

McCall

(2005). The complexity of intersectionality. Journal of Women in Culture and Society, 30(3), 1771–1800. https://doi.org/10.1086/426800

38.

McKay

. (2009). Predicting retention in honors programs. Journal of the National Collegiate Honors Council, 10(1), 77–87. https://digitalcommons.unl.edu/nchcjournal/253

39.

McMaster

N. C.

Vook

(2019). The contribution of intersectionality to quantitative research into educational inequalities. Review of Education, 7(2), 271–292. https://doi.org/10.1002/rev3.3116

40.

Mead

A.D

. (2018). Socioeconomic equity in honors education: Increasing numbers of first- generation and low-income students. Journal of the National Collegiate Honors Council, 19(1), 25–31. https://digitalcommons.unl.edu/nchcjournal/576

41.

Miller

A.L

. (2018). Ways we can do better: Bridging the gap between gifted education and honors colleges. Journal of the National Collegiate Honors Council -Online Archive, 19(2), 39–44. https://digitalcommons.unl.edu/nchcjournal/588/

42.

Miller

Chepp

Willson

Padilla

J. L.

Padilla

J. L.

(2014). Cognitive interviewing methodology: A sociological approach for survey question evaluation. John Wiley & Sons. https://doi.org/10.1002/9781118838860

43.

Miller

A. L.

Dumford

A. D.

(2018). Do high-achieving students benefit from honors college participation? A look at student engagement for first-year students and seniors. Journal for the Education of the Gifted, 41(3), 217–241. https://doi.org/10.1177/0162353218781753

44.

Miller

A. L.

Speirs Neumeister

K. L.

(2017). The influence of personality, parenting styles, and perfectionism on performance goal orientation in high-ability students. Journal for Advanced Academics, 28(4), 313–344. https://doi.org/10.1177/1932202X17730567

45.

National Collegiate Honors Council. (2016). NCHC 2016 census of U.S. honors programs and colleges summary table. https://cdn.ymaws.com/nchc.site-ym.com/resource/resmgr/research/NCHC2016CensusSummaryTable05.pdf

46.

National Collegiate Honors Council. (2020). Honors enrollment management: Towards a theory and practice of inclusion. https://cdn.ymaws.com/nchc.site-ym.com/resource/resmgr/docs/resourcecenter/diversity_inclusion/nchc_enrollmentmanagement9.2.pdf

47.

Nichols

Ailts

Chang

K.-L.

(2016). Why not honors: Understanding students’ decisions not to enroll and persist in honors programs. Honors in Practice, 12, 33–58. https://digitalcommons.unl.edu/nchchip/251/

48.

Nichols

T. J.

Chang

K.-L. M.

(2013). Factors influencing honors college recruitment, persistence, and satisfaction at an upper-midwest land grant university. Journal of the National Collegiate Honors Council, 14(2), 105–127. https://digitalcommons.unl.edu/nchcjournal/399

49.

Padilla

J.-L.

Benítez

(2014). Validity evidence based on response processes. Psicothema, 26(1), 136–144. https://doi.org/10.7334/psicothema2013.259

50.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

(Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Academic Press. https://doi.org/10.1016/b978-0-12-590241-0.50006-x

51.

Peshkin

(1988). In search of subjectivity. One’s own. Educational Researcher, 17(7), 17–21. https://doi.org/10.3102/0013189 ( 017007017

52.

Pittman

A.A.

(2001). Diversity issues & honors education. Innovations in Undergraduate Research and Honors Education: Proceedings of the Second Schreyer National Conference 2001. https://digitalcommons.unl.edu/nchcschreyer2

53.

Rhea

D. M.

Goodwin

(2014). High-impact recruiting: A focus group of prospective honors students. Honors in Practice-Online Archive, 10, 111–118. https://digitalcommons.unl.edu/nchchip/200

54.

Rinn

A. N.

Plucker

J. A.

(2004). We recruit them, but then what? The educational and psychological experiences of academically talented undergraduates. Gifted Child Quarterly, 48(1), 54–67. https://doi.org/10.1177/001698620404800106

55.

Rinn

A. N.

Plucker

J. A.

(2019). High-ability college students and undergraduate honors programs: A systematic review. Journal for the Education of the Gifted, 42(3), 187–215. https://doi.org/10.1177/0162353219855678

56.

Roulston

Shelton

S. A.

(2015). Reconceptualizing bias in teaching qualitative research methods. Qualitative Inquiry, 21(4), 332–342. https://doi.org/10.1177/1077800414563803

57.

Ryan

Gannon-Slater

Culbertson

M. J.

(2012). Improving survey methods with cognitive interviews in small- and medium-scale evaluations. American Journal of Evaluation, 33(3), 414–430. https://doi.org/10.1177/1098214012441499

58.

Santos

C. E.

Toomey

R. B.

(2018). Integrating an intersectionality lens in theory and research in developmental science. Envisioning the Integration of an Intersectional Lens in Developmental Science: New Directions for Child and Adolescent Development, 2018(161), 7–15. https://doi.org/10.1002/cad.20245

59.

Saukani

Ismail

N. A.

(2019). Identifying the components of social capital by categorical principal component analysis (CATPCA). Social Indicators Research, 141(2), 631–655. https://doi.org/10.1007/s11205-018-1842-2

60.

Scott

R.I.

Smith

P.J

. (2016). Demography of honors: The national landscape of honors education. Journal of National Collegiate Honors Council, 17(1), 73–91. https://digitalcommons.unl.edu/nchcjournal/512/

61.

Scott

R.L.

Smith

P.J.

Cognard-Black

A.J

. (2017). Demography of honors: The census of U.S. Honors programs and colleges. Journal of the National Collegiate Honors Council – Online Archive, 548(1), 189–224. https://digitalcommons.unl.edu/nchcjournal/548

62.

Sireci

Faulkner-Bond

(2014). Validity evidence based on test content. Psicothema, 26(1), 100–107. https://doi.org/10.7334/psicothema2013.256

63.

Spurrier

(2008). The newest “basic characteristic” of a fully developed honors program. Honors in Practice-Online Archive, 4, 191-197. https://digitalcommons.unl.edu/nchchip/66

64.

Strayhorn

T. L.

(2017). Using intersectionality in student affairs research. New Directions for Student Services, 2017 (157), 57–67. https://doi.org/10.1002/ss.2020

65.

Thompson

(2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association. https://doi.org/10.1037/10694-000

66.

Ticknor

C.S.

Frazier

A.D.

Williams

Thompson

. (2020). Using possible selves and intersectional theory to understand why students of color opt out of honors. Journal of the National Collegiate Honors Council, Online Archive, 21(2), 67–93. https://digitalcommons.unl.edu/nchcjournal/663/

67.

U.S. Department of Education (USDOE)—Office for Civil Rights. (2016). 2013-14 Civil rights data collection: A first look. https://ocrdata.ed.gov/assets/downloads/2013-14-first-look.pdf

68.

Vietze

van Herpen

S. G. A.

Dias-Broens

Severiens

S. E.

Meeuwisse

(2022). Self-selection from higher education: A meta-review of resources for academic decision-making of mainstream and underrepresented students. Research in Post-Compulsory Education, 27(3), 454–477. https://doi.org/10.1080/13596748.2022.2076057

69.

Warner

L. R.

Shields

S. A.

(2013). The intersections of sexuality, gender, and race: Identity research at the crossroads. Sex Roles, 68 (11-12), 803–810. https://doi.org/10.1007/s11199-013-0281-4

70.

Weller

B. E.

Bowen

N. K.

Faubert

S. J.

(2020). Latent class analysis: A guide to best practice. Journal of Black Psychology, 46(4), 287–311. https://doi.org/10.1177/0095798420930932

71.

Willis

G. B.

(2015). Research synthesis: The practice of cross-cultural cognitive interviewing. Public Opinion Quarterly, 79(S1), 359–395. https://doi.org/10.1093/poq/nfu092

72.

Witenko

Mireles-Rios

Rios

V. M.

(2017). Networks of encouragement: Who’s encouraging Latina/o students and White students to enroll in honors and advanced-placement courses? Journal of Latinos and Education, 16(3), 176–191. https://doi.org/10.1080/15348431.2016.1229612

73.

Wolfensberger

M. V. C.

Offringa

G. J.

(2012). Qualities honours students look for in faculty and courses. Journal of the National Collegiate Honors Council-Online Archive, 13(2), 171–182. https://digitalcommons.unl.edu/nchcjournal/361

74.

Zissette

Atujuna

Tolley

E. E.

Okumu

Auerback

J. D.

Hodder

S. L.

Aral

S. O.

Adimora

A. A.

(2020). Cognitive testing of an instrument to evaluate acceptability and use of pre-exposure prophylaxis products among women. Applied Cognitive Psychology, 34(1), 78–84. https://doi.org/10.1002/acp.3590