Exploring the Evidence to Interpret Differential Item Functioning via Response Process Data

Abstract

Evaluating differential item functioning (DIF) in assessments plays an important role in achieving measurement fairness across different subgroups, such as gender and native language. However, relying solely on the item response scores among traditional DIF techniques poses challenges for researchers and practitioners in interpreting DIF. Recently, response process data, which carry valuable information about examinees’ response behaviors, offer an opportunity to further interpret DIF items by examining differences in response processes. This study aims to investigate the potential of response process data features in improving the interpretability of DIF items, with a focus on gender DIF using data from the Programme for International Assessment of Adult Competencies (PIAAC) 2012 computer-based numeracy assessment. We applied random forest and logistic regression with ridge regularization to investigate the association between process data features and DIF items, evaluating the important features to interpret DIF. In addition, we evaluated model performance across varying percentages of DIF items to reflect practical scenarios with different percentages of DIF items. The results demonstrate that the combination of timing features and action-sequence features is informative to reveal the response process differences between groups, thereby enhancing DIF item interpretability. Overall, this study introduces a feasible procedure to leverage response process data to understand and interpret DIF items, shedding light on potential reasons for the low agreement between DIF statistics and expert reviews and revealing potential irrelevant factors to enhance measurement equity.

Keywords

DIF Mantel–Haenszel response process data random forest ridge logistic regression

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing.

Arieli-Attali

Simmering

V. R.

(2019). Understanding examinees’ choices in a self-adapted test: A hidden Markov modeling of process data. Frontiers in Psychology, 10, 83. https://doi.org/10.3389/fpsyg.2019.00083

Bauer

D. J.

Belzak

W. C. M.

Cole

V. T.

(2020). Simplifying the assessment of measurement invariance over multiple background variables: Using regularized moderated nonlinear factor analysis to detect differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 43–55. https://doi.org/10.1080/10705511.2019.1642754

Belzak

W. C. M.

(2023). The multidimensionality of measurement bias in high-stakes testing: Using machine learning to evaluate complex sources of differential item functioning. Educational Measurement: Issues and Practice, 42(1), 24–33. https://doi.org/10.1111/emip.12486

Bosch

Paquette

(2018). Metrics for discrete student models: Chance levels, comparisons, and use cases. Journal of Learning Analytics, 5(2), 86–104. https://doi.org/10.18608/jla.2018.52.6

Bradley

A. P.

(1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7), 1145–1159. https://doi.org/10.1016/S0031-3203(96)00142-2

Breiman

(2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324

Chawla

N. V.

Bowyer

K. W.

Hall

L. O.

Kegelmeyer

W. P.

(2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953

Cheema

J. R.

(2019). Cross-country gender DIF in PISA science literacy items. European Journal of Developmental Psychology, 16(2), 152–166. https://doi.org/10.1080/17405629.2017.1358607

10.

Cheng

C. P.

Chen

C. C.

Shih

C. L.

(2020). An exploratory strategy to identify and define sources of differential item functioning. Applied Psychological Measurement, 44(7–8), 548–560. https://doi.org/10.1177/0146621620931190

11.

Cohen

A. S.

Kim

S. H.

Wollack

J. A.

(1996). An investigation of the likelihood ratio test for detection of differential item functioning. Applied Psychological Measurement, 20(1), 15–26.

12.

Collins

L. M.

Schafer

J. L.

Kam

(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351.

13.

Cook

(2018). Gender differences in adult numeracy skills: What is the role of education? Educational Research and Evaluation, 24(6–7), 370–393. https://doi.org/10.1080/13803611.2018.1540992

14.

Couronné

Probst

Boulesteix

A. L.

(2018). Random forest versus logistic regression: A large-scale benchmark experiment. BMC Bioinformatics, 19(1), 270. https://doi.org/10.1186/s12859-018-2264-5

15.

De Boeck

Jeon

. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102.

16.

Eichmann

Greiff

Brandhuber

Naumann

(2020). Using process data to explain group differences in complex problem solving. Journal of Educational Psychology, 112(8), 1546–1562. https://doi.org/10.1037/edu0000446

17.

Ercikan

Gierl

M. J.

McCreith

Puhan

Koh

(2004). Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada’s national achievement tests. Applied Measurement in Education, 17(3), 301–321.

18.

Ercikan

Guo

(2020). Use of response process data to inform group comparisons and fairness research. Educational Assessment, 25(3), 179–197. https://doi.org/10.1080/10627197.2020.1804353

19.

Fidalgo

Á. M.

Hashimoto

Bartram

Muñiz

. (2007). Empirical Bayes versus standard Mantel-Haenszel statistics for detecting differential item functioning under small sample conditions. The Journal of Experimental Education, 75(4), 293–314. https://doi.org/10.3200/JEXE.75.4.293-316

20.

Fidalgo

Á. M.

Mellenbergh

G. J.

Muñiz

. (2000). Effects of amount of DIF, test length, purification, type on robustness and power of Mantel-Haenszel procedures. Methods of Psychological Research, 5(3), 43–53.

21.

Finch

W. H.

(2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4), 278–295. https://doi.org/10.1177/0146621605275728

22.

Finch

W. H.

Hernández Finch

M. E.

(2013). Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educational and Psychological Measurement, 73(6), 973–993. https://doi.org/10.1177/0013164413494776

23.

Gardner

Brooks

(2018). Student success prediction in MOOCs. User Modeling and User-Adapted Interaction, 28(2), 127–203. https://doi.org/10.1007/s11257-018-9203-z

24.

Genuer

Poggi

J. M.

Tuleau

(2008). Random Forests: Some methodological insights (arXiv:0811.3619). arXiv. https://doi.org/10.48550/arXiv.0811.3619

25.

Ghali

Panda

Hassanien

A. E.

Abraham

Snasel

(2012). Social networks analysis: Tools, measures and visualization. In Abraham

(Ed.), Computational social networks: Mining and visualization (pp. 3–23). Springer. https://doi.org/10.1007/978-1-4471-4054-2

26.

Goldhammer

Kroehne

(2014). Controlling individuals’ time spent on task in speeded performance measures: Experimental time limits, posterior time limits and response time modeling. Applied Psychological Measurement, 38, 255–267. https://doi.org/10.1177/0146621613517164

27.

Goldhammer

Martens

Lüdtke

(2017). Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics. Large-Scale Assessments in Education, 5(18), 1–25. https://doi.org/10.1186/s40536-017-0051-9

28.

Hagberg

A. A.

Schult

D. A.

Swart

P. J.

(2008, June 28). Exploring network structure, dynamics, and function using NetworkX [Conference session]. 7th Python in Science Conference, Pasadena, CA, United States, pp. 11–15.

29.

Liao

Jiao

(2019). Clustering behavioral patterns using process data in PIAAC problem-solving items. In Veldkamp

Sluijter

(Eds.), Theoretical and practical advances in computer-based educational measurement (pp. 189–212). Springer. https://doi.org/10.1007/978-3-030-18480-3_10

30.

von Davier

(2015). Identifying feature sequences from process data in problem-solving items with N-grams. In van der Ark

Bolt

Wang

W. C.

Douglas

Chow

S. M.

(Eds.), Quantitative psychology research. Springer proceedings in mathematics & statistics (Vol. 140, pp. 173–190). Springer. https://doi.org/10.1007/978-3-319-19977-1_13

31.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129–145). Lawrence Erlbaum. https://doi.org/10.1037/14047-004

32.

Huang

Wilson

Wang

(2016). Exploring plausible causes of differential item functioning in the PISA science assessment: Language, curriculum or culture. Educational Psychology, 36(2), 378–390. https://doi.org/10.1080/01443410.2014.946890

33.

Huggins-Manley

A. C.

Booth

B. M.

D’Mello

S. K.

(2022). Toward argument-based fairness with an application to AI-enhanced educational assessments. Journal of Educational Measurement, 59, 362–388. https://doi.org/10.1111/jedm.12334

34.

James

Witten

Hastie

Tibshirani

(2013). An introduction to statistical learning with applications in R. Springer. https://doi.org/10.1007/978-1-4614-7138-7

35.

Juba

H. S.

(2019). Precision-recall versus accuracy and the role of large data sets. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), Article 01. https://doi.org/10.1609/aaai.v33i01.33014039

36.

Kleinbaum

D. G.

Klein

(1994). Logistic regression: A self-learning text. Springer. https://doi.org/10.1007/978-1-4419-1742-3

37.

Kuang

Sahin

(2023). Comparison of disengagement levels and the impact of disengagement on item parameters between PISA 2015 and PISA 2018 in the United States. Large-Scale Assessments in Education, 11(4), 1–31. https://doi.org/10.1186/s40536-023-00152-0

38.

Leng

Bezirhan

Khorramdel

Fishbein

Davier

M. von

. (2024). Examining gender differences in TIMSS 2019 using a multiple-group hierarchical speed-accuracy-revisits model. Educational Measurement: Issues and Practice, 43(3), 64–75. https://doi.org/10.1111/emip.12606

39.

Liao

Jiao

(2019). Mapping background variables with sequential patterns in problem-solving environments: An investigation of United States adults’ employment status in PIAAC. Frontiers in Psychology, 10, 646. https://doi.org/10.3389/fpsyg.2019.00646

40.

Lin

P. Y.

Lin

Y. C.

(2014). Examining student factors in sources of setting accommodation DIF. Educational and Psychological Measurement, 74(5), 759–794. https://doi.org/10.1177/0013164413514053

41.

Liou

P. Y.

Bulut

(2020). The effects of item format and cognitive domain on students’ science performance in TIMSS 2011. Research in Science Education, 50(1), 99–121. https://doi.org/10.1007/s11165-017-9682-7

42.

Lobo

J. M.

Jiménez-Valverde

Real

(2008). AUC: A misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography, 17(2), 145–151. https://doi.org/10.1111/j.1466-8238.2007.00358.x

43.

Magis

Beland

Tuerlinckx

De Boeck

(2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42, 847–862. https://doi.org/10.3758/BRM.42.3.847

44.

Magis

De Boeck

(2012). A robust outlier approach to prevent type I error inflation in differential item functioning. Educational and Psychological Measurement, 72(2), 291–311. https://doi.org/10.1177/0013164411416975

45.

Molenaar

Feskens

(2024). Relating violations of measurement invariance to group differences in response times. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000655

46.

Naidu

Zuva

Sibanda

E. M.

(2023). A review of evaluation metrics in machine learning algorithms. In Silhavy

Silhavy

(Eds.), Artificial intelligence application in networks and systems (pp. 15–25). Springer. https://doi.org/10.1007/978-3-031-35314-7_2

47.

Oliveri

M. E.

Ercikan

Zumbo

(2013). Analysis of sources of latent class differential item functioning in international assessments. International Journal of Testing, 13(3), 272–293. https://doi.org/10.1080/15305058.2012.738266

48.

Pedregosa

Varoquaux

Gramfort

Michel

Thirion

Grisel

Blondel

Prettenhofer

Weiss

Dubourg

Vanderplas

Passos

Cournapeau

Brucher

Perrot

Duchesnay

(2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825–2830.

49.

Perrone

(2006). Differential item functioning and item bias: Critical considerations in test fairness. Studies in Applied Linguistics and TESOL, 6(2), Article 2. https://doi.org/10.7916/salt.v6i2.1548

50.

Probst

Wright

M. N.

Boulesteix

A. L.

(2019). Hyperparameters and tuning strategies for random forest. WIREs Data Mining and Knowledge Discovery, 9(3), e1301. https://doi.org/10.1002/widm.1301

51.

Qiao

Jiao

(2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9, 2231. https://doi.org/10.3389/fpsyg.2018.02231

52.

Robitzsch

Rupp

A. A.

(2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and logistic regression analysis. Educational and Psychological Measurement, 69(1), 18–34. https://doi.org/10.1177/0013164408318756

53.

Sahin

Colvin

K. F.

(2020). Enhancing response time thresholds with response behaviors for detecting disengaged examinees. Large-Scale Assessments in Education, 8(5), 1–24. https://doi.org/10.1186/s40536-020-00082-1

54.

Salles

Dos Santos

Keskpaik

(2020). When didactics meet data science: Process data analysis in large-scale mathematics assessment in France. Large-Scale Assessments in Education, 8(1), 7. https://doi.org/10.1186/s40536-020-00085-y

55.

Scott

(2012). What is social network analysis?Bloomsbury Academic. https://doi.org/10.5040/9781849668187

56.

Shermis

M. D.

Mao

Mulholland

Kieftenbeld

(2017). Use of automated scoring features to generate hypotheses regarding language-based DIF. International Journal of Testing, 17(4), 351–371. https://doi.org/10.1080/15305058.2017.1308949

57.

Shermis

M. D.

St George

(1992). Item bias in mathematics achievement: The Progressive Achievement Tests for Mathematics. Psychological Test Bulletin, 5, 98–112.

58.

Shin

Wang

Pinto Junior

W. N.

Gierl

M. J.

(2024). An engagement-aware predictive model to evaluate problem-solving performance from the study of adult skills’ (PIAAC 2012) process data. Large-Scale Assessments in Education, 12, 6. https://doi.org/10.1186/s40536-024-00194-y

59.

Strobl

Boulesteix

A. L.

Zeileis

Hothorn

(2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8(1), 25. https://doi.org/10.1186/1471-2105-8-25

60.

Tang

Wang

Liu

Ying

(2020). Latent feature extraction for process data via multidimensional scaling. Psychometrika, 85(2), 378–397. https://doi.org/10.1007/s11336-020-09708-3

61.

Tang

Wang

Liu

Ying

(2021). An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology, 74(1), 1–33. https://doi.org/10.1111/bmsp.12203

62.

Ulitzsch

Molter

Nichterlein

Niedermeier

Pohl

(2021). Combining clickstream analyses and graph-modeled data clustering for identifying common response processes. Psychometrika, 86(1), 190–214. https://doi.org/10.1007/s11336-020-09743-0

63.

van Buuren

Groothuis-Oudshoorn

. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1–67. https://www.jstatsoft.org/v45/i03

64.

Wang

Tang

Liu

Ying

(2023). Subtask analysis of process data through a predictive model. British Journal of Mathematical and Statistical Psychology, 76(1), 211–235. https://doi.org/10.1111/bmsp.12290

65.

Wise

S. L.

Gao

(2017). A general approach to measuring test-taking effort on computer-based tests. Applied Measurement in Education, 30(4), 343–354. https://doi.org/10.1080/08957347.2017.1353992

66.

Woodcock

Howard

S. J.

Ehrich

(2020). A within-subject experiment of item format effects on early primary students’ language, reading, and numeracy assessment results. School Psychology, 35(1), 80–87. https://doi.org/10.1037/spq0000340

67.

A. D.

Ercikan

(2006). Using multiple-variable matching to identify cultural sources of differential item functioning. International Journal of Testing, 6(3), 287–300. https://doi.org/10.1207/s15327574ijt0603_5

68.

Zhang

Wang

Liu

Ying

(2023). Accurate assessment via process data. Psychometrika, 88(1), 76–97. https://doi.org/10.1007/s11336-022-09880-8

69.

Zhu

Shu

von Davier

A. A.

(2016). Using networks to visualize and analyze process data for educational assessment. Journal of Educational Measurement, 53(2), 190–211. https://www-jstor-org-443.web.bisu.edu.cn/stable/43940613

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.41 MB