Objective: The objective of the present studies was to apply a novel method for structural knowledge assessment to process control to assess the potential of its measures as a training outcome. Background: Traditionally, knowledge is assessed by verbal achievement tests on the subject matter. However, traditional methods are regarded as limited in their ability to assess higher-order learning or understanding. Method: Two experiments (Experiment 1, N = 41; Experiment 2, N = 50) were conducted in which participants were given a 4-hr training session on a simulated process control task. At a later testing session, participants worked on the task for 70 min and completed knowledge tests on declarative, procedural, and structural knowledge. Structural knowledge was measured with the computer-based Association Structure Test (AST), which combines an association task and Pathfinder network on the basis of relatedness ratings. Results: In both studies, structural knowledge was significantly related to diagnostic performance, and evidence was found for internal consistency as well as convergent and predictive validity. Conclusion: Findings indicate that structural assessment with the AST shows promise as a training outcome in process control. Application: Potential applications of this research include the improvement of training design, delivery, and evaluation.
Anderson, J.R. ( 1983). A spreading activation theory of memory. Journal of Verbal Learning and Verbal Behavior, 22, 261-295.
2.
Beatty, I.D., & Gerace, W.J. ( 2002). Probing physics students’ conceptual knowledge structures through term association. American Journal of Physics , 70, 750-758.
3.
Benta, M.I. ( 2005). Studying communication networks with AGNA 2.1Cognition, Brain, Behavior, 9, 567-574.
4.
Bonato, M. ( 1990). Wissensstrukturierung mittels Struktur-Lege-Techniken: Eine graphentheoretische Analyse von Wissensnetzen [Knowledge structuring with structure-laying-techniques: A graph-theoretical analysis of knowledge networks]. Frankfurt, Germany: Peter Lang.
Burkolter, D., Kluge, A., Sauer, J., & Ritzmann, S. ( 2010). Comparative study of three training methods for enhancing process control performance: Emphasis shift training, situation awareness training and drill and practice. Computers in Human Behavior . (2010), doi:10.1016/j.chb.2010.02.011.
7.
Chase, W.G., & Simon, H.A. ( 1973). The mind’s eye in chess. In W. G. Chase (Ed.), Visual information processing (pp. 215-281). New York, NY: Academic Press.
8.
Csardi, G. ( 2009). Package "igraph": Routines for simple graphs, network analysis. R library documentation. Retrieved from http://igraph.sourceforge.net/doc/R/igraph.pdf
9.
Davis, M.A., Curtis, M.B., & Tschetter, J.D. (2003). Evaluating cognitive training outcomes: Validity and utility of structural knowledge assessment. Journal of Business and Psychology, 18, 191-206.
10.
Day, E., Arthur, W., & Gettman, D. ( 2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86, 1022-1033.
11.
Dienes, Z., & Perner, J. ( 1999). A theory of implicit and explicit knowledge. Behavioral and Brain Sciences, 22, 735-808.
12.
Eckert, A. ( 1998). Kognition und Wissensdiagnose: Die Entwicklung und empirische Überprüfung des computerunterstützten wissensdiagnostischen Instrumentariums Netzwerk-Elaborierungs-Technik (NET) [Cognition and knowledge diagnosis: The development and empirical analysis of the computer-supported method for knowledge diagnosis Network-Elaboration-Technique (NET)]. Lengerich, Germany: Pabst.
13.
Goldsmith, T.E., & Davenport, D.M. (1990). Assessing structural similarity of graphs. In R. W. Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 75-87). Norwood, NJ: Ablex.
14.
Goldsmith, T.E., & Johnson, P.J. ( 1990). A structural assessment of classroom learning. In R. W. Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 241-254). Nowood, NJ: Ablex.
15.
Goldsmith, T.E., & Kraiger, K. ( 1997). Applications of structural knowledge assessment to training evaluation. In J. K. Ford (Ed.), Improving training effectiveness in work organizations (pp. 73-96). Mahwah, NJ: Erlbaum.
16.
Gopher, D., Weil, M., & Siegel, D. ( 1989). Practice under changing priorities: An approach to the training of complex skills. Acta Psychologica, 71, 147-177.
17.
Gower, J.C. ( 1966). Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika, 53, 325-328.
18.
Hoole, E. (2006). Integrating and evaluating mathematical models of assessing structural knowledge: Comparing associative networking methodologies. Unpublished doctoral dissertation , James Madison University, Harrisonburg, VA.
19.
Kluwe, R.H. ( 1997). Acquisition of knowledge in the control of simulated technical system. Le travail humain, 60, 61-85.
20.
Kragt, H., & Landeweerd, J. (1974). Mental skills in process control. In E. Edwards & F. P. Lees (Eds.), The human operator in process control (pp. 135-145). London, England: Taylor & Francis.
21.
Kraiger, K., Ford, J.K., & Salas, E. ( 1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology Monograph, 78, 311-328.
22.
Kraiger, K., & Jung, K.M. ( 1997). Linking training objectives to evaluation criteria. In M. Quiñones & A. Ehrenstein (Eds.), Training in a rapidly changing workplace: Applications of psychological research (pp. 151-179). Washington, DC: American Psychological Association.
23.
Kraiger, K., Salas, E., & Cannon-Bowers, J.A. (1995). Measuring knowledge organization as a method for learning during training. Human Factors, 37, 804-816.
24.
Landeweerd, J.A. ( 1979). Internal representation of a process, fault diagnosis and fault correction. Ergonomics, 22, 1343-1351.
25.
Meyer, B. ( 2008). The effects of computer-elicited structural and group knowledge on complex problem solving performance. (Open access doctoral dissertation, Humboldt University Berlin, 2008 ). Retrieved from http://edoc.hu-berlin.de/dissertationen/meyer-bertolt-2008-05-16/PDF/meyer.pdf )
26.
R Development Core Team. (2007). An introduction to R. Notes on R: A programming environment for data analysis and graphics. Retrieved from http://www.r-project.org
27.
Rothe, H.-J., & Warning, J. ( 1991). Zur Identifikation von semantischen Relationen im fachspezifischen Gedächtnisbesitz von Experten vermittels Analogiebildung [On the identification of semantic relations in experts’ subject-specific memory by the means of analogy building]. Zeitschrift für Psychologie, Suppl . 11, 375-384.
28.
Rowe, A.L., Cooke, N.J., Hall, E.P., & Halgren, T.L. ( 1996). Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing. Journal of Experimental Psychology, 2, 31-47.
29.
Sauer, J., Burkolter, D., Kluge, A., Ritzmann, S., & Schüler, K. (2008). The effects of heuristic rule training on operator performance in a simulated process control environment . Ergonomics, 51, 953-967.
30.
Sauer, J., Wastell, D.G., & Hockey, G.R.J. (2000). A conceptual framework for designing microworlds for complex work domains: A case study of the Cabin Air Management System. Computers in Human Behavior, 16, 45-58.
31.
Saus, E.-R., Johnson, B.H., Eid, J., Riisem, P.K., Andersen, R., & Thayer, J.F. ( 2006). The effect of brief situational awareness training in a police shooting simulator: An experimental study. Military Psychology, 18, 3-21.
32.
Schmidt, F.L., & Hunter, J.E. ( 1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
33.
Schoofs, D., Preuss, D., & Wolf, O.T. ( 2008). Psychosocial stress induces working memory impairments in an n-back paradigm. Psychoneuroendocrinology, 33, 643-653.
Schvaneveldt, R.W. ( 2009). What is the coherence measure and when is it appropriate? Retrieved from http://www.interlinkinc.net/FAQ.html#Coherence
36.
Schvaneveldt, R.W., Durso, F.T., Goldsmith, T.E., Breen, T.J., Cooke, N.M., Tucker, R.G., & De Maio, J.C. (1985). Measuring the structure of expertise. International Journal of Man-Machine Studies , 23, 699-728.
37.
Schvaneveldt, R.W., Tucker, R., Castillo, A., & Bennett, W., Jr. (2001, November). Knowledge acquisition in distributed mission training. Paper presented at the 2001 Simulation and Education Conference, Orlando, FL.
38.
Wickens, C.D., & Hollands, J.G. (2000). Engineering psychology and human performance (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
39.
Wickens, C.D., Lee, J.D., Liu, Y., & Gordon Becker, S.E. (2004). An introduction to human factors engineering (2nd ed.). Upper Saddle River, NJ: Pearson.
