Research performed during the last decade on estimation of mental workload has resulted in a subset of techniques exhibiting global sensitivity. Other techniques have also been demonstrated to have sensitivity in particular applications and can be used for diagnostic purposes. This paper reviews techniques that have shown sufficient sensitivity and robustness to be used in test and evaluation. Topics such as time scale considerations, available documentation, and appropriate procedures are also discussed. Recommendations are then made for workload estimation in test and evaluation situations.
Get full access to this article
View all access options for this article.
References
1.
Antin, J. F., and Wierwille, W. W. (1984). Instantaneous measures of mental workload: An initial investigation. In Proceedings of the Human Factors Society 28th Annual Meeting (pp. 6–10). Santa Monica, CA: Human Factors and Ergonomics Society.
2.
Battiste, V., and Bortolussi, M. (1988). Transport pilot workload: A comparison of two subjective techniques. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 150–154). Santa Monica. CA: Human Factors and Ergonomics Society.
3.
Biers, D. W., Polzella, D. J. and Mcinerney, P. (1988). A physical measure of subjective workload. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 1131–1135). Santa Monica, CA: Human Factors and Ergonomics Society.
4.
Bittner, A. C., Byers, J. C., Hill, S. G.Zaklad, A. L., and Christ, R. E. (1989). Generic workload ratings of a mobile air defense system (LOS-F-H). In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1476-–1480). Santa Monica. CA: Human Factors and Ergonomics Society.
5.
Byers, J. C., Bittner, A. C., Hill, S. G., Zaklad, A. L., and Christ, R. E. (1988). Workload assessment of a remotely piloted vehicle (RPV) system. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 1145–1149). Santa Monica, CA: Human Factors and Ergonomics Society.
6.
Casali, J. G., and Wierwille, W. W. (1983). A comparison of rating scale, secondary task, physiological. and primary task workload estimation techniques in a simulated flight task emphasizing communications load. Human Factors, 25, 623–641.
7.
Casali, J. G., and Wierwille, W. W. (1984). On the measurement of pilot perceptual workload: A comparison of assessment techniques addressing sensitivity and intrusion issues. Ergonomics, 27, 1033–1050.
8.
Casper, P., Shively, R., and Hart, S. (1987). Decision support for workload assessment: Introducing W C FIELDE. In Proceedings of the Human Factors Society 31 st Annual Meeting (pp. 72–76). Santa Monica. CA: Human Factors and Ergonomics Society.
9.
Corwin, W. H., Sandry-Garza, D. L.Bifcrno, M. H., Boucek, G. P., Logan, A. L., Jonsson, J. E., and Metalis, S. A. (1989a). Assessment of crew workload measurement methods, techniques, and procedures: Volume I. Process, methods. and results (Tech. Report WRDC-TR-89-7006). Wright-Patterson Air Force Base, OH: Wright Research and Development Center. Air Force Systems Command.
10.
Corwin, W. H., Sandry-Garza, D. L., Biferno, M. H., and Boucek, G. P. (1989b). Assessment of crew workload measurement methods. techniques, and procedures. Volume II. Guidelines for the use of workload assessment techniques in aircraft certification (Tech. Report WRDC-TR-89-7006). Wright-Patterson Air Force Base, OH: Wright Research and Development Center, Air Force Systems Command.
11.
Courtright, J., and Kuperman, G. (1984). Use of SWAT in USAF system T&E. In Proceedings of the Human Factors Society 28th Annual Meeting (pp. 700–703). Santa Monica. CA: Human Factors and Ergonomics Society.
12.
Eggemeier, F. T. (1988). Properties of workload assessment techniques. In P. A. Hancock and N. Meshkati (Eds.), Human mental workload (pp. 41–62). Amsterdam: North-Holland.
13.
Eggemeier, F. T., and Wilson, G. F. (1991). Subjective and performance-based assessment of workload in multitask environments. In D. L. Damos (Ed.), Multiple task performance (pp. 217–278). London: Taylor & Francis.
14.
Eggemeier, F. T., Wilson, G. F., Kramer, A. F., and Damos, D. L. (1991). General considerations concerning workload assessment in multi-task environments. In D. L. Damos (Ed.), Multiple task performance (pp. 207–216). London: Taylor & Francis.
15.
Eggleston, R. G. (1984). A comparison of projected and measured workload ratings using the Subjective Workload Assessment Technique (SWAT). In Proceedings of the 1984 IEEE National Aerospace and Electronics Conference (pp. 827–831). New York: Institute of Electrical and Electronics Engineers.
16.
Gawron, V. J., Schiflett, S. G. and Miller, J. C. (1989). Measures of in-flight workload. In R. S. Jensen (Ed.). Aviation psychology (pp. 240–287). Aldershot, England: Gower.
17.
Geer, C. W. (1976). Analysts guide for the analysis sections of MIL-H-46855 (Tech. Report 0180-19476-1). Seattle: Boeing Aerospace Company.
18.
Hancock, P. A. and Meshkati, N. (Eds.). (1988). Human mental workload. Amsterdam: North-Holland.
19.
Harris, R. M., Hill, S. G., and Lysaght, R. J. (1989). OWLNEST: An expert system to provide operator workload guidance. In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1486–1490). Santa Monica, CA: Human Factors and Ergonomics Society.
20.
Hart, S. G., and Hauser, J. R. (1987). Inflight application of three pilot workload measurement techniques. Aviation. Space and Environmental Medicine, 58, 402–410.
21.
Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX(Task Load Index): Results of experimental and theoretical research. In P. A. Hancock and N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North-Holland.
22.
Hart, S. G., and Wickens, C. D. (1990). Workload assessment and prediction. In H. R. Booher (Ed.), Manprint: An approach to systems integration (pp. 257–296). New York: Van Nostrand Reinhold.
23.
Haworth, L., Bivens, C., and Shively, R. (1986). An investigation of single-piloted advanced cockpit and control configurations for nap-of-the-earth helicopter combat mission tasks. In Proceedings of the 1986 Meeting of the American Helicopter Society (pp. 657–672). Washington, DC: American Helicopter Society.
24.
Hill, S. G., Byers, J. C., Zaklad, A. L., and Christ, R. E. (1989). Subjective workload assessment during 48 continuous hours of LOS-F-H operations. In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1129–1133). Santa Monica, CA: Human Factors and Ergonomics Society.
25.
Hill, S. G., Zaklad, A. L., Bittner, A. C., Byers, J. C., and Christ, R. E. (1988). Workload assessment of a mobile air defense system. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 1068–1072). Santa Monica, CA: Human Factors and Ergonomics Society.
26.
Hughes, E. R., Hassoun, J. A., Ward, G. F., and Rueb, J. D. (1990). An assessment of selected workload and situation awareness metrics in a part-mission simulation (Tech. Report ASD-TR-90-5009). Wright-Patterson Air Force Base, OH: DCS for Integrated Engineering and Technical Management, Aeronautical Systems Division, Air Force Systems Command.
27.
Iavecchia, H. P., Linton, P. M., Bittner, A. C., Jr., and Byers, J. C. (1989). Operator workload in the UH-60A Black Hawk: Crew results vs. TAWL model prediction. In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1481–1485). Santa Monica, CA: Human Factors and Ergonomics Society.
28.
Itoh, Y., Hayashi, Y., Tsukui, I., and Saito, S. (1989). Heart rate variability and subjective mental workload in flight task validity of mental workload measurement using H. R. V. method. In M. J. Smith and G. Salvendy (Eds.), Work with computers: Organizational, management, stress and health aspects (pp. 209–216). Amsterdam: Elsevier.
29.
Kramer, A. F. (1991). Physiological metrics of mental workload: A review of recent progress. In D. L. Damos (Eds.), Multiple task performance (pp. 279–328). London: Taylor & Francis.
30.
Kramer, A. F., Sirevaag, E., and Braune, R. (1987). A psychophysiological assessment of operator workload during simulated flight missions. Human Factors, 29, 145–160.
31.
Lutmer, P. A., and Eggemeier, F. T. (1990, May). Effect of intervening task performance on subjective ratings of mental workload. Presented at the Fifth Mid-Central Ergonomics Meeting, Dayton, OH.
32.
Lysaght, R. J., Hill, S. G., Dick, A. O., Plamondon, B. D., Linton, P. M., Wierwille, W. W., Zaklad, A. L., Bittner, A. C., Jr., and Wherry, R. J. (1989). Operator workload: Comprehensive review and evaluation of operator workload methodologies (Tech. Report 851). Fort Bliss, TX: U.S. Army Research Institute, Field Unit.
33.
Moray, N. (1988). Mental workload since 1979. In D. J. Oborne (Ed.), International Reviews of Ergonomics (pp. 123–150). London: Taylor & Francis.
34.
Moroney, W. F., Biers, D. W., Eggemeier, F. T., and Mitchell, J. A. (1992). A comparison of two scoring procedures with the NASA task load index in a simulated flight task. In Proceedings of the 1992 National Aerospace and Electronics Conference (pp. 734–740). New York: Institute of Electrical and Electronics Engineers.
35.
NASA Ames Research Center. (n.d.). NASA Task Load Index (NASA-TLX): A paper and pencil package, Version 1.0. Moffett Field, CA: Human Performance Research Group, NASA Ames Research Center.
36.
Nataupsky, M., and Abbott, T. S. (1987). Comparison of workload measures on computer-generated primary flight displays. In Proceedings of the Human Factors Society 31st Annual Meeting (pp. 548–552). Santa Monica, CA: Human Factors and Ergonomics Society.
37.
Noy, Y. I. (1987). Theoretical review of the secondary task methodology for evaluating intelligent automobile displays. In Proceedings of the Human Factors Society 31st Annual Meeting (pp. 205–209). Santa Monica, CA: Human Factors and Ergonomics Society.
38.
Nygren, T. E. (1991). Psychometric properties of subjective workload measurement techniques: Implications for their use in the assessment of perceived mental workload. Human Factors, 33, 17–33.
39.
O'Donnell, R. D., and Eggemeier, F. T. (1986). Workload assessment methodology. In K. R. Boff, L. Kaufman, and J. Thomas (Eds.), Handbook of perception and human performance: Volume ll. Cognitive processes and performance (pp. 42/1–42/49). New York: Wiley.
40.
Ogden, G. D., Levine, J. M., and Eisner, E. J. (1979). Measurement of workload by secondary tasks. Human Factors, 21, 529–548.
41.
Reid, G. B., and Colle, H. A. (1988). Critical SWAT values for predicting operator overload. In Proceedings of the Human Factors Society 32nd Annuai Meeting (pp. 1414–1418). Santa Monica, CA: Human Factors and Ergonomics Society.
42.
Reid, G. B., and Nygren, T. E. (1988). The Subjective Workload Assessment Technique: A scaling procedure for measuring mental workload. In P. A. Hancock and N. Meshkati (Eds.), Human mental workload (pp. 185–218). Amsterdam: North-Holland.
43.
Reid, G. B., Potter, S. S., and Bressler, J. R. (1989). Subjective Workload Assessment Technique (SWAT): A user's guide (Tech. Report AAMRL-TR-89-023). Wright-Patterson Air Force Base, OH: USAF Armstrong Laboratory.
44.
Reid, G. B., Shingledecker, C. A., Hockenberger, R. L., and Quinn, T. J. (1984). Aprojective application of the subjective workload assessment technique. In Proceedings of the 1984 IEEE National Aerospace and Electronics Conference (pp. 824–826). New York: Institute of Electrical and Electronics Engineers.
45.
Roscoe, A. H. (1982). Heart rate as an in-flight measure of pilot workload. In M. L. Frazier and R. B. Crombie (Eds.), Proceedings of the workshop on flight testing to identify pilot workload and pilot dynamics (AFFTC-TR-82-5; pp. 337–349). Edwards Air Force Base, CA: Air Force Flight Test Center.
46.
Roscoe, A. H. (1987). In-flight assessment of workload using pilot ratings and heart rate. In A. H. Roscoe (Ed.), The practical assessment of pilot workload (AGARD-ograph 282). (pp. 78–82). Neuilly Sur Seine, France: Advisory Group for Aerospace Research and Development.
47.
Rosco, A. H., and Ellis, G. A. (1990). A subjective rating scale for assessing pilot workload in flight: A decade of practical use (Tech. Report 90019). Farnborough, England: Royal Aerospace Establishment.
48.
Ruggiero, F., and Fadden, D. (1987). Pilot subjective evaluation of workload during a flight test certification programme. In A. H. Roscoe (Ed.), The practical assessment of pilot workload (AGARDograph 282: pp. 32–36). Neuilly sur Seine, France: Advisory Group for Aerospace Research and Development.
49.
Shingledecker, C. A. (1980, August). Operator strategy: A neglected variable in workload assessment. Presented at the American Psychological Association Annual Meeting, Montreal, Canada.
50.
Shingledecker, C. A. (1987). In-flight workload assessment using embedded secondary radio communications tasks. In A. H. Roscoe (Ed.), The practical assessment of pilot workload (AGARDograph 282). (pp. 11–14). Neuilly Sur Seine, France: AGARD.
51.
Shively, R., Battiste, V., Matsumoto, J., Pepiton, D., Bortolussi, M., and Hart, S. G. (1987). Inflight evaluation of pilot workload measures for rotorcraft research. In Proceedings of the Fourth Symposium on Aviation Psychology (pp. 637–643). Columbus, OH: Department of Aviation, Ohio State University.
52.
Skipper, J. H., Rieger, C. A., and Wierwille, W. W. (1986). Evaluation of decision tree rating scales for mental workload estimation. Ergonomics, 29, 585–599.
53.
Tsang, P. S., and Johnson, W. W. (1989). Cognitive demands in automation. Aviation, Space and Environmental Medicine, 60, 130–135.
54.
Vicente, K. J., Thornton, D. C., and Moray, N. (1987). Spectral analysis of sinus arrhythmia: A measure of mental effort. Human Factors, 29, 171–182.
55.
Vidulich, M. A. (1989). The use of judgment matrices in subjective workload assessment: The Subjective Workload Dominance (SWORD) Technique. In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1406–1410). Santa Monica, CA: Human Factors and Ergonomics Society.
56.
Vidulich, M. A., and Bortolussi, M. R. (1988a, June). Speech recognition in advanced rotorcraft: Using speech controls to reduce manual control overload. Presented at the American Helicopter Society National Specialists' Meeting-Automation Applications in Rotorcraft, Atlanta, GA.
57.
Vidulich, M. A., and Bortolussi, M. (1988b). A dissociation of objective and subjective workload measures in assessing the impact of speech controls in advanced helicopters. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 1471–1475). Santa Monica, CA: Human Factors and Ergonomics Society.
58.
Vidulich, M. A., and Tsang, P. S. (1987). Absolute magnitude estimation and relative judgment approaches to subjective workload assessment. In Proceedings of the Human Factors Society 31st Annual Meeting (pp. 1057–1061). Santa Monica, CA: Human Factors and Ergonomics Society.
59.
Vidulich, M. A., Ward, G. F., and Schueren, J. (1991). Using the Subjective Workload Dominance (SWORD) technique for projective workload assessment. Human Factors, 33, 677–691.
60.
Ward, G. F., and Hassoun, J. A. (1990). The effects of headup display (HUD) pitch ladder articulation, pitch number, location and horizon line length on unusual altitude recovery in the F-16 (Tech. Report ASD-TR-90-5008). Wright-Patterson Air Force Base, OH: DCS for Integrated Engineering and Technical Management, Aeronautical Systems Division, Air Force Systems Command.
61.
Whitaker, L., Peters, L., and Garinther, G. (1989). Tank crew performance: Effects of speech intelligibility on target acquisition and subjective workload assessment. In Proceedings of the Human Factors Society 33rd Annual Meeting (pp. 1411–1413). Santa Monica, CA: Human Factors and Ergonomics Society.
62.
Wickens, C. D. (1984). Processing resources in attention. In R. Parasuraman and R. Davies (Eds.), Varieties of attention (pp. 63–102). New York: Academic.
63.
Wickens, C. D. (1991). Processing resources and attention. In D. L. Damos (Eds.), Multiple task performance (pp. 3–34). London: Taylor & Francis.
64.
Wierwille, W. W. (1979). Physiological measures of aircrew mental workload. Human Factors, 21, 575–593.
65.
Wierwille, W. W. (1981). Instantaneous mental workload: Concept and potential methods for measurement. In Proceedings of the 1981 IEEE International Conference on Cybernetics and Society (pp. 604–608). New York: Institute of Electrical and Electronics Engineers.
66.
Wierwille, W. W. (1988). Important remaining issues in mental workload estimation. In P. A. Hancock and N. Meshkati (Eds.), Human mental workload (pp. 315–327). Amsterdam: North-Holland.
67.
Wierwille, W. W., and Casali, J. G. (1983). A validated rating scale for global mental workload measurement applications. In Proceedings of the Human Factors Society 27th Annual Meeting (pp. 129–133). Santa Monica, CA: Human Factors and Ergonomics Society.
68.
Wierwille, W. W., Casali, J. G., Connor, S. A., and Rahimi, M. (1985). Evaluation of the sensitivity and intrusion of mental workload estimation techniques. In W. B. Rouse (Ed.), Advances in man-machine systems research (Vol. II. pp. 51–127). Greenwich, CT: JAI Press.
69.
Wierwille, W. W., and Connor, S. A. (1983). Evaluation of twenty workload assessment measures using a psychomotor task in a moving-base aircraft simulator. Human Factors, 25, 1–16.
70.
Wierwille, W. W., Rahimi, M., and Casali, J. G. (1985). Evaluation of 16 measures of mental workload using a simulated flight task emphasizing mediation activity. Human Factors, 27, 489–502.
71.
Wierwille, W. W., and Williges, R. (1978). Survey of operator workload assessment techniques (Final Tech. Report S-78-101). Blacksburg, VA: Systemetrics. Inc.
72.
Wilson, G. F., and Eggemeier, F. T. (1991). Psychophysiological assessment of workload in multi-task environments. In D. L. Damos (Ed.), Multiple task performance (pp. 329–360). London: Taylor & Francis.
73.
Wilson, G. F., and Fullenkamp, P. (1991). A comparison of pilot and WSO workload during training missions using psychophysiological data. In Proceedings of the Western European Association for Aviation Psychology: Volume II: Stress and error in aviation (pp. 27–34). Brighton. England: Western European Association for Aviation Psychology.
74.
Wilson, G. F., and O'Donnell, R. D.(1988). Measurement of operator workload with the neurophysiological workload test battery. In P. A. Hancock and N. Meshkati (Eds), Human mental workload (pp. 63–100). Amsterdam: North-Holland.
75.
Wilson, G. F., Purvis, B., Skelly, J., Fullenkamp, P., and Davis, I. (1987). Physiological data used to measure pilot workload in actual and simulator conditions. In Proceedings of the Human Factors Society 31st Annual Meeting (pp. 779–783). Santa Monica, CA: Human Factors and Ergonomics Society.
76.
Yeh, Y. Y., and Wickens, C. D. (1988). Dissociation of performance and subjective measures of workload. Human Factors, 3D, 111–120.
