Objective: This paper uses a review of previous studies to provide a recommendation for the optimal scale size of the Air Traffic Workload Input Technique (ATWIT). Background: The ATWIT is a measure of workload that was originally a 10-point scale, but subsequent research includes a 7-point variation of this scale. Scale size is known to impact assessment reliability, and more reliable scales produce stronger effect sizes and reduce costs that are associated with experimentation. Therefore, it is important to know whether the 7-point or 10-point version of the scale is more reliable. Method: The authors conducted a preliminary meta-analysis of 15 studies. The analysis examined correlations between ratings using the ATWIT and aircraft count (an objective measure of difficulty) to compare effect sizes across studies with a 7-point scale and a 10-point scale. Results: Findings indicated that the strength of the correlation between ATWIT ratings and aircraft count was greater for the 10-point version of the ATWIT than for the 7-point version. Conclusion: The 10-point scale appears to be more appropriate for the ATWIT. However, the authors recommend further research to examine and control for the effects of potential confounds.
References
1.
*AhlstromU.Friedman-BergF. (2006). Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics, 36, 623–636. doi: 10.1016/j.ergon.2006.04.002
2.
AllenM.YenW. (2002). Introduction to measurement theory. Long Grove, IL: Waveland Press, Inc. ISBN: 1-57766-230-X
3.
AllendoerferK.GalushkaJ. (1999). Air traffic control system baseline methodology guide (DOT/FAA/CT-TN99/15). Atlantic City International Airport, NJ: William J. Hughes Technical Center.
4.
AllendoerferK.GalushkaJ.MogfordR. (2000). Display system replacement baseline research report (DOT/FAA/CT-TN00/31). Atlantic City International Airport, NJ: William J. Hughes Technical Center.
5.
BendigA. (1954a). Reliability and the number of rating scale categories. Journal of Applied Psychology, 38, 38–40. doi: 10.1037/h0055647
6.
BendigA. (1954b). Reliability of short rating scales and the heterogeneity of rated stimuli. Journal of Applied Psychology, 38, 167–170. doi: 10.1037/h0059072
7.
BenjaminA.TullisJ.LeeJ. (2013). Criterion noise in ratings-based recognition: Evidence from the effects of response scale length on recognition accuracy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(5), 1601–1608. doi: 10.1037/a0031849
8.
BorensteinM.HedgesL.HigginsJ.RothsteinH. (2009). Introduction to meta-analysis. Hoboken, NJ: John Wiley & Sons.
9.
ChangL. (1994). A psychometric evaluation of 4-point and 6-point likert-type scales in relation to reliability and validity. Applied Psychological Measurement, 18(3), 205–215. doi: 10.1177/014662169401800302
10.
CicchettiD.ShowalterD.TyrerP. (1985). The effect of number of rating scale categories on levels of interrater reliability: A monte carlo investigation. Applied Psychological Measurement, 9, 31–36. doi: 10.1177/014662168500900103
11.
*CrutchfieldJ.RosenbergC. (2007). Predicting subjective workload ratings: A comparison and synthesis of operational and theoretical models (DOT/FAA/AM-07/6). Washington, DC: Office of Aerospace Medicine.
12.
EndsleyM.MogfordR.AllendoerferK.SnyderM.SteinE. (1997). Effect of free flight conditions on controller performance, workload, and situation awareness (DOT/FAA/CT-TN97/12). Atlantic City International Airport, NJ: William J. Hughes Technical Center.
13.
EndsleyM.RodgersM. (1997). Distribution of attention, situation awareness, and workload in a passive air traffic control task: Implications for operational errors and automation (DOT/FAA/AM-97/13). Oklahoma City, OK: FAA Civil Aeromedical Institute.
14.
*HahS.WillemsB. (2008). The relationship between aircraft count and controller workload in different en route workstation systems. In Proceedings of the 52nd Annual Meeting of the Human Factors and Ergonomics Society, 52, 44–48. doi: 10.1177/154193120805200111
15.
*HahS.WillemsB.PhillipsR. (2006). The effect of air traffic increase on controller workload. In Proceedings of the 50th Annual Meeting of the Human Factors and Ergonomics Society, 50–54. doi: 10.1177/154193120605000111
16.
HunterJ.SchmidtF. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage Publications, Inc. ISBN:1-4129-0479-X
17.
JenkinsG.TaberT. (1977). A monte carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 29, 66–68. doi: 10.1037/0021-9010.62.4.392
18.
KrosnickJ.HolbrookA.VisserP. (2006). Optimizing brief assessments in research on the psychology of aging: A pragmatic approach to survey and self-report measurement. In CarstensenL.HartelC. (Eds.), National research council (us) committee on aging frontiers in social psychology, personality, and adult development psychology (pp. 231–239). Washington, DC: National Academies Press. ISBN: 0-309-10064-X
19.
*LeeP. (2005). A non-linear relationship between controller workload and traffic count. In Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society, 49, 1129–1133. doi: 10.1177/154193120504901206
20.
*ManningC.MillsS.FoxC.PfleidererE.MogilkaH. (2001a). Investigating the validity of performance and objective workload evaluation research (POWER; DOT/FAA/AM-01/10). Washington, DC: Office of Aerospace Medicine.
21.
*ManningC.MillsS.FoxC.PfleidererE.MogilkaH. (2001b). The relationship between air traffic control communication events and measure of controller taskload and workload. In the 4th USA/Europe Air Traffic Management R&D Seminar. Santa Fe, NM. http://atmseminar.org/seminarContent/seminar4/papers/p_161_HF.pdf
22.
*ManningC.MillsS.FoxC.PfleidererE.MogilkaH. (2002). Using air traffic control taskload measures and communication events to predict subjective workload (DOT/FAA/AM-02/4).Washington, DC: Office of Aerospace Medicine.
NunnallyJ.BernsteinI. (1994) Psychometric theory (3rd Ed.). New York: McGraw-Hill, Inc. ISBN: 0-07-047849-X
25.
PasekJ.KrosnickJ. (2010). Optimizing survey questionnaire design in political science: Insights from psychology. In LeighleyJ. (Eds.), The Oxford handbook of American elections and political behavior (pp. 27–50). Oxford, UK: Oxford University Press. ISBN: 0199604517
26.
PfleidererE. (2005). The good, the not-so-bad, and the ugly: Computer-detected altitude, heading, and speed changes in en route air traffic control. In Proceedings of the Mini-Conference on Human Factors in Complex Sociotechnical Systems, (4), 1–5. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.537.4041&rep=rep1&type=pdf
27.
PrestonC.ColmanA. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1–15. doi:10.1016/S0001-6918(99)00050-5
28.
*RantanenE. (2004). Development and validation of objective performance and workload measures in air traffic control (AHFE-04-19/FAA-04-07). Savoy, IL: Aviation Human Factors Division Institute of Aviation.
29.
ShadishW. R.CookT.D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin Company. ISBN: 0395-61556-9
30.
SollenbergerR.HaleM. (2011). Human-in-the-loop investigation of variable separation standards in the en route air traffic control environment. In Proceedings of the 55th Annual Meeting of the Human Factors and Ergonomics Society, 66–70. doi: 10.1177/1071181311551014
31.
*SollenbergerR.La DueJ.CarverB.HeinzeA. (1997). Human factors evaluation of vocoders for air traffic control environments phase II: ATC simulation (DOT/FAA/CT-TN97/25). Atlantic City International Airport, NJ: Federal Aviation Administration Technical Center.
32.
*SollenbergerR.SteinE. (1995). The effects of structured arrival and departure procedures on TRACON air traffic controller memory and situation awareness (DOT/FAA/CT-TN95/27). Atlantic City International Airport, NJ: Federal Aviation Administration Technical Center.
33.
*SteinE. (1985). Controller workload: An examination of workload probe (DOT/FAA/CT-TN84/24). Atlantic City International Airport, NJ: Federal Aviation Administration Technical Center.
34.
SteinE. (1989). Parallel approach separation and controller performance (DOT/FAA/CT-TN89/50). Atlantic City International Airport, NJ: Federal Aviation Administration Technical Center.
35.
TruittT.McAnultyD.WillemsB. (2004). Effects of collocation and reduced lateral separation standards in the New York integrated control complex (DOT/FAA/CT-TN04/08). Atlantic City International Airport, NJ: Federal Aviation Administration William J. Hughes Technical Center.
36.
TruittT.MuldoonR. (2010). Data communications segment 2 airport traffic control tower human-in-the-loop simulation (DOT/FAA/TC-10/05). Atlantic City International Airport, NJ: Federal Aviation Administration William J. Hughes Technical Center.
37.
VidulichM.TsangP. (2012). Mental workload and situation awareness. In SalvendyG. (3rd Eds.), Handbook of human factors and ergonomics (3rd ed., pp. 243–273). Hoboken, NJ: John Wiley & Sons, Inc. ISBN: 978-0-470-52838-9
38.
*WillemsB.AllenR.SteinE. (1999). Air Traffic Control Specialist visual scanning II: Task load, visual noise, and intrusions into controlled airspace (DOT/FAA/CT-TN99/23). Atlantic City International Airport, NJ: Federal Aviation Administration William J. Hughes Technical Center.
39.
*WillemsB.HeineyM. (2002). Decision support automation research in the en route air traffic control environment (DOT/FAA/CT-TN02/10). Atlantic City International Airport, NJ: Federal Aviation Administration William J. Hughes Technical Center.
40.
*YangJ.RantanenE.ZhangK. (2010). The impact of time efficacy on air traffic controller situation awareness and mental workload. The International Journal of Aviation Psychology, 20(1), 74–91. doi: 10.1080/10508410903416037.
