Abstract
Conducting a complete human factors/ergonomics (HF/E) audit of a unique technological device can be complicated. Designing an instrument to aid HF/E practitioners in this process can help to ensure that organizational and operator needs are taken into consideration, relevant HF/E principles are followed, and future audits for similar devices are streamlined. Our team designed such an instrument to evaluate a locomotive remote control device. We provide illustrations of the process cycle and time requirements for each phase to be used as a planning tool for HF/E practitioners who are faced with a similar challenge.
Keywords
Having an advocate for human factors/ergonomics in the company smoothes the way for recommended HF/E improvements to device design.
Human factors/ergonomics (HF/E) professionals are frequently involved in various parts of the design cycle – studying the needs of the user, planning designs that balance the abilities of the user with the intent of the technology, and evaluating the usability of products. Conducting scientific, in-depth, and efficient HF/E evaluations under time constraints is a challenging task and calls for efficiently incorporating multiple methodologies.
CSX Transportation tasked the Cognitive Ergonomics Laboratory with performing an HF/E audit of the Cattron Accuspeed™ Operator Control Unit. The control unit (see Figure 1) is used in remote-controlled freight rail switchyard operations. In place of a locomotive engineer operating the engine, remote control operators (RCOs) use the control unit to control movement of the locomotive while assembling (i.e., “building”) a train.

Top of the control unit with all controls labeled.
The purpose of this article is to describe the method we used to conduct an HF/E evaluation of the control unit. Additionally, we describe challenges that arose throughout the process and how we addressed those challenges.
HF/E Audit Overview
At the time the audit was assigned to our team, a new version of the control unit was in the redesign phase and set to begin production within months. Therefore, the audit had to be performed expeditiously to ensure that any HF/E concerns could be identified and conveyed to designers before the new control unit began production. Given that the rail industry only recently began integrating remote control technology into day-to-day operations, the amount of HF/E research on the control unit was limited (Reinach & Viale, 2006).
In the early stages of our project, we identified a body of technical reports commissioned by the Federal Railroad Administration on control units. This research was conducted by Reinach and colleagues and made up nearly all the remote control unit studies in the United States. They completed a multiphase investigation on the safety of remote control operations (Reinach, 2004; Reinach & Acton, 2006; Reinach, Fadden, Gamst, Acton, & Bartlett, 2006; Reinach & Viale, 2006). Compared with these previous studies, our analyses were part of a more focused consideration of one control unit model and were conducted during a much shorter period: 5 months.
We wanted to identify sources giving rise to HF/E issues with the control unit that originated at the organizational, task, RCO, or device level. To accomplish this in a systematic way, we outlined a set of stages that consisted of methods typically leveraged by HF/E practitioners during the evaluation process.
We began by familiarizing ourselves with the control unit (Phase 1) and conducting a needs analysis for the RCO’s job (Phase 2). Once we better understood how the control unit was used and the job demands involved in its implementation, we developed a master list of HF/E standards and principles from multiple sources, including technical standards documents, principles from textbooks, and other sources (Phase 3). We then eliminated principles that did not relate to the control unit and added “future research needs” to record desired subsequent analyses.
For example, to address the principle “Keep the flow of work moving along smooth curves over space, minimizing backtracking, extra movement, delays and needless inventory” (Lehto & Buck, 2008), we opted to conduct a time-and-motion study and link analysis. This modified list of principles and future needs served as our primary evaluation instrument for the control unit and enabled us to identify gaps in our initial analysis (Phase 4). Finally, we obtained feedback from our industry contact to tailor the document for CSX decision makers (Phase 5).
Although we understood that the process would be iterative, had we realized the true extent of the iterative nature of the process, we might have excluded certain tasks and allocated more time to accomplishing others. For example, to construct the task analysis, we would have spent less time using the control unit simulator and proceeded directly to the interviews and cognitive walk-throughs. Although we originally thought we could create a reasonable task analysis by using the simulator alone, it was beneficial only in familiarizing us with overall RCO duties needed to understand certain tasks. Adding cognitive walkthrough and verbal protocol analysis techniques was much more informative when fleshing out the task analysis.
The following sections describe each phase of our process in detail. Table 1 lists the phases along with the percentage of time allocated to each to illustrate how resource intensive each step of an evaluation process can be. Subphases are listed below each phase.
Percentage of Time Spent Working on Each Phase of the Audit
Note. RCO = remote control operator; SME = subject matter expert.
Phase 1: Familiarization
To prepare us for the assignment, CSX provided our team with a TrainMaster™ Rail Operation Simulation system complete with 14 train yard scenarios and a replica control unit. As none of our team members is a certified RCO and therefore could not participate in live remote operations, this simulator enabled us to gain firsthand experience of train yard procedures and operations.
Phase 2: Needs Analysis
We conducted a multipronged analysis to better understand the needs of the RCOs and the organization. This analysis was derived from a compilation of previous methodologies and encompassed an evaluation of the needs of the organization, the needs of the users of the device, and the nature of the task in which the device was used (Salas, Wilson, Priest, & Guthrie, 2006).
At the beginning of the needs analysis phase, CSX appointed a senior RCO instructor as a confederate for the team. This subject matter expert (SME) also had multiple years of experience as an RCO and was available to answer questions throughout the process. Establishing a close relationship with an experienced SME enabled our team to learn about the resources and constraints surrounding the introduction of new technology into train yard operations. The SME also was able to express struggles with integrating technology into operations and identify areas where support for this transition could be improved.
Organizational analysis
We used a variety of resources to gain an understanding of the context and organizational culture in which the RCOs work. A review of the CSX training literature, safety manuals, and general information about CSX Transportation via its Web site, as well as interviews with CSX management, enabled us to learn about system-wide elements that influence RCO behaviors and attitudes. To better understand the general guidelines with which RCOs were expected to work, we reviewed the policies, procedures, and operating rules.
Task analysis
The first iteration of hierarchical task analyses (Kirwan & Ainsworth, 1992) started during Phase 1 of the process. Researchers used the TrainMaster Rail Operation Simulation system to identify important jobs on the switchyard.
As we progressed, we used interviews, cognitive walk-throughs, and talk-throughs with SMEs to explore each job in greater detail. Involving the SMEs at this stage was useful not only in checking the task analyses for completeness but also for identifying features of the control unit that could potentially create difficulty for some users. For example, controls that the RCOs were instructed to use for nearly all their tasks were located on the right side of the control unit. This feature might make it difficult for left-handed RCOs to effectively use the control unit.
The iterative nature of the audit process became particularly apparent to our team during the needs analysis phase. For example, the goal of the interviews during this subphase was to gather more information regarding the task analysis. However, what we learned in this subphase made the researchers aware of gaps in organizational analysis that needed to be addressed. When we revisited the organizational analysis to fill those gaps, we discovered more details that were helpful in building a more complete task analysis.
Job/person analysis
To gain a better understanding of the RCO and the job, we conducted a series of interviews with the SME, who served as the stand-in human resources representative for RCO sourcing and development. We learned about RCO selection criteria and key demographics. The physical rigors of the job, in addition to competencies required to efficiently interact with the control unit, became apparent.
For example, we learned that RCOs often operate the control unit with one hand while riding on the side of a train (see Figure 2). We noted that performing this action while simultaneously using controls on the control unit that are contralateral to the free hand might prove challenging for some users.

An operator poses to illustrate how the control unit is used to control a train while riding on the side of a box car. Although situations requiring the operator to reach across the control unit are rare, activating appropriate controls in this manner might be challenging for some operators.
Phase 3: Instrument Design Evaluation
The control unit was a combination of display panels and controls. Our team composed a master list of principles and standards related to these categories (e.g., Lehto & Buck, 2008; Sanders & McCormick, 1993; Wagner, Birt, Snyder, & Duncanson, 1996). At least two, and usually three, members of our team reviewed each principle or standard (i.e., item) on the list, discussed the relevance of the item with regard to remote control devices, and reached a unanimous decision whether to keep the item because it was relevant to the control unit, combine it with a similar item, or consider the item as not applicable. For example, control principles from the master list that related to slider controls were determined to be irrelevant because the control unit did not have these types of controls.
Additionally, our team identified Federal Aviation Administration standards that were similar to HF/E design principles (and similar design principles across HF/E handbooks), so these items were combined into one item on the evaluation instrument. After we eliminated all irrelevant principles and combined similar items, there were a total of 370 applicable items on our evaluation instrument against which we analyzed the control unit.
Phase 4: Instrument Implementation Evaluation
We used each relevant principle to guide an in-depth analysis of the control unit against that principle, noting features that were consistent with the principle or, conversely, violated it. Whenever possible, the team made recommendations about changing features on the control unit to comply with the principles. These recommendations were intended to inform designers of potential HF/E issues specific to the device (see Table 2).
Selected Items From the Evaluation Instrument Used to Guide Analysis of the RCOs’ Remote Control Unit
Note. Numeric categories follow Lehto and Buck (2008). RCO = remote control operator; FAA = Federal Aviation Administration; SME = subject matter expert.
Human Factors Design Principle for Label Character Size and Control Unit Label Width Measurements
Source. Lehto and Buck (2008) 18-10: Width of label characters.
Human Factors Design Standard for Toggle Switch Specifications and Control Unit Toggle Measurements
Note. FAA = Federal Aviation Administration.
This phase also highlighted the iterative nature of the audit. In other words, if it was not possible to immediately evaluate a principle upon inspection of the control unit, the team identified methodologies (e.g., additional task analyses, time-and-motion studies, SME interviews) from previous phases that would be necessary to gather the requisite information for the audit.
For example, we discovered a problem with one of the displays when applying HF/E principles indicating that information displayed across like devices should be consistent with regard to the order and amount of information that is presented (Wagner et al., 1996). The status display, a scrolling display used to provide the RCO with low-level information about system states, provided different information categories across control units. We noted that the inconsistency in status information across control units might create a challenge for RCOs.
At one point in the audit, we noted the need for a time-and-motion study. We conducted a link analysis of SMEs’ interactions with the control unit using the RCO training simulator to evaluate the control unit against control sequence principles, such as “If controls are usually activated in a definite sequence, arrange them to follow that sequence” (Lehto & Buck, 2008, p. 713).
Figure 3 (page 11) shows a link analysis of the control unit interactions during flat rail yard operations with an emergency brake scenario included in the simulation. Our link analyses showed the proportion of control activations made by the RCO across the duration of the simulation and provided a graphical depiction of the sequence of control activations encountered during principal tasks. Each node corresponds to a specific control, and the node diameter indicates the total proportion of control activations made during the simulation. The links between nodes represent the order of control activations, with arrows indicating the direction of the activation sequence. The width of each link corresponds to the proportion of control activations for a particular sequence.

Link analysis depicting control unit activations made by a remote control operator during flat yard operations, including a simulated emergency scenario.
Most large links in our link analysis were the loops indicating consecutive activations of the same controls, which were obviously not a concern. However, one large link that initially appeared to be out of sequence was the one connecting the speed control (located on the right side of the control unit) to the left vigilance control (located on the left side of the control unit) and back again. However, two features of the process suggested that the control unit design was fine in this regard. First, the operators typically use two-handed operations (left hand for vigilance, right hand for speed), and second, the right vigilance switch (which is functionally identical to the left vigilance switch) is located next to the speed control and therefore complies with the principle, should the RCO choose to use this redundant control, as when hanging on the side of a car.
Product of evaluation instrument
We arranged the identified HF/E issues in a matrix comprising 15 overarching human factors concerns and 21 control unit features. Some issues had been identified by Reinach and colleagues (Reinach & Viale, 2006) as occurring in other control units, but the vast majority of potential issues we identified using the Cattron control unit had not been articulated previously.
We reported all violations, prioritized them according to perceived importance, and made change recommendations for some. We did not view some violations as particularly problematic. For example, one standard states that lines associated with each control setting should be placed at the base of a rotary control device to better indicate the current setting of the control. Although the control unit did not have these demarcations, LED lights next to the control values, along with tactile information about the position of the knob, provided redundant feedback.
Some issues were particularly important to our CSX liaison, so we provided possible solutions for those issues as well. For example, he was interested in the possible ergonomic concerns resulting from the position and range of rotation of the rotary control devices. We investigated this issue further by referencing additional ergonomics standards, computing force measurements, and making suggestions regarding alternative control options.
Phase 5: Feedback
After reviewing our deliverable, the CSX liaison provided suggestions on tailoring the document for the industry reader. As a result of this feedback, we became more sensitive to legal issues, wording, document organization, and use of scientific jargon while writing subsequent versions. We also became aware of the effectiveness of photographs to accompany our description of the HF/E issues. For example, we used Figure 2 to illustrate the difficulty that some RCOs might face when using the control unit while riding the train.
As a proponent of HF/E research, our CSX liaison was influential in setting up a meeting with the manufacturer of the control unit. As a result of the meeting, we were asked to evaluate the new prototype that was nearing production deadlines; consequently, production was delayed in order for CSX to consider some of our suggestions. Because of time constraints, we were provided with images of the prototype in place of the actual model. However, in a matter of days, we were able to effectively use the evaluation tool we created to produce a short report identifying possible HF/E issues with the new design.
Final Thoughts
Even though several features of the control unit were not consistent with HF/E design principles, these findings were only one aspect of our overall human factors audit. A number of other considerations – for example, the way in which tasks are carried out by a legacy workforce using a legacy product, and the cost of implementing new technology – played a big role in deciding which design changes to recommend to stakeholders.
For example, one standard related to lifting items close to the body to avoid injuring the back was identified as a possible issue when the control unit was evaluated against the evaluation instrument in conjunction with our task analysis. However, follow-up interviews with RCO managers revealed that company policies prohibited the RCO from lifting objects while wearing the control unit, and procedures were in place to communicate these needs to other team members. This example illustrates the importance of the multifaceted nature of our audit. Had we not considered organization-level factors and simply relied on a human factors engineering approach, we would have reported an unnecessary recommendation.
In summary, we adopted a multiphase audit to evaluate a remote control device used in the rail industry and developed an audit tool by consolidating numerous HF/E principles and standards. We then applied this tool to the analysis through an analytical and empirical approach under significant time constraints.
HF/E advocates at CSX were absolutely critical in raising the visibility of the research team within the organization and helped to ensure the availability of important resources throughout the iterative process. Such access helped to improve the overall quality of the audit, such that the proposed design of a device would further take into account human capabilities and limitations. Ultimately we were able to take the tools developed over the course of months in the initial audit and apply those issues to the new prototype in a matter of days. Because internal HF/E advocates were able to convince decision makers in the company, some (not all) of our suggestions were incorporated in the new prototype despite production delays.
