Proceedings of the Human Factors and Ergonomics Society Annual Meeting

An experiment was conducted to examine the potential negative effects of automatic task components in situations requiring re-use or inhibition of those components. Participants trained on a category search task for 8,400 trials in three consistent (CM) and one varied mapping (VM) conditions. Following training, 2,352 trials were completed in seven transfer conditions. Results suggest that skill transfers to similar task situations. However, the data demonstrate that if the transfer situations are incompatible or prior learning must be inhibited, performance is disrupted. Although each condition improved after 336 transfer trials, performance did not reach pre-transfer levels in incompatible or inhibited conditions. The present data are useful for predicting transfer performance when skill components are trained to automaticity using a part-task methodology.

A study was performed to examine the role of consistency in the development and transfer of automatic processing. Subjects performed a rule-based memory search task in which they compared multidimensional probes to either one, two or three memory set rules. Results indicated that learning occurred in the absence of consistency at lower levels of task description (e.g. mappping of individual task components to responses) as long as higher level consistencies existed in the task (e.g. consistent mapping of task components to a conceptual framework). High positive transfer was obtained despite replacement of the exemplars of the memory set rules, suggesting that learning was not specific to the items encountered during training. On the other hand, the magnitude of positive transfer was reduced when the rules were replaced suggesting that most of the learning took place at the level of specific rules. Some evidence was also obtained for more general process-based learning.

It has been demonstrated that highly trained, automatic processes can transfer across certain memory search tasks; the degree to which these processes may be exhibited in visual search tasks has not been established, however. We examined this issue by testing the transfer of highly trained, automatized components of a semantic category, visual search task to stimulus situations of varying degrees of relatedness. We developed an adaptive version of the multiple-frame detection task (Schneider and Shiffrin, 1977) in order to test performance at the limits of visual search capacity. During training, frame-time was the dependent variable and was determined by each participant's performance ability. Each received 6,090 trials on exemplars from a single semantic category. Transfer consisted of two sessions, 330 trials per session. Transfer performance reveals that participants became highly proficient at the task: by Session 3 accuracy stabilized at around 80%. Accuracy increased in direct relation to the degree of semantic relatedness of the transfer category to the previously trained category. Frame times decreased according to a normal power function. These data demonstrate the importance of consistent training for the development of “high performance” skills and effective transfer of these skills to other, related tasks. These results have important implications for training “high performance” skills in which visual search processes a crucial role.

This investigation addresses fundamental aspects of the reliability and stability of both basic cognitive functions and automatic processing components of skills. In the present experiment we investigated the pattern of component skill retention (or decay) exhibited after training on automatic and controlled processing task components. Participants were trained on a hybrid memory/visual, semantic-category search task and received low (720 trials), moderate (2,160 trials) and high (4,320 trials) amounts of consistently mapped (CM) training plus variably mapped (VM) training (720 trials). Retention was tested at five time intervals: one day, 30 days, 90 days, 180 days and 365 days following training. Critical data for this investigation involve the pattern of performance decay across conditions and retention intervals. After an initial decline in the first 30 days following training, performance in CM conditions remained stable from Day 30 to Day 365. VM performance was erratic. The present research has practical and theoretical significance for elucidation of the mechanisms underlying long-term retention of automatic processes. Specification of these mechanisms is essential in the attempt to predict performance after a period of inactivity, to determine what constitutes appropriate refresher training, and to designate which skill components to emphasize during training.

Deep space missions such as Voyager rely upon a large team of expert analysts who monitor activity in the various engineering subsystems of the spacecraft and plan operations. Senior teammembers generally come from the spacecraft designers, and new analysts receive on-the-job training. Neither of these methods will suffice for the creation of a new team in the middle of a mission, which may be the situation during the Magellan mission. New approaches are recommended, including electronic documentation, explicit cognitive modelling, and coached practice with archived data.

Operational studies have revealed a need to focus attention on team training, and a need for effective teamwork skills for successful training performance. The present study was designed to develop an assessment scale that can be used by instructors of various training situations, which will yield a measure of the degree of teamwork required in their situations. Data obtained from the scale show psychometrically sound properties of the scale (high internal consistency and high item-total correlations) and initial validity of it (the ability to distinguish various training situations as to the extent of teamwork that is required). Recommendations for future research are also discussed.

This study investigates the usefulness of existing performance measures for evaluating the outcome effectiveness of team tasks. It describes a method to identify the measures most appropriate for evaluating training on different types of tasks and under different performance conditions. Six prototype team tasks served as rating stimuli that were used to evaluate 15 objective and 23 subjective team performance measures. Raters (N=33) assessed the usefulness of these performance measures for evaluating performance on each team task under three different scenarios. These scenarios asked how useful the measure would be for: (1) evaluating the performance of teams that want to improve and develop skills; (2) evaluating the performance of teams that have learned the task and need to maintain performance; and (3) helping a consultant to appraise the performance of the team. Results indicated reliable panel ratings; factor analyses of each objective and subjective performance measure correlation matrix revealed five-factor solutions for each domain, and these solutions were consistent across tasks and scenarios. Performance rating means varied significantly by task type, but generally were consistent across scenarios. The ratings are sensitive to task type and can be used systematically to specify relevant dimensions of team evaluation.

A Measure of Effectiveness (MOE) is a criterion. A system that scores well according to the criterion will be accepted as effective, meaning that it achieves what is intended. This seems simple, but in fact well-meaning managers can fail to find an adequate measure, are unclear about what is intended for the system, and may even misidentify the system. The method described here is a means to help avoid common mistakes.

With the method, a manager or analyst (a user) builds a model of his or her own effectiveness assessment strategy using specially designed interactive software. The user enters data prompted by the software and views feedback consisting of graphs and ordered lists describing the user's inputs in various ways. Feedback gives the users alternative viewpoints for their inputs.

As a case study, an MOE was constructed for a Soviet artillery unit within an attacking regiment.

This paper summarizes lessons learned from several projects related to embedded training (ET) and describes functional characteristics of an embedded training authoring system. Both desired and mandatory features of an ET authoring system are discussed for several applications. The relationship between embedded training and paperless technical manuals is also discussed as are engineering constraints imposed by the host system.

This paper describes a model for the implementation of embedded training (ET) which may find applications in a variety of military systems. In addition, several of the “lessons learned” during the development of this methodology are summarized. Finally, recommendations for model enhancement are discussed.

Newly developed cost and training effectiveness models are being used by training developers to control costs and to insure systematic training device design. The problem for the user is how to select the appropriate design aid. Unfortunatly, there are no quick objective methods on which to base this selection. The selection decision for a particular application can be made based on three issues. The first issue is how the design aid addresses device instructional and fidelity features. The second issue is how the design aid formalizes the device design decision process. The third issue is to compare the systems on their ease of implementation. Two decision aids are analytically evaluated on their approach to training device design: OSBATS (Optimization of Simulation Based Training Systems), which is in prototype development, and ASTAR (Automated Simulator Test and Assessment Routine), which is ready to be fielded. These decision aids are based on differing theoretical approaches to formalizing training device design. OSBATS's taxonomy of fidelity features relates instructional features to individual tasks. OSBATS contains a tradeoff function which uses historical cost and benefit values for individual features. It uses large amounts of detailed information to drive its algorithms. ASTAR is a management tool which organizes the diverse interests of a design group to address design issues. ASTAR obtains judgments about instructional approach and device similarity for each training objective. ASTAR facilitates communication between members of a design team and insures a consensus on the issues.

A major problem in training device design is specifying the appropriate level of fidelity and required instructional features for learning. This research effort was designed to acquire detailed information about tasks and training device fidelity features. The standard method for developing information about task and fidelity relationships has been to conduct research into training methods using varying degrees of fidelity, or to extrapolate from evaluations of training programs based on newly developed training devices. The rotary-wing operations domain was selected as the basis for gathering detailed relationship data. A Training Device Fidelity analysis was conducted on many of the devices at the Army Aviation School at Ft. Rucker. A survey was then developed that crossed the tasks being trained on the AH-64 CWEPT (Cockpit, Weapons, and Emergency Procedures Trainer) and the UH-1 CPT (Cockpit Procedures Trainer) with the device characteristics present on those training devices. The survey was administered to instructors using the training devices. The survey responses were categorized, and the consensus results are being used in developing expert system rules. The conclusion drawn is that adequate data can be collected using surveys to generate experience-based (versus opinion-based or device evaluation-based) rules for determining necessary and sufficient fidelity aspects for training devices. The method can be used in any training domain that requires training devices, where guidance is inferential and opinion-based, and where those devices are costly and/or need to be very effective.

The objective of this investigation was to identify air combat mission tasks that could be trained using existing multiship simulator technology. Forty-two mission ready F-15 pilots and 16 tactical air controllers rated their need for additional training on 41 air combat tasks. These pilots and controllers then participated in four days of air combat training using McDonnell Aircraft Company's simulation facility. This training allowed the participants to practice two-ship tactics in an unrestricted combat environment which included multiple air and ground threats, electronic combat, and real-time kill removal. Following training, the participants rated the value of their current unit training and training provided by the multiship simulation. Pilots rated the multiship simulator training superior to their current unit training for 22 of the 41 air combat tasks. Pilots also rated their need for additional training in those 22 combat tasks from “very” to “extremely” desirable. The controllers indicated that all combat tasks were better trained in the multiplayer simulation than in their current unit training program. Interviews and questionnaires also identified a number of strengths and weaknesses of the simulation that provide “lessons learned” for the development and use of future multiplayer air combat simulations.

This paper draws upon both an extensive review of the literature, and a series of experiments manipulating motion-based (videotaped) versus static (35-mm slide) presentations of instructional material across a variety of instructional conditions. Performance measures in the experiments included both hands-on tasks and conceptual knowledge tests. Results indicated that electromechanical maintenance performance did not differ significantly between statically and dynamically trained groups across a variety of types and complexities of electromechanical maintenance tasks and instructional strategy conditions.

Six college-age male subjects performed one hundred, two-minute trials on a second-order tracking task. After each trial, subjects estimated perceived workload using both the NASA TLX and SWAT workload assessment procedures. Results confirmed an expected performance improvement on the tracking task which followed traditional learning curves within the performance of each individual. Perceived workload also decreased for both scales across trials. While performance variability significantly decreased across trials, workload variability remained constant. One month later, the same subjects returned to complete the second experiment in the sequence which was a retention replication of the first experiment. Results replicated those for the first experiment except that both performance error and workload were at reduced overall levels. Results in general affirm a parallel workload reduction with performance improvement, an observation consistent with a resource-based view of automaticity.

Three tank gunnery trainers were studied to determine learning transfer over repeated trails. Devices included the TOPGUN trainer, a part-task, reduced-fidelity tank gunnery trainer; the Videodisk Gunnery Trainer (VIGS), another part-task, limited-fidelity trainer; and the Conduct-of-Fire Trainer (COFT), a full-fidelity trainer. The objective was to determine the degree of gunnery skills transfer between the part-task gunnery trainers and the full-fidelity simulator. COFT criterion performances were examined for two pretraining groups (either TOPGUN first, then VIGS, or VIGS first, then TOPGUN) and a control group in order to determine which pretraining sequence leads to better performance. Each training group, composed of 20 subjects, received two multiple-mission engagement trials on four consecutive days (2 VIGS-2 TOPGUN, or vice versa) before COFT transfer. Results showed significant Group and Trial effects for transfer between TOPGUN and VIGS and significant transfer to COFT performance regardless of the prior sequence of training.

Two experiments demonstrated that people who receive specific instructions (SI subjects) for using a word processor are able to accomplish initial tasks more quickly than people who receive more general instructions (GI subjects). Experiment 1 found, however, that SI subjects were unable to do a novel transfer task unless they received hints while GI subjects had no trouble with the transfer task. A production rule analysis was used to guide a revision of the specific instructions so that those instructions promoted generalization. Experiment 2 used these revised specific instructions and found that SI subjects were now able to do a novel transfer task about as well as GI subjects. These results suggest that a production system is a useful tool for analyzing instructions and predicting user performance and that specific instructions designed to promote generalization may be the most effective type of instructions.

This study attempted to determine if training and familiarization with a face composite system would improve the quality of the produced composites. Subjects were trained in the use of the Mac-a-Mug Pro system over two sessions during which they constructed eleven composites (six from memory and five with the face in view). The results indicate that the composites produced while the target face was in view were significantly better than the composites produced from memory, and that both improved with practice. Initial training with the composite system prior to exposure to the first face or after the first face did not affect composite quality. These results have implications for the training of personnel at high risk of witnessing a crime.

The area represented by this title is far too broad to cover in a short article. Therefore, rather than trying to summarize the status of the field, I will provide pointers to three recent books in the area that very adequately convey the status of the field. Some major omissions of the current research will be covered under the topic of future directions.

Advanced technologies, including artificial intelligence (Al), hypertext, and natural language processing (NLP), are transforming the Mind/Machine Interface. This presentation focuses on two large development projects underway that use these technologies in unique ways. Their use is guided by the three natural means of communication between people: saying, coaching, and showing; as metaphors for using advanced technology interfaces. The two projects are aimed at developing job and training aids for the Army. The most complete example is the Maintenance Aid Computer for HAWK–Intelligent Institutional Instructor (MACH-III). This is the largest and most successful implementation of an ITS to date (Psotka, Massey, and Mutter, 1988). MACH-III was developed by Bolt, Beranek, and Newman (BBN), to provide training in organizational maintenance of the main radar of the HAWK air defense guided missile system. Its core is a huge qualitative simulation of the radar. The complexity of the simulation and the troubleshooting problem space demand a unique hypertext interface, whose structure and function are only beginning to be understood. Some preliminary evaluation results from the U.S. Army Air Defense Artillery School (USAADASCH), Ft. Bliss, Texas are beginning to show its effectiveness. The other project, Building Robust Dual Grammar Exercisers (BRIDGE), will begin to explore the architextual structure of hypertext systems within the context of advanced technologies for military machine translation and military foreign language training. From this perspective, hypertext is a bridging technology that links the existing strengths of qualitative simulations with the future power of natural language processing.

Grace is an intelligent tutoring system (ITS) that will be used to teach programming in Cobol to about 300 NYNEX Service Company employees a year. It is the first ITS to be built by an industry laboratory for use within that industry. Grace has been a successful development project primarily because of the focus on usefulness and the use of iterative design. This paper describes Grace as a case study of finding a place for an ITS, ensuring that the users find it useful, and using prototype-evaluate cycles.

Grace, the NYNEX COBOL tutor, is being built in a corporate environment following the philosophy of iterative design and test. Grace and the student interact in a mixed-initiative dialogue. Grace's side of the dialogue is controlled by a simulation based upon the ACT* theory of cognitive skill acquisition (Anderson, 1983, 1987b). This simulation is theory-driven and largely, but not completely, embodied in a production system architecture. The student-tutor dialogue is mediated by an interface whose design is empirically driven and embodied in a multi-media system of windows, text, hypertext, mouse gestures, menus, node selections, typing-in, and so. Construction of the simulation and the tutor interface are being tested and revised through a series of user trials. The trials are conducted at one of the sites at which the tutor will be used. Students participating in the trial are from the same population as our target audience.

This study presents a critical analysis of the state of current technologies, methods, and tools used in cognitive task-analysis. Methods for cognitive task-analysis, derived from methods used in cognitive science, are relatively new and have not been systematized. Current methodologies demand considerable time and expertise to conduct properly and often yield data which is difficult to readily translate into practical application. This paper examines these problems and proposes some directions for future research and training program development.

Humans have the ability to monitor and control their conscious cognitive processes. This ability, called metacognition, implies that people can learn to optimize their cognitive processes. Recent research in metacognition provides new ways of accelerating learning and skill transfer through an improvement in the decision-making, problem-solving, and attentional skills of trainees. This paper provides a review of recent research in metacognition and presents recommendations for assessing and facilitating metacognitive skill in trainees.

This is an introduction to cognitive task analysis as it may be used in Naval Air Systems Command (NAVAIR) training development. The focus of a cognitive task analysis is human knowledge, and its methods of analysis are those developed by cognitive psychologists. This paper explains the role that cognitive task analysis can play in the development of advanced training systems and presents the findings from a preliminary cognitive task analysis of airborne weapons operators. Cognitive task analysis is a collection of powerful techniques that are quantitative, computational, and rigorous. The techniques are currently not in wide use in the training community, so examples of this methodology are presented along with the results.

As more and more work makes use of computers, the need for simple usable methods for analyzing and documenting computer-based operator tasks is increasing. Computer-based tasks can be very complex and difficult to analyze, describe, and train. Traditional methods for describing tasks are often inadequate. The purpose of this paper is to present two cases where methods were borrowed from software requirements definition and design and applied to analysis and documentation of operator tasks for complex software-based systems. The situation associated with each case is described. The methodology borrowed and adapted is then described and comments are made concerning the effectiveness of the approach. Finally, some summary comments are made.