The Formative Evaluation of a Decision Support System for Designing Training Devices

Formative evaluation in the broadest sense refers to the measurement of some system in order to make direct and immediate differences in the procedures, mechanisms, and goals of that system during development. The objective of this formative evaluation is to address three areas: 1) increase our understanding of how the targeted users make decisions, 2) train the user about how the system makes decisions and presents information, and 3) develop information about interface and modeling changes needed in the system. What is needed in the design of decision aids is an understanding of both how the user perceives the decision process, and how the decisions actually could be effectively made. Both of these issues must be addressed before the user will accept and use the decision aid. The system also needs to be able to accept and use the information that the user considers necessary, as well as to present both recommendations and supporting information in acceptable formats. We have applied a structured interview within our formative evaluation process as a basis for integrating the user in the develop, revise, and deliver cycle. The structured interview was conducted on-line, demonstrating what the system does while explanations of how it works are provided. The responses have provided information about whether the user thinks the system addresses the correct issues, the users agreement with the system analyses, and a report of the users decision processes. By including the user in the review of the developing system, the design of the prototype more accurately reflects the user's decision processes, as well as providing more usable output. This study will provide some insight about one method for evaluating decision aids early in the development process.