Problem Solving by Multiple Experts in a Distributed Diagnostic Context

This paper outlines results, both behavioral and methodological, of a pilot study whose objective was to develop a method for learning why experienced technicians' diagnoses of a supposedly self-diagnostic avionics system appeared to be false at rates approaching 50%, and for recommending actions to improve diagnostic performance. In this context, the cost of falsely removing a replaceable avionics module was high: thorough testing of a false ‘pull’ typically would require the better part of a day for a skilled specialist using costly test equipment, only to conclude: ‘re-tests ok’.

Fifteen subject-matter experts solved three problems concerning avionics diagnosis in a counterbalanced experimental design. Results from analysis of scored verbal protocols suggest a multiplicity of problem-solving strategies used both across as well as within individuals. They also suggest that an important factor in developing a problem-specific diagnostic strategy is the user's estimate of cost for obtaining the needed data; often the cost is estimated to be too high, and the data are foregone, even when they are believed to be available ‘somewhere in the system’. Thus, problem-solvers appeared to knowingly engage in risky decision-making behavior that reflected compromises among conflicting goals. Another result was methodological: the ‘leveraged expert’ approach to scenario-driven problem solving provides rich data and useful insights into dealing with multiple experts in a problem domain.