Metrics for Evaluation of Software Technology to Support Intelligence Analysis

For the past two years we have been involved in evaluation of software technologies designed to improve intelligence analysis. We have been conducting evaluations both in the laboratory and in operational environments. While usability is an important aspect of software for intelligence analysis, our work has gone beyond usability and focused on utility. We have been using top-down and bottom-up procedures to develop metrics and evaluation methodologies, including literature reviews and expert opinions, laboratory studies, baseline creation, and field observations. Our laboratory work has focused on strategic, open source analysis. Our field work to date has been more in the tactical area all source domain. In this paper, we discuss a number of metrics we have developed and outline some issues involved in evaluation of software to support intelligence analysts.