Human in the Loop Evaluation of a Mixed-Initiative System for Planning and Control of Multiple UAV Teams

This paper describes a ‘Human in the Loop' evaluation of an early prototype mixed-initiative control system that generated plans for single operator supervision of multiple unmanned aerial vehicles (UAV) missions. The objectives of the evaluation were to assess to what extent human operators were able to understand the plans generated by the automated controllers as well as to point to additional support requirements to drive further development of mixed-initiative planning systems. Six former fighter pilots served as test participants. Multiple convergent measures were utilized to evaluate the ability of users to understand and evaluate the plans generated by the automated controller. While test participants were able to understand the plans, the results provided compelling evidence for the need to communicate more effectively the rationale behind plan elements proposed by the automated controller and to provide ‘levers' to enable the user to modify the plan. These results point to important challenges for design of ‘mixed-initiative’ controllers to enable the human and automated controller to function as effective ‘partners’.