Experimentation for Envisioned Worlds: Understanding the Limitations of the Military after Action Review as Experimental Data

The research reported here was conducted to explore the efficacy of using the military After Action Review (AAR) process as a data collection methodology. During a weeklong U.S. Army experiment, we investigated, by means of a periodically administered questionnaire and an end-of-day log, the interactions participants reported having with other team members. When comparing the results of these tools, we found strong tendencies of the primacy and recency effects. These findings are important to researchers working in military command and control experimentation settings where the AAR is a commonly used data collection mechanism. This paper identifies data collection practices that can help avoid potential difficulties when using end-of-day retrospective reports as an experimental methodology.