Retrospective versus Concurrent Thinking-Out-Loud in Usability Testing

This study examined the effect of concurrent and retrospective thinking-out-loud (TOL) on the frequency and value of user verbalizations during a software usability test. Three groups of users first learned to use an off-the-shelf database management package by means of a short tutorial and then engaged in six structured tasks. Users in the Concurrent condition thought-out-loud while performing the six tasks, whereas those in Retrospective-Immediate and Retrospective-Delayed conditions thought-out-loud while watching the videotape of their interaction with the software. Results indicated that there were no significant differences among the three conditions in performance or subjective evaluation of the software. More importantly, a verbal protocol analysis revealed that users in the Retrospective conditions spent more time making statements which had high value for designers than in the Concurrent condition. The value of verbalizations generated by the Retrospective conditions were not impacted by the 24 hour delay. The results were interpreted within context of differences in workload and in terms of the trade-off between increased value gained by using the retrospective paradigm versus increased cost of additional time to conduct the usability test.