Abstract
Fifteen usability professionals participated in a usability test under one of three simulated real time viewing conditions: (1) Screen data (S) where the evaluators saw only the image of the user's computer screen; (2) Screen plus Audio data (SA) where the user's verbalizations could be heard in addition to viewing the screen image; and (3) Screen plus Audio plus Video data (SAV) where the evaluator's additionally saw an image of the user's face in real time. Results indicated no significant differences in the total number of problems found under the three viewing conditions although there was some evidence that the problem space differed particularly from the subjective questionnaire data collected directly from the users. In rating the severity of the problems encountered, the agreement among the usability professionals was low and did not vary as a function of the number of years of the usability professional's experience. More importantly, however, ratings of severity varied as a function of viewing condition with those usability professionals in the face condition perceiving the same problems to be more severe. When considering only the most severe problems (on which there was agreement in severity ratings), the number of severe problems uncovered was less with the questionnaires than under the three real-time viewing conditions. The results are discussed in terms of real time usability evaluation and terms of the implications for remote usability testing.
Get full access to this article
View all access options for this article.