9. evaluation techniques


9.6 What are the factors governing the choice of an appropriate evaluation method for different interactive systems? Give brief details.

answer available for tutors only

Any of the following may be included:

The stage in the cycle at which the evaluation is carried out Evaluation of a design seeks to provide information to feed the development of the physical artefact, and tends to involve design experts only and be analytic. Evaluation of an implementation evaluates the artefact itself, is more likely to bring in users as subjects and is experimental.

The style of evaluation Laboratory studies allow controlled experimentation and observation but lose something of the naturalness of the user's environment. Field studies retain the latter but do not allow control over user activity.

The level of subjectivity or objectivity of the technique More subjective techniques, e.g. cognitive walkthrough or think aloud, rely largely on the expertise of the evaluator, who must recognize problems and understand what the user is doing. Objective techniques, e.g. controlled experiments, should produce repeatable results that are not dependent on the persuasion of the particular evaluator.

The type of measures provided Quantitative measurement is usually numeric and can be easily analyzed using statistical techniques. Qualitative measurement is non-numeric and therefore more difficult to analyze, but can provide important detail that cannot be determined from numbers.

The information provided Techniques such as controlled experiments are good at providing low-level information, e.g. 'is a particular font readable?'. Higher-level information, e.g. 'is the system usable?', can be gathered using survey techniques, which provide a more general impression of the user's view of the system.

The immediacy of the response Methods such as think aloud record the user's behaviour at the time of the interaction. Others, e.g. post-task walkthrough, rely on the user's recollection of events.

The level of interference implied Techniques that are obvious to the user during the interaction run the risk of influencing the way the user behaves.

The resources required Resources to consider include equipment, time, money, subjects, expertise of evaluator and context.

Other exercises in this chapter

ex.9.1 (ans), ex.9.2 (ans), ex.9.3 (ans), ex.9.4 (ans), ex.9.5 (ans), ex.9.6 (tut), ex.9.7 (tut), ex.9.8 (open), ex.9.9 (tut)

all exercises for this chapter