HUMAN-COMPUTER INTERACTION SECOND EDITION
Dix, Finlay, Abowd and Beale

Search Results

Search results for evaluation
Showing 81 to 90 of 110 [<< prev] [next >>] [new search]

Chapter 11 Evaluation techniques 11.6.1 Factors distinguishing evaluation techniques Page 436

We can identify at least eight factors which distinguish different evaluation techniques and therefore help us to make an appropriate choice. These are

Chapter 11 Evaluation techniques 11.6.1 Factors distinguishing evaluation techniques Page 436

the stage in the cycle at which the evaluation is carried out

Chapter 11 Evaluation techniques 11.6.1 Factors distinguishing evaluation techniques Page 436

the style of evaluation

Chapter 11 Evaluation techniques Design vs. implementation Page 436

The first factor to affect our choice of evaluation method is the stage in the design process at which evaluation is required. As we saw earlier in this chapter, it is desirable to include evaluation of some sort throughout the design process. The main distinction between evaluation of a design and evaluation of an implementation is that in the latter case a physical artefact exists. This may be anything from a paper mock-up to a full implementation, but it is something concrete which can be tested. Evaluation of a design on the other hand precedes this stage and seeks instead to provide information to feed the development of the physical artefact.

Chapter 11 Evaluation techniques Design vs. implementation Page 436

Roughly speaking, evaluation at the design stage tends to involve design experts only and be analytic, whereas evaluation of the implementation brings in users as subjects and is experimental. There are of course exceptions to this: participatory design (see Chapter 6) involves users throughout the design process, and techniques such as cognitive walkthrough are expert based and analytic but can be used to evaluate implementations as well as designs.

Chapter 11 Evaluation techniques Design vs. implementation Page 436

Early evaluation, whether of a design or an early prototype or mock-up, will bring the greatest pay-off since problems can be easily resolved at this stage. As more commitment is made to a particular design in the implementation, it becomes increasingly difficult for changes to be made, no matter what the evaluation suggests. Ironically, the most resources are often ploughed into late evaluations. This is less profitable and should be avoided, although obviously some evaluation with users is required with a complete, or almost complete, system.

Chapter 11 Evaluation techniques Laboratory vs. field studies Page 437

We have already discussed the pros and cons of these two styles of evaluation. Laboratory studies allow controlled experimentation and observation while losing something of the naturalness of the user's environment. Field studies retain the latter but do not allow control over user activity. Ideally the design process should include both styles of evaluation, probably with laboratory studies dominating the early stages and field studies conducted with the new implementation.

Chapter 11 Evaluation techniques Subjective vs. objective Page 437

Evaluation techniques also vary according to their objectivity -- some techniques rely heavily on the interpretation of the evaluator, others would provide the same information more or less regardless of who is performing the evaluation. The more subjective techniques, such as cognitive walkthrough or think aloud, rely to a large extent on the knowledge and expertise of the evaluator, who must recognize problems and understand what the user is doing. They can be powerful if used correctly and will provide information that may not be available from more objective methods. However, the problem of evaluator bias should be recognized and avoided. One way to decrease the possibility of bias is to use more than one evaluator. Objective techniques, on the other hand, should produce repeatable results which are not dependent on the persuasion of the particular evaluator. Controlled experiments are an example of an objective measure. These avoid bias and provide comparable results but may not reveal the unexpected problem or give detailed feedback on user experience. Ideally, both objective and subjective measures should be used.

Chapter 11 Evaluation techniques Qualitative vs. quantitative measures Page 437

The type of measurement provided by the evaluation technique is also an important consideration. There are two main types: quantitative measurement and qualitative measurement. The former is usually numeric and can be easily analyzed using statistical techniques. The latter is non-numeric and is therefore more difficult to analyze, but can provide important detail which cannot be determined from numbers. The type of measure is related to the subjectivity or objectivity of the technique, with subjective techniques tending to provide qualitative measures and objective techniques, quantitative measures. This is not a hard and fast rule, however. It is sometimes possible to quantify what is in fact qualitative information by mapping it onto a scale or similar measure. A common example of this is in questionnaires where qualitative information is being sought (for example, user preferences) but a quantitative scale is used. This is also common in experimental design where factors such as the quality of the user's performance are used as dependent variables, and measured on a quantitative scale.

Chapter 11 Evaluation techniques Information provided Page 437

The level of information required from an evaluation may also vary. The information required by an evaluator at any stage of the design process may range from low-level information to enable a design decision to be made (for example, which font is most readable) to higher-level information, such as 'Is the system usable?' Some evaluation techniques, such as controlled experiments, are excellent at providing low-level information -- an experiment can be designed to measure a particular aspect of the interface. Higher-level information can be gathered using questionnaire and interview techniques which provide a more general impression of the user's view of the system.

processed in 0.003 seconds