9. evaluation techniques


Choose an appropriate evaluation method for each of the following situations. In each case identify

(i) The participants.
(ii) The technique used.
(iii) Representative tasks to be examined.
(iv) Measurements that would be appropriate.
(v) An outline plan for carrying out the evaluation.

(a) You are at an early stage in the design of a spreadsheet package and you wish to test what type of icons will be easiest to learn.
(b) You have a prototype for a theatre booking system to be used by potential theatre-goers to reduce queues at the box office.
(c) You have designed and implemented a new game system and want to evaluate it before release.
(d) You have developed a group decision support system for a solicitor's office.
(e) You have been asked to develop a system to store and manage student exam results and would like to test two different designs prior to implementation or prototyping.


Note that these answers are illustrative; there are many possible evaluation techniques that could be appropriate to the scenarios described.

Spreadsheet package

(i) Subjects Typical users: secretaries, academics, students, accountants, home users, schoolchildren
(ii) Technique Heuristic evaluation
(iii) Representative tasks Sorting data, printing spreadsheet, formatting cells, adding functions, producing graphs
(iv) Measurements Speed of recognition, accuracy of recognition, user-perceived clarity
(v) Outline plan Test the subjects with examples of each icon in various styles, noting responses.

Theatre booking system

(i) Subjects Theatre-goers, the general public
(ii) Technique Think aloud
(iii) Representative tasks Finding next available tickets for a show, selecting seats, changing seats, changing date of booking
(iv) Measurements Qualitative measures of users' comfort with system, measures of cognitive complexity, quantitative measures of time taken to perform task, errors made
(v) Outline plan Present users with prototype system and tasks, record their observations whilst carrying out the tasks and refine results into categories identified in (iv).

New game system

(i) Subjects The game's target audience: age, sex, typical profile should be determined for the game in advance and the test users should be selected from this population, plus a few from outside to see if it has wider appeal
(ii) Technique Think aloud
(iii) Representative tasks Whatever gameplay tasks there are - character movement, problem solving, etc.
(iv) Measurements Speed of response, scores achieved, extent of game mastered.
(v) Outline plan Allow subjects to play game and talk as they do so. Collect qualitative and quantitative evidence, follow up with questionnaire to assess satisfaction with gaming experience, etc.

Group decision support system

(i) Subjects Solicitors, legal assistants, possibly clients
(ii) Technique Cognitive walkthrough
(iii) Representative tasks Anything requiring shared decision making: compensation claims, plea bargaining, complex issues with a diverse range of expertise needed.
(iv) Measurements Accuracy of information presented and accessible, veracity of audit trail of discussion, screen clutter and confusion, confusion owing to turn-taking protocols
(v) Outline plan Evaluate by having experts walk through the system performing tasks, commenting as necessary.

Exam result management

(i) Subjects Exams officer, secretaries, academics
(ii) Technique Think aloud, questionnaires
(iii) Representative tasks Storing marks, altering marks, deleting marks, collating information, security protection
(iv) Measurements Ease of use, levels of security and error correction provided, accuracy of user
(v) Outline plan Users perform tasks set, with running verbal commentary on immediate thoughts and considered views gained by questionnaire at end.

Other exercises in this chapter

ex.9.1 (ans), ex.9.2 (ans), ex.9.3 (ans), ex.9.4 (ans), ex.9.5 (ans), ex.9.6 (tut), ex.9.7 (tut)

all exercises for this chapter