EXERCISE 9.8 [extra - not in book]
This extended exercise is designed to give you practice in writing, testing and administering a questionnaire to a true user population. Although it does not train you in the very fine points of questionnaire design, it does alert you to the basic problems in obtaining valid responses from people.
In addition to practice in valid questionnaire design and questionnaire administration, the exercise asks you to focus on finding information about a user interface to a new computer system, by studying an analogous system. Its intent is to help you develop probing skills (through good question design). These skills can then be used to find out what failures and successes users are having with a system and even the underlying causes of these successes and failures.
The 7 steps of this exercise are:
1. Selection of an analogous interface
These steps are given in more detail below. Read through them all before you begin.
1. Decide on a new user interface
for which you will collect information from potential
2. The type of information you are to obtain about the user interface through the careful design of your questionnaire is:
(a) How easy has the system been for them to learn?
From your reading about questioning people and good questionnaire design, you should know that you cannot directly ask the above questions and obtain very good answers: (a) is too ambiguous; (b) is much too broad to get useful answers; (c) is too difficult for new users; (d) is again ambiguous and the users may not have the information to answer (e). Also, since the amount of difficulty a person has with the system depends on that person's previous experience, whether they are computer science majors, whether they are highly motivated, whether they have a good friend who is helping them out a lot and whether they are very intelligent, questions have to be asked about these factors as well.
Design a questionnaire to administer to the users of the system of your choice. Take as much care as you can in the choice of your questions, taking into account the issues discussed.
3. Administer this draft questionnaire to 2 users to find out if they understand the questions in the same way you meant the questions. Give them the questionnaire to fill in and then asking them what their answers mean and why they thought your question meant. This is called pilot testing the questionnaire.
4. Use the feedback you received from your 2 trial respondents to redesign your questionnaire. If the design changes radically, it is a good idea to test out your questionnaire again on 2 other people.
5. When you think your questionnaire has been tested enough and will work on the targeted set of users, you have to find users outside of computer science who fit the eligibility requirements for your survey (as many as you can - six is a suggested minimum but note that this low number of respondents would not normally be used in a real-world study). Ask your chosen users to fill out one of your questionnaires.
6. Summarize the data collected from your questionnaires. The structured question answers are usually presented as percentages, e.g. 25 percent responded 'strongly disagree' to the question 'Should the system always have menus available?' Often the percentages are presented across demographic data, e.g. '30 percent of the women and 35 percent of the men would like to have fewer commands to learn.' A clear way to present this information is in tables.
Use the data results of your questionnaire to consider changes that might be made to the user interface to make it easier for users to learn and use the system. These can be changes in manuals and training as well as detailed changes to the interface commands and the documentation.
Now translate these ideas into how you would design your new interface to your system to solve the problems highlighted by your survey. Obviously, if the problems are in areas where there are few parallels between the studied system and your own, the information is of much less use than if it is directly applicable - be careful.
7. Write up and present the results of the survey. This should draw out the users' problems with the current interface, with the final portion discussing how your interface design will avoid these problems. You should also include a discussion of the reasons for each question or set of questions in your questionnaire, with an explanation of any changes you made between the draft and the final versions.
answer available for tutors only