This week I spent writing my design document ready for Jonny to compile into a group design document. Below is my design document, then a draft of the testing methods



Testing methods

Testing brief

Sample – 52 testers; 26 control group testers, 26 ‘achievement’ testers
–University of Bolton: project room
Min. Play-time for testers – 15 minutes
Max. Play-time for testers – 1 hour
Results collection – Questionnaire taken before testing,
Observation and note taking conducted by our team,
Interview conducted after testing.

Test plan

For our testing, we will need to find a sample of 52 people. These 52 people will be split up into two groups; 25 people will play through the base game with no achievements, and the other 26 people will play through the whole game, with achievements.
All the variables for the testing will be the same; the testing will be conducted within the same environment per tester, minimum play time 15 minutes, maximum play time 1 hour, testing will be done on PCs. The only different variable will be either the base game or the game with achievements.

We are testing to see if the achievements influence the players’ in-game behaviour, so one main thing for us is that the testers do not know the true intent of our testing, and we want them to play our game naturally and in the same manor the testers would normally play games with.
We will conduct the testing so that the players/sample believe that the reason for them testing our game is for usability testing; we will have to have no leading questions, and have hidden questions which relate to achievements, without putting focus on the achievements themselves. We will be deceiving the sample slightly; the reason for this is to gain unbiased results from the sample, to observe their interactions with the achievements.

Gathering results

For our testing, we will be gathering results in three different ways; pre-testing questionnaire, observation note-taking, and a one-on-one interview with each tester.

The pre-test questionnaire will take basic personal information from the sample (age, gender); in addition to this we will be taking information such as how often they play games (rating scale), and including what genre of games they play most often (multiple choice).
The structure of the questionnaire will be based on the survey question structure within the book “Essential ethnographic methods…” Questions should be in a question or statement format, open-ended questions should be avoided; the sample must be able to answer the questions, avoid negative or double negative questions, and keep questions and responses as short as possible. The questions involving how much the sample like something will have a rating scale, where they list from 1 to 10 (1 being a negative response, and 10 being a positive response). “This article describes the process for developing and testing questionnaires and posits five sequential steps involved in developing and testing a questionnaire: research background, questionnaire conceptualization, format and data analysis, and establishing validity and reliability.” (R.B. Radhakrishna, 2007).

For the observations, we will create a sheet listing specific behaviours which we are looking out for; e.g. if a player stays for the testing for longer than 20 minutes, and they are obviously looking to collect achievements and going out of their way to collect achievements. With the different samples, the same observation sheets will be used; this is so we can compare the observations and see if the same behaviour patterns are appearing throughout both samples.

Once each tester has played through one version of the game, and they have played it for a minimum of 20 minutes, the testers will be asked to conduct a one-on-one meeting with one of our team members. Each tester will be asked set questions which we will all decide on before the testing, however once all the set questions have been asked, we will ask the tester if they have any comments about anything that happened within the game, and these will be noted by the person conducting the interview. This is so we can collect some focused quantitative data, but it also gives us an opportunity to collect some qualitative data, which may serve as a contrast when we come to compare the control tester group against the achievement tester group.
There will be no leading questions, and the testers will not be asked anything specifically to do with achievements; as far as the testers will know is that they are all usability testing.


– Stephen L. Schensul, et al. (1999). “Essential ethnographic methods: observations, interviews, and questionnaires.” CA: AltaMira Press. 318.

–  Crawford, I. M. (1990) Marketing Research Centre for Agricultural Marketing Training in Eastern and Southern Africa. Harare Zimbabwe.

– Sudman, S. and Bradburn, N. M. (1973), Asking Questions, pp. 208 – 28.

– AGS. (1997). Marketing and Agribusiness Texts. Available: Last accessed 29/08/10.

-R.B. Radhakrishna. (2007). Tips for Developing and Testing Questionnaires/Instruments. Tools of the Trade. 45 (1).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: