"Who am I?": The importance of ecological validity

Usability testing is at its most effective when the people who participate in the user testing are ACTUAL users! This seems so simple that it's silly to talk about it. However, it's surprising how often this principle does not translate into real-life user testing.

For a research study to possess ecological validity, the methods, materials and setting of the study must approximate the real-life situation that is under investigation.[1]
At the far end of the spectrum, we know that asking employees to "pretend" to be customers while they complete the usability test is an ineffective (and risky) substitute for the real user. This type of quick-and-dirty usability testing can provide false results, and can involve a significant amount of time and money to be spent on a concept or an interaction design that simply doesn't work for the real user.

A less-noticeable type of user substitution happens more often, when one participant is asked to play multiple roles. This is common when there are multiple user groups and there is not enough time and money to test individuals from every single user group. So, one participant is asked to complete 2 or 3 tasks that are relevant to them personally, and then they are given tasks for other user groups. Here are some examples: "Suppose you are a manager. What would you do now?" or "You are a truck driver, and you are looking for a map. Please show me where you would find that." In the latter instance, one participant said, "Well, I was looking over here, but then I remembered that I am a truck driver. I had forgotten." Hearing these comments is evidence you are asking the user to role-play, which will leave you with unreliable data regarding user behaviour and user needs with respect to your absent user group.

In a recent round of user testing, I was able to use actual material that the user group sees on a daily basis. The research was for a stock photography site, and the user groups were designers and researchers who spend their time on stock photograph sites, looking for specific photos. These individuals receive “creative briefs” from clients which are basic descriptions of the type of photo they want, and other supplementary information (such as other photos that approximate what they want, or a tagline if the assignment is an ad). A frequent user task involves sorting through various photos from stock sites to find ones they think fit the requirements. When testing with this user group, I was able to use these “creative briefs” in the user tasks – it was a task that this group of users performs on a daily basis, so their search behaviour and their interaction with the test system is almost identical to their real life scenarios ensuring results with a high-degree of validity.

This high degree of ecological validity is mostly applicable to traditional usability testing. When conducting walkthroughs with a prototype or testing conceptual ideas, there are often no “real-life” tasks since the product does not exist yet. But the more the proposed tasks resemble “real-life tasks” for “real-life users”, the more reliable the results that fuel the creation of a more usable and useful product

[1] Brewer, M. (2000). Research Design and Issues of Validity. In Reis, H. and Judd, C. (eds) Handbook of Research Methods in Social and Personality Psychology. Cambridge: Cambridge University

Why Context is Important

I’ve read many articles on contextual advertising and behavioural marketing. It seems that the marketing folks are catching on to what we have known for years – understanding people’s behaviours and the context in which they use a service or product is much more predictive of outcomes than knowing their demographics or psychographic profile.

In the context of the aforementioned article, marketers are finding that they are much more successful with their online advertising for mini-vans when they target web users who have used the keyword in their online search. This tells the marketer that they are interested in minivans and that they are likely looking for a new vehicle now. It turns out that this is more important to know than whether the user is a male, age 25-34 with 2 kids and a house in the suburbs. While those demographic characteristics may describe the population of mini-van owners, they don’t accurately predict who will purchase mini-vans in the current time period. What behavioural marketing is doing is not only displaying ads in context (of someone looking for the product) but altering the content based on individual consumer behaviour.

Similarly, with web design, it is more important to know what users are doing on the site (frequent tasks) and why they have come to the site for that particular visit than what their demographic characteristics are. Organizations spend a lot of money on research to understand who their users are. But without knowing why those users come to the site and what they are trying to accomplish on the site at that particular moment a marketer may end up annoying a new customer with advertising for a product that they recently purchased or missing the opportunity to market to a customer who is ready to purchase.

In the same fashion, government services websites might need to know whether users are coming to the site to find information or complete a transaction (request for service) or both. If both, how will they complete their tasks – will they start to complete a form and then go search for more information on a topic before returning to complete the form? What questions might they need answered before they feel comfortable submitting a form? These are the kinds of questions and issues that arise when considering user behaviours and the context in which different behaviors occur. It is this type of contextual information that will help the site designers determine if an automatic save should be implemented to allow users to leave a partially completed form and return to it later or if a direct link to FAQs should be available from the electronic form page.

While demographics and user opinions are useful for describing the characteristics of the user population, they won’t provide effective design guidance as will observing user behaviours and the context for those behaviours.