Educating students on how to conduct usability tests is part of the MA programme I currently study (click here to enlarge the module map shown above). The product I decided to test was a developer portal, that is, a website aimed at developers and offering further information about microservices and their APIs. When shaping the test, I learned how important it is to have an excellent understanding of the product itself and the target group(s).
As it would not be possible to test everything the developer portal offers, I came up with a comprehensive, yet concise task set for my usability test. The task set reflected the core things a developer would need to do at some point of using the developer portal, like subscribing to a microservice and testing its API for a first “Hello World”. Concurrently, the task set respected the time the participants would voluntarily dedicate to my usability test. So, one test session was designed to take about one hour. The literature reassured me that this was a reasonable time.
However, I tended to be in doubt whether the whole setup was close enough to what would happen in reality or not. This is even more important as it is crucial to have a well-designed test if you want to make use of the test’s results.
Recently, I came across a book that reinforced my doubts. The book, published in 2011 (click this DOI link for more information), is a collection of approaches that tackle on improving the usability of complex information systems. Editor Michael Albers reflects on “traditional” usability testing and how it focuses maybe too much on quantitative aspects like how many buttons were hit in which time.
Many traditional usability tests focus on function (essentially the button pushing) and not on the process of how the information is used.
Michael Albers
He admits that button design and the whole look & feel of a system are nonetheless vital as they can contribute to user-satisfaction and perceived user-friendliness: studies have shown that both contribute to better decisions by users in a complicated information situation. However, he made a case for more sophisticated approaches to usability and usability testing for complex information systems.
Find a summary of Chapter One of the book below (click here to enlarge the image). It has the same title as the book, Usability of Complex Information Systems: Evaluation of User Interaction, and it states that:
- People have mental models of the world.
- This can result in a “tunnel vision” which may lead to wrong decisions.
- Therefore, and before making any bunches of data available, complex systems should start by defining the current situation.
- In doing so, people’s “contextual awareness” would be raised so that they would know how the information presented in a particular situation relates to each other.
- This would be the starting point for people in making well-informed decisions and less errors.
API docs and me 
