Doing anything with more regularity will provide performance benefits and provide insights into how to do it better, and over the last couple years we've really ramped up the frequency of our user testing.
Sooo, here are five lessons that I want to share from our latest round of testing (for our online course publisher). They really helped me to improve our process.
1. Keep the technology barrier low
We used to have our own 'testing lab' (a MacBook Pro running Silverback hooked up to PC peripherals), but we quickly found that people got confused by the unfamiliar interface.
Where possible we now get users to test on their own machines at their own desks to remove this barrier. This has the benefit of making our volunteers feel at ease by being in familiar surroundings. In other situations like guerilla testing, we've found that a tablet works really well, due to being portable and not requiring any accessory beyond a finger.
2. Don't go out alone
It's really hard to ask the questions and capture the answers effectively without help, so bring along a colleague. This also helps when approaching people cold because you will be more relaxed and confident, and from their perspective two random strangers approaching you is less unsettling than one.
3. Groups are good
Generally you get fewer in-depth responses when testing your product with a group of people, the trade-off being the greater breadth of replies. However, group discussion of individual responses leads to additional insight that you will not get when talking with one person. Make sure you allow time your testing script to accommodate this off-piste discussion.
4. Be realistic
It's better to test a few features thoroughly, than rush through a whole raft of different aspects. Also, it's really important not to run too many sessions at one time; it’s tiring and you’ll miss insight through fatigue and response bias (you re-interpret what someone says because you've heard the same response 8 times already that morning).
We found 4 large-scale (~45 minute) sessions was enough for one day - and around 9 smaller scale (~10 minute) sessions.
5. Be enthusiastic
You're proud of what you've done, but you want to make it better. Don't be defensive of any issues, instead thank the user for finding the problem - they've done you a big favour.
Always remember that people are giving up their time for you, for no reward - so make it an enjoyable experience!