We like our University wiki very much, so we were very sad to have to remove Single Sign-On (SSO) just a few days after adding it.
As we roll out services which ask a person to log-in we try to make sure that they use SSO and we’re also slowly introducing it to applications which are already in use, like the wiki. As it turns out, thorough testing to make sure that the application works as you expect is really hard. In our case, if you had already used SSO, but then followed a link to the wiki or opened a bookmarked wiki page, it said that you hadn’t logged in! It was very frustrating, and required you to try and open the page a second time before it would work.
This example didn’t crop up in our testing (although you can guarantee it will next time we try), but it turned out to be a common problem where, for example, lecturers had linked to wiki pages from our VLE, Moodle, but the students suddenly couldn’t access them. This meant that the problem hit the students, who told their lecturer, who told the VLE support staff, who then told us. So just by missing a simple use-case, during what we considered an upgrade, we managed to affect many different groups of people and were forced to investigate, experiment and ultimately reverse the change.
Software testing is admittedly hard, but what this shows is that during each change which affects how people interact with your service, it makes sense to map out the common use cases for those people and try them out yourself, both before and after the change because your usage may not reflect the experience of a great number of your users.
