In the last weeks, we've dealt with a particularly pesky mistake to make from a tester point of view, because it is a failure in how we test.
As bugs go, different ones show themselves in different ways. This particular one has limited visibility to our customers, as they can only see second order symptoms. But the cost of it has been high - blocking work of multiple other teams, diverting them from their intended use to create some good valuable items for our users, and instead making them create tooling to keep their system alive as we're oversharing data towards them.
So there was a bug. A bad bug. Not a cosmetic one. But also not one visible easily for an end user.
The bug was created by one of our most valued developers.
Since it was created by someone who we've grown to rely on, other people in the team looked at the pull request feeling confident in acceptance. After all, the developer is valued and for a reason of consistency in great work. No one saw the bug.
As we were testing the system, we made few wrong judgements:
- We relied on the unit and system level test automation, that tests the functionality from a limited perspective.
- We didn't explore around the changes because exploring from another system as user perspective requires special attention and we did not call for it.
- We relied on repeating tests as we had before, and none of the tests we did before would have paid attention to the volume of information we were sending.
- We had limited availability of team members, and we only see in hindsight that the changes were into a critical component.
- Figuring out how the pull requests could work better to identify problems or if they are more about consistency of style and structure as they've grown to be
- Figuring out how to better integrate deep exploratory testing activities towards system functionalities (over user functionalities)