Monday, October 1, 2018

Deep Exploratory Testing

There's a famous saying by Linus Torvalds:
Given enough eyeballs, all bugs are shallow. 
Crowdsourcing references often like to quote this, pointing out that out of the bugs we could find in testing, the users in production end up finding over masses all the relevant ones, even if they did not report. A crowd could do well in hitting a bunch of bugs.

For the purposes of me doing and guiding exploratory testing, I find it really beneficial to think in terms of shallow vs. deep testing. Shallow can be done with less skills, and with less time. Deep testing requires more skills, more insights, a foundation of learning that is built in layers and requires time.

Many people find that agile somehow guides them to only doing shallow testing. They feel their testing is always squeezed to the end of the sprints, and that it is so that development schedule is flexible, while testing schedule is fixed. However, they may fail to see the opportunity of testing continuing after the release, focusing on going deeper.

Shallow testing find shallow bugs. Shallow bugs are easy to find, they are obvious and would become a problem in production immediately. Deep testing finds deep bugs. It may lead us shallow bugs that just take a bit more of an effort to see, combinations and conditions that take time to set up. But it also may lead us to bugs some don't consider bugs: things that threaten the value of the product, things that should be different to be better.

Going deep happens in layers. You don't repeat the same, but you go further, deeper. You start before implementing. You continue while implementing. You don't stop for releasing. You don't have to, because you are not on a project. Agile made it a continuous process where there is no end.

Sum it up, and it totals to deep testing. Miss the skills, and all you get is shallow. The additive way of doing testing is not regression testing. It is finding new perspectives and exploratory testing is the core practice in doing that. 


  1. This is an excellent, concise piece of writing on a subject that has not been explored nearly enough. I too have seen the challenges of agile testing with testers that spend the bulk of their time maintaining & debugging flaky e2e automation or trying to close out tickets in the acceptance queue.These testers often don't get the chance to do any deep testing and their exploratory skills atrophy. I'm currently working to make deep exploratory testing a part of every story in my org however there's a whole training & education aspect for both the testers, Devs & management.

  2. I would have hoped that Torvalds was too intelligent to make such a profoundly ignorant statement, but maybe he knows nothing about testing.

    There is no reason to believe that crowdtesting will find deep, complex bugs. The remuneration model discourages such testing and the vast majority of people who do crowdtesting have no significant skill. Testers in general have poor exploratory testing skills and people who do crowdtesting tend to be the ones who are so unskilled they can't get a proper job. People outside of software development are typically hopeless at noticing and reporting even the most obvious bugs, so user feedback is usually almost useless.

    From what I have seen, testing on agile projects is usually done very poorly - weak automated checks augmented by trivial manual tests based on vague stories. It's no wonder that software quality is getting worse, as reported in a recent report from Tricentis.

    Sadly, testing in agile projects is usually controlled by development managers or scrum masters, and almost none of them have any idea what good testing looks like. To make things worse, testers seem to go along with what they're told to do instead of showing how testing can be done much better.