Friday, November 18, 2011

Falling into a trap - talking of Exploratory Testing

I attended the public defense of dissertation for Juha Itkonen today. His doctoral dissertation with the title "Empirical studies on exploratory software testing" is a topic particularly close to my heart, since it was the topic I started with but failed to continue striving for in the academic sense. Juha started in the same research group after me, took the relevant topic and made all the effort to research and publish what he could in the limited timeframe (of nearly 10 years).

When Juha's opponent started, he went first for the definition of exploratory testing and distinguising exploratory testing and non-exploratory testing. This, I find, is a trap, and an easy one to fall into.

Juha explained some of the basic aspects of what he has summarized of exploratory testing, but struggled to make the difference of exploratory vs. non-exploratory. He pointed out that learning is essential in exploratory testing, and changing direction based on learning. And the the opponent pointed out that typically people who test based on test cases also learn and add new tests as needed based on what they learned. Juha went on explaining that if you automate a script and remove the human aspect, that is clearly not exploratory. And the opponent pointed out that a human could look at the results and learn.

What Juha did not address in his defense, or in the papers I browsed through, is the non-linear order of the tasks. He still works on the older definition of simultaneous activities, and thus, in my opinion, misses a point of tester being in control what happens first, after, again and for how long each of the activities endure to achieve a goal.

A relevant challenge (to me) is that all testing is exploratory. There's always a degree of freedom, tester's control and learning when you do testing. Trying to make an experienced tester follow a script, even if such existed, is not something that happens - they add, remove, repeat and do whatever needs to be done, including hiding the fact that they explore. All real-world approaches are exploratory testing to a degree.

I still have a problem to solve: if all testing is exploratory, what words should I use to describe the wasteful, document-oriented approaches, that take a lot of effort for creating little value. There's a lot of real-world examples where we plan too much, what we plan is the wrong things as we do the planning at time we know the least. And what's even worse, due to be belief in planning, we reserve too little time for the hands-on testing work and related learning, and with the schedule pressure, fail to learn even when the opportunity supposingly is there.

Low quality exploratory testing is still exploratory testing. Getting to the value comes from skill. And with skill, you are likely to get better results with some planning and enough exploration-while-testing, than skipping the planning. And yet with skill, you could use too much time on planning and preparing, due to logistics of how the software arrives to you.

Ending with the idea from the opponent for today's dissertation: the great explorers (Amundsen - south pole / example coming from a norwegian professor) planned a lot. What made the difference between those that were successful and those that were not, was not whether they planned or not, but that they planned holistically, thinking of all kinds of aspects, and keeping their minds open for learning while at it - making better choices for the situation at hand and not sticking to the plan like carrying back stone samples at cost of one's life.


  1. Hi,

    Nice summary, thanks. Looking forward to see what new material Juha Itkonen will produce.

    I think the difference between exploratory and non-exploratory testing is that the latter one is "checking" and not "testing". In the common case of hybrid-testing, we could use either that as a term or make up a new one. However, I would advocate against using the word "testing" in the context of purely scripted work.

    Exploratory testing can show it's value for example when there is a severe time-pressure, because you can execute the test cases without documenting too much. I completely agree on the idea that initial planning is basically always needed, but it takes experience to guesstimate how much pre-planning should happen. This is a skill I hope to improve continuously.

    Regarding the last paragraph, I like this quote from Eisenhower (taken from Wikipedia):

    "I tell this story to illustrate the truth of the statement I heard long ago in the Army: Plans are worthless, but planning is everything. There is a very great distinction because when you are planning for an emergency you must start with this one thing: the very definition of "emergency" is that it is unexpected, therefore it is not going to happen the way you are planning."

  2. I'm really pleased you got to Juha's dissertation defence - I had no idea they were public!

    Re: planning - I find that the act of planning (sometimes observing the people around me making plans) teaches me about my (and their) values and so lets me reveal (and tune) the underlying strategy. That then allows me to improvise / plan-on-the-fly / respond gracefully to change / adjust my actions congruently with those around me.

    Re: carrying back stone samples at cost of one's life. One never knows quite how a risk will pan out - otherwise it wouldn't be a risk, but a certainty. I have sympathy for those who take heroic risks for extraordinary chances, and know that in the field one's understanding of current risk is tainted by one's effort and the closeness of the goal. I'm tempted to make a fatuous comment about dead men not typically getting their rocks back to the university - at least not at the first go.

    I'm reading through Juha's dissertation. You make a fair point about exploration and testing. However, I wonder about "likely" in "And with skill, you are likely to get better results with some planning and enough exploration-while-testing, than skipping the planning." It seems to me you might mean "reasonable" - which opens up the question about a raft of "reasonable" and "questionable" behaviours - and where the judgement to separate them comes from. I'm keen to see the patterns, tools and skills that Juha's research subjects demonstrated.