Thursday, November 21, 2019

A Good Week for Exploratory Testing


Exploratory Testing is a 35-year-old approach to testing, created back in its days to describe a style of testing common in Silicon Valley companies that was clearly different from what we saw as mainstream.

In the 35 years of Exploratory Testing, we've grown to understand it better.

We now understand that the core of exploratory testing is the degree to which learning from the test we do now impacts our choice of what we do next.


We understood clearly that the old and traditional way of testing where people design and document tests to be later executed effectively is not what we would call exploratory testing. Exploratory testing is something else.

Where we still struggle is understanding the relationship of test automation and exploratory testing.

Test automation is a process in which we learn. Exploratory testing is a process in which we learn. When we center incremental and iterative, and learning in multiple dimensions, they are the two sides of the same coin.

This was a good week for exploratory testing, because someone many of us follow raised it up by writing about it.
Fowler describes it as:
  • a style of testing that emphasizes a rapid cycle of learning, test design, and test execution
  • avoiding separation of script creation and execution
  • avoiding predetermined expected behavior (being open to more than what we predetermine)
  • seeking to find new behaviors not covered by already defined tests
  • seeking failures defined tests don't catch
  • informal but relying on discipline to do it well
  • something that is good to consider as a task type of its own
  • requiring skill, curiosity and a learning mindset
 Sounds like how great test automation is created! But this is exploratory testing? YES!

Fowler concludes his article: 
Even the best automated testing is inherently scripted testing - and that alone is not good enough.
I argue that the best automated testing is inherently exploratory testing. The bad automated testing is inherently scripted testing. Because test automation, when it really works out, is a product of documenting what we are learning in an executable artifact.

I specialize in exploratory testing. While I read and even occasionally write automation code, it is a platform for a great exploratory testing. It extends my reach. Exploratory testing includes test automation.





In testing, we start with a baseline of knowledge and tools. I start my day with a very different baseline that someone new to my organization. Where exploratory and scripted approaches differ is in where our feedback loops are, and what learning is welcomed. Scripted approach learns about testing in the end, rather than continuously.



6 comments:

  1. Thanks for an interesting post and links,
    I wonder on what resources you base this number
    "Exploratory Testing is a 35-year-old approach"?

    ReplyDelete
    Replies
    1. Try this one from 12 years ago: "Exploratory Testing after 23 years". I had the pleasure of sitting in audience to listen to Cem deliver that keynote at CAST. http://www.kaner.com/pdfs/ETat23.pdf

      Delete
  2. The earliest reference I could find to the term "exploratory testing" is about 1993 or so. https://www.satisfice.com/exploratory-testing

    ReplyDelete
    Replies
    1. Internet is 36 years old. Exploratory testing is 35 years old. Have you considered that the earliest references may not be blog posts and online articles?

      The earliest published work that I have read is from Cem Kaner 1988 'Testing Computer Software'. It is a book I grew up to be a tester with.

      You can find timeframes online from Cem Kaner's keynote slides from 12 years ago: "Exploratory Testing after 23 years". http://www.kaner.com/pdfs/ETat23.pdf

      Many of us discovered this great way of testing thanks to prof. Kaner.

      Delete
  3. From my experience you are both right, each in your own context. Test automation is a great exploratory testing experience in its early stages- development and early stages of deployment, but soon after that it gets a life of its own and becomes strictly scripted. Developing test automation is an eye opener for the entire team about the design of the system as a whole in stages of the project where there are no customers, but once you move on and a test runs comfortably in CI it becomes a simple script

    ReplyDelete
    Replies
    1. First of all, my point that these are not separate things but one thing. For people who do test automation, concept easily includes smart thinking and learning while creating and maintaining it (exploratory testing). For people who do exploratory testing, concept easily includes using tools to get effectively answers to questions we're exploring (test automation). We are both right and we seem to even agree expect for one sentence in the end. Folks like Fowler do brilliant exploratory testing. They just call it automation. I have highest respect for this work.

      Whereas you assigning me context, you are not correct. My context is working with a product that has been around for decades, rewritten over its existence and with extensive automation for more than 10 years. Exploratory testing is not an early stages development thing, it is the approach to smart people doing smart things that makes sense for testing software in different phases of its lifecycle.

      Strictly scripted is context of software that is dead - not changing.

      Delete