Sunday, September 20, 2020

There is Non Exploratory Testing

Celebrating my personal 25 years of growth as an exploratory tester doing exploratory testing either all or much of my work time, I regularly stop to think what makes it important to me. 

It has given my results as a tester a clear boost over the years. The better I am at that, the more magical the connections of information I can make seem to people who are not as practiced at it. I see everything as a system - people, working conditions and constraints, the software we test, the world around us. And as complex systems, I know I can design probes to change them, but I can'd design the change exactly. 

Exploratory testing is how I can get more done with less time and effort. 

On side of appreciating how it helped me grow, I look at people around me, some of which are now growing on their own paths, and some which have grown tired but do testing as it is all they know for a career. I recognize plenty of non exploratory testing around me, even if we like to say think that all (good) testing is exploratory. Not all testing is exploratory testing. You can use the very same techniques in testing in a non exploratory and exploratory fashion. Some people still box exploratory testing into that one Friday afternoon a month when they let go of their stiff harnesses and see what happens when it's just a person with the software created, on a quest to learn something new worth reporting.

Exploratory testing is testing with mutually supportive set of ideals and practices. Today, I want to talk about the four ideals that I seek to recognize testing as exploratory testing.  


At core of exploratory testing is learning. And not just learning every now and then, but really centering learning, focusing on learning, and letting learning change your plans. 

As you are testing, you come to an idea. Perhaps what you were doing now is tedious or boring, and the negative feelings you had let your subconscious roam free and you remember something completely different and connect it with your application. Perhaps what you came to realize was exactly about what you were observing - asymmetry in functionalities, or feeling that you've seen something like this before. 

When learning guides you, now you have a choice: you can park the new idea - maybe make a note of it, you can do the new idea - letting go of what was ongoing now, or you can discard it. 

The learning impacts you in the moment, in your short term plans, in your long term plans. It impacts what and how you test, but also what and how you communicate to others. The mindset of learning has you thinking about yourself, your abilities and your reactions, creating tests to yourself. Experimenting new ways that don't come natural, seeing if what you believe of yourself is true, and becoming a backbone on a journey to be a better person. 

The tool you're sharpening through learning is you and other people around you. 

When you see testing efforts with very little learning happening, test cases being repeated, and focus on reapplying recipes with a low quality retrospects and lack of introspection, you are likely not doing exploratory testing. 

When you're being trained to become an exploratory tester, you may see it as you're not given the answers. You're taught to figure out the answers. You're not given a test case with expected values, you're told there is a feature in your software that you need to figure out if it works. You're expected to turn that work that appears 5 minute work to 5 hour work and 5 days work by understanding how it it connected to information and value in software creation. All work of testing has surprising depth to it, and it is easy to miss. 


Another significant ideal in exploratory testing is agency. A word that does not even translate to my own language has become a core of the way I think of exploratory testing. In sociology, agency is defined as the capacity of individuals to act independently and to make their own free choices. Free choice is not exactly what we give to testers in organizations as per common experience. Testers and testing is often very constrained activity, like a project manager of my past once told me: "No one in their right mind could enjoy testing. Marking test cases done daily is the only way to force people to do it.". 

Exploratory testing isn't fully free choices, but it's making the testing box of choices free and adding agency to other choices testers can do as full members of their work communities. 

Agency is constrained by structure, such as social class (power assigned to testers), ability (skill of exploratory testing, skill of programing, skill of business domain understanding...) and customs (tester's don't make decisions, testers don't program). Exploratory testing is a continuous act of changing those structures to enable best possible testing. 

And obviously, best possible testing is a journey we are on, not a destination. The world around us changes and we change with it. 

Not everyone in current organizations has agency. Testers are considered low social class somewhere. Testers are not given much choices in how testing is done, process determines that. Testing without testers does not include much choices, process determines that. Our organizations become constrained to our test cases, barely adding per recipe and keeping added tests afloat. That is not exploratory testing. 

For me agency explains what draws me to exploratory testing. Having agency, and allowing others around me agency is my core value. I frame it as being a rebel, finding my own path, and ensuring that while serving the world I am in, I'm a free agent, not an item. 

Opportunity Cost

Either learning or agency don't yet capture the constraint in the world of exploratory testing. We are all constrained by the time we have available. Opportunity cost is the ideal that we respect our limited time, and use it for best possible outcome. We need to make choices between what we do, and whatever we choose to do, leaves out something else we could not do on that time. Being aware of those choices and making those choices in a way where we also see the cost of things we don't do is central to exploratory testing. 

We can spend time hands on with the application or we can spend time creating a software system that does testing of our application. Both are important, valuable activities. We intertwine them under whatever constraints we have in the moment optimizing for the cost towards value. 

Not all organizations allow testing to consider opportunity cost. The experiment is set somewhere else. The skills constrain possible choices, and changing skills isn't considered a real possibility. 

Systems Thinking

Finally, the fourth ideal around exploratory testing is systems thinking. Software my team creates runs with software someone else created, and our users don't care which of us causes the problem they see. A lot of times teams have developers feeling responsible for their own code, but testers are responsible for the system. Software is part of something users are trying to achieve, that too is part of the system. There's other stakeholders. Software does not exist in a bubble. Exploratory testing starts with rejecting the bubble. 

Not all organizations break the bubble for all their teams. They create multiple bubbles and hierarchies. 


I've seen too much of testing that is test-case based, recipe based, founded on bad retrospective capabilities and limited to a scope smaller than it needs, due to more reasons that I can list in this moment. The change, as I see it, starts with agency used for learning.