As I started working on identifying an experience report I would deliver for #DEWT5 in January on Test Strategy, I hit a wall of confusion that I have not quite overcome yet. What is test strategy anyway - for me, in practice? And which experience related to it would I want to share?
At first I was looking at my two more recent products I work with and I made a few observations:
Test Strategy is the ideas that guide test design.
At first I was looking at my two more recent products I work with and I made a few observations:
- On one product, I owned strategy and another tester owned testing. On the other, I owned both strategy and testing. I'm more sloppy on communicating my ideas on the latter.
- On one product, there is never enough testing but it's completely deadline-driven to do best work with what is available. On the other product, schedule is flexible to include the right testing before releasing.
- For both products, there's nothing about releasing that would say that testing stops there. With continuous flow of releases, we react to feedback outside the time for the feature.
- There is no test strategy document. There is nothing I could clearly call test strategy. Even the ideas of how to test are more of generic guidelines than a project-specific strategy.
Looking at the two products, I came to realise that the way we work on both of these products is continuous flow of changes into an overall vision of a product and having a strategy other than generic ideas of "provide relevant information" and "do the most important thing next" seem off. I would not call the checklists we create to model the product as strategy - they help think about scope of testing though. I would not call the Jira tasks that outline testing time boxes strategy, but they were a way of discussing strategic choices of where to use time, what to do in a shallow and what in a deep manner. But as skills grew, we have up even those tasks and just work on the development tasks without plans of any kinds - a day at a time.
In relation to the changes going into the build to be released, I can well outline the work we have done and what we should do. I notice primarily choosing what we'll test and how by the set of skills available - I have developers test what they will manage (with a stretch), I'll have product managers test what they will have patience for and I'll test the stuff that is hard to give to people who are less skilled in the types of testing I do.
It seems to me that my test strategy is just the next idea of an experiment to do to provide information by testing. I try to choose the one that I believe will at least be of value, with the principle that time will anyway run out.
Looking back at a test plan I wrote at my previous place of work though, I've clearly seen strategic thinking as in identifying what high-level areas and ideas to guide those areas as very important. But there someone else owned the strategy and testing, and I would just suffer from the results of poor strategic thinking that would drive focus on a too narrow set of things.
So this left me wondering: if test strategy is the next idea to guide testing and builds an idea at a time, the goodness of the next ideas relies on who is thinking up the ideas. Introducing more versatile ideas as strategy without implementing the ideas as testing could be a good approach then. In particular, transforming people who have one idea and then are out of ideas of what aspects to test for. But what am I missing if I just don't build anything more of strategy-like as I do now in my projects?
Could test strategy be the ideas that have guided test design, built one idea at a time - the next? Playing with the temporal dimension of strategy seems at least an intriguing idea.