Thursday, August 4, 2016

Test automation in hindsight

For quite a bit of time now, I've no longer considered myself anti-automation. But I've still been pro-exploratory testing, very heavily. I've believed, and had evidence in my project to support it, that the exploratory testing mindset is what my programmer heavy team has been missing.

I don't find same problems over an over again. I don't test in the same ways. I take each change as a unique thing. I encourage creating test automation, but leave the hands-on work on that for my programmers. There is many of them, and just one of me.

With this approach, we've succeeded well. We've learned to release daily. We work well together. Much of my knowledge of exploratory testing has rubbed on the programmers, and they test better than before.

Recently, I've made a decision to move on from my current organization and today I believe I'm only days away from completing that decision into an action. My organization has been aware of my intentions, as the two years I promised them has turned into 4,5 wonderful years with an amazing team and product.

Simultaneously, the programmer side is going through a major change. Out of my 7 programmer colleagues, 1 moved to another business area "temporarily" half a year ago. The temporary became permanent, and 3 more people will move. And as a result of this making all of us consider our choices, 1 more programmer made a personal decision to try working with another company. So my programmer colleagues are down from 7 to 2. Quite a change.

Now the low level of test automation is becoming painful. Programmer know-how is lower and mistakes when moving to new areas and introducing new people are more likely.

Seeing a deadline to my availability makes me rethink my test strategic choices in hindsight. With the choices I made on emphasizing the exploratory and distributing that skill, I did not prepare the product well for the future now at hand. The documentation I've created will be helpful for a new tester joining in. But while I'm gone, the ability to release with the level of quality we've grown accustomed to will go down.

If I made a strategic choice of encoding as much of my knowledge into test automation, would the product have better chances of future? Would that have changed the success of the past? I can't really know.

One thing I know: I want to focus in my next job more into the idea of the product doing great without me. And that is going to mean a clear focus in figuring out how do I better balance exploratory testing and creating test automation together. It's about balancing short-term and long-term benefits. It's about balancing what I love doing and what I feel needs to be done.

I'm still proud of being a good tester as in delivering good information. I'm just admitting that right now I will be better when I start codifying more of that wisdom into test automation.

I'm delighted to realize that the last month is still my opportunity to codify one of the most complicated areas in our product, printouts. My focus is now on ensuring the product has the best chances to continue being amazing without my contribution and automation plays a big role in that.

For any of you non-programming testers out there: I was one of you just a few years ago. Mob programming and strong-style pairing made me comfortable working with code even through I was saying I'm not interested. My exploratory testing skills still make me special. Putting that together with automation makes me better. Choices in the order in which you learn are arbitrary. Learning anything takes time and focus. Starting somewhere enables you to look through your choices in hindsight, and take note on things you want to change for future. 


  1. That sounds like a huge challenge that you declare here - up until now, I've only heard of test projects deteriorating, as some slight changes occur (the changes that break the test are fine because they will be addressed immediately, but what about changes that don't break it and get forgotten? I find that sometimes no documentation is better than partial documentation).
    Besides, I find it hard to relate to that goal - if 80% of the team are being replaced (or simply leaving), it is very expected that the quality (and speed) will decline. In fact, in such a case I would be more worried about coded tests being an ultimate source of truth, as there is no one who knows why and when they were written, and is in no position to spot logic errors in them.

    Would the impact of you leaving be as significant if only you and maybe one or two others were leaving? If the answer is no, I think it is an indication of a superb job you did in making the team resilient to personnel changes (note that I say "team", since the product is a static object, it does not act on it's own, but rather acted upon.)

  2. In my team in general, I have hard time seeing people relying on coded tests as ultimate truths, but they would be helpful in reminding what details we've agreed on by now when team members change.

    New people will struggle with or without tests. It's just the nature of how new people in new projects are.

    1. I thought about it, and needed to clarify a little more. I'm well aware that the speed will drop. And that is not where I think automation would be useful. It would be useful in their ability to maintain the features now in production. They will be doing minimal changes, that was already agreed with the 3 leaving.