Sunday, October 11, 2009

Analyzing own learnings for value of tests

I was reading Markus Hjort's blog ( about challenges with testers and trust - or lack of thereof. In his story, he looks at the testers wanting to test leap days from the outside and ponders about how to make people understand what kind of problems are likely and worth the testing effort.

I started my testing career a number of years ago in localization testing. While finding the problems with that type of testing just as challenging, there is the part of "working original software" to compare against giving it its own twist of flavor.

Not that many years ago, years after my first introduction to software testing and learnings from localization testing, I started to look at the value and logic of my own testing. I realized my early learnings from localization testing, namely that there's a difference between localized operating systems and I should treat e.g. Finnish and German OS as separate for my testing, had a deep impact on what I was still doing.

I went back to thinking how did I learn that there was a difference. I still can remember the feelings of inadequacy, surprise and shock when I got the feedback of missing bugs early in my career. It was enough to teach me not to fail that way again. As the saying goes, mistake once is ok, but twice, the same, that's just stupid.

As I was looking at how I test and how would I rationalize it to myself, I also asked myself whether there was value in the type of tests I did. I went back to my notes, to realize that there had not been any indication whatsoever of this OS-language difference biting the software I had been testing, for quite a while.

I had to admit to myself, that I had made an unconscious choice in my priorities. I had decided that this one was worth the time and effort, even though it was not, as there had been a subtle-to-me change in technology related to OS-language versions that made the risk less relevant than what I thought on my experiences in the past. I had used time on repeating something, and thus not using the same amount of time on something else.

I started this reflection from the idea that there's perhaps lack of trust for testers due to lack of quality experienced in the past. How to tackle that best in agile? I'd say, let the testers test, but ask them to gradually let go. First, ask them not to run a test they are used to running every week for a week or two, let them go back to it, and reflect. Did it break? If not, why would you think it does? It may bring to surface similar experiences than the one I described. And for me it has.

Having the test automated is the second option for me, if there's value. The value-question needs to be addressed first.