Unknown unknowns from another perspective - numbers and tacit knowledge

At EuroSTAR Alexandra Casapu talked about "Fooled by the Unknown Unknowns". Thinking through her story, I still think that her approach to turning her stumblings in a project into a success story, which it truly is,  shows the same remarkable characters she has exhibited as tester growing into excellence.

When I suggested that she would use her first "own" feature as a story, I did not have yet in mind what kind of a story that would make. Thus looking at the story from another angle seemed like a fun exercise.

The numbers

I collected some data points about the effort:

My first ideas in collecting this is that the story is not about a project. It's about a testing assignment within a project. It's about an assignment that did not, even at the most focused times get the full attention of the tester, as even January turns out to be just a third of the hours in that month.

The assignment is about coping with many things at once, and taking responsibility of how much time you need as a tester. This was the first assignment, where the priority of the task was such that more time could have been taken - and was, right after the feedback. The assignment is also about importance of not delaying feedback, as the decisions on going forward with the customer were pressing at the end of January.

The data points also remind me, that this is very typical in how things happen: from decision to start to customers actually starting may take quite a while. I would not be surprised that as the budgeting cycle is now at hand, we'd only now see the issues we've missed. And that we missed, in numbers, quite a significant amount of things the customers did pay attention to, and the bugs that were put on wishlist did not come back from the customers. We differentiate between bugs and change requests, where with change requests we may claim we could not have the info available, except that my other project keeps showing that surprisingly a tester may have info available about needs that are not specified anywhere. Thus I think some of those were also escapes from us.

I'm also reminded that the external push to start questioning what you actually know may not need to be that big of an effort. In this case, it was less than a working day of discussing documentation and playing with the software. And with that, the change in numbers by the original tester was already significant.

Looking at this and thinking about what happened, I think the time it takes for things to sink in also plays a relevant factor. The later bugs found by the tester were not about new problems introduced, but about new learning introduced to the tester to pay attention to the right kinds of aspects. Every round this testing seems to be getting better.

Tacit knowledge?

The talk also left me thinking about role of tacit knowledge. A point a lot of people picked up and commented on was the tester's remoteness, and being deprived of communications with the business & developers. An example of tacit knowledge that could have helped her would be that she would have seen me test something similar. Then again, I wasn't testing the same / similar, I was on completely different areas.

Looking at what was missed, there's clearly a piece of tacit knowledge that was not put forward in the discussions - in addition to the info that was told but that did not sink in as usually happens with information you have no context for yet. No one needed to tell me that this customer, who we talked about by name, is important. It's a customer with some reputation in Finland, so knowing the importance is knowledge of the local culture. In hindsight, I did not even think that providing more context on the customer than what the product data reveals could have been useful.

It will be interesting to look more deeply into tacit knowledge in the research we're starting at work. Or rather, the subjects of which we start to become at work. I still think that a lot of the issues we were facing are basic lessons new testers need to learn, and can't learn from books or courses. Thinking is multiple dimensions, using all sources in a proper series of actions so that they enable you to learn more and don't close out options. And getting feedback on how you do in real projects is really valuable. All a junior needs is a little push to become a lot better.

Every day we learn, every day makes us a little better. I love the playful competition in us learning side by side, that keeps challenging me after all these years.  

How to treat a testing contractor

With EuroSTAR 2013 behind me, there's a story to share. I got to listen a great presentation 'Fooled by Unknown Unknowns' by Alexandra Casapu twice, as she was selected to do a do-over session as something you should have seen if you missed the first time.

Alexandra works with me (or for me), and inspired by her authentic story and great improvement in testing with our product, I have my own story to tell.

As a test manager / tester in a customer organization using services from Romania, I could go with a suspicious approach. I could require that the remote tester for visibility purposes creates test cases and runs marks them passed or failed. I could require detailed notes, detailed hour reports and detailed monitoring. I could be suspicious on how she uses her time, and focus on ensuring the time it took to prepare for this presentation was not in any way invoiced from the customer. I could give her the least interesting assignments in our testing that need to get done, and cherry pick the fun stuff for myself. I could give her ridiculously small timeframes to test large features, and then complain that she did not find all the problems. I could show her no support in thinking how to make the testing she does better, implicitly guiding her to shallowness. I could require compensation for missing the bugs she missed that I had to find in the feature she talked about.

I don't do any of that. And I wish there was more customers who buy testing services that would not do that either.

Instead, I guide her work like I'd like my work to be guided. I require thinking, and taking time to reflect and make sure we pay for those hours too. I provide feedback, by testing in same areas and by reviewing ideas of what to cover. I treat her as a colleague she is, with the idea that treating every day of testing as a learning opportunity, every day at work makes us a step better. And I strive for the same excellence in testing myself using most of my time on testing too. 

There are many customer organizations outsourcing testing without involving a skilled tester on their side. These customers may set unrealistic expectations and requirements on how the work should be done, and not have the necessary viewpoint to assessing whether the testing performed by the contractor was any good. The criteria of goodness may degenerate into numbers of passed test cases, instead of valuable testing done with a reasonable effort. With the trust missing, we end up building crazy approaches to manage the distrust.

Someone asked me on my way home from EuroSTAR if Alexandra asked permission to talk about our project. My reply was, that she did not ask - I requested her to do share a real story with real details, as that is a direction that I feel testing community needs. I'm fortunate to work for a company that shares my view on this to an extent that we're having researchers watch us work on testing, and hopefully soon enough publish results about what is it that skilled testers actually do.