At EuroSTAR Alexandra Casapu talked about "Fooled by the Unknown Unknowns". Thinking through her story, I still think that her approach to turning her stumblings in a project into a success story, which it truly is, shows the same remarkable characters she has exhibited as tester growing into excellence.
When I suggested that she would use her first "own" feature as a story, I did not have yet in mind what kind of a story that would make. Thus looking at the story from another angle seemed like a fun exercise.
The numbers
I collected some data points about the effort:
My first ideas in collecting this is that the story is not about a project. It's about a testing assignment within a project. It's about an assignment that did not, even at the most focused times get the full attention of the tester, as even January turns out to be just a third of the hours in that month.
The assignment is about coping with many things at once, and taking responsibility of how much time you need as a tester. This was the first assignment, where the priority of the task was such that more time could have been taken - and was, right after the feedback. The assignment is also about importance of not delaying feedback, as the decisions on going forward with the customer were pressing at the end of January.
The data points also remind me, that this is very typical in how things happen: from decision to start to customers actually starting may take quite a while. I would not be surprised that as the budgeting cycle is now at hand, we'd only now see the issues we've missed. And that we missed, in numbers, quite a significant amount of things the customers did pay attention to, and the bugs that were put on wishlist did not come back from the customers. We differentiate between bugs and change requests, where with change requests we may claim we could not have the info available, except that my other project keeps showing that surprisingly a tester may have info available about needs that are not specified anywhere. Thus I think some of those were also escapes from us.
I'm also reminded that the external push to start questioning what you actually know may not need to be that big of an effort. In this case, it was less than a working day of discussing documentation and playing with the software. And with that, the change in numbers by the original tester was already significant.
Looking at this and thinking about what happened, I think the time it takes for things to sink in also plays a relevant factor. The later bugs found by the tester were not about new problems introduced, but about new learning introduced to the tester to pay attention to the right kinds of aspects. Every round this testing seems to be getting better.
Tacit knowledge?
The talk also left me thinking about role of tacit knowledge. A point a lot of people picked up and commented on was the tester's remoteness, and being deprived of communications with the business & developers. An example of tacit knowledge that could have helped her would be that she would have seen me test something similar. Then again, I wasn't testing the same / similar, I was on completely different areas.
Looking at what was missed, there's clearly a piece of tacit knowledge that was not put forward in the discussions - in addition to the info that was told but that did not sink in as usually happens with information you have no context for yet. No one needed to tell me that this customer, who we talked about by name, is important. It's a customer with some reputation in Finland, so knowing the importance is knowledge of the local culture. In hindsight, I did not even think that providing more context on the customer than what the product data reveals could have been useful.
It will be interesting to look more deeply into tacit knowledge in the research we're starting at work. Or rather, the subjects of which we start to become at work. I still think that a lot of the issues we were facing are basic lessons new testers need to learn, and can't learn from books or courses. Thinking is multiple dimensions, using all sources in a proper series of actions so that they enable you to learn more and don't close out options. And getting feedback on how you do in real projects is really valuable. All a junior needs is a little push to become a lot better.
Every day we learn, every day makes us a little better. I love the playful competition in us learning side by side, that keeps challenging me after all these years.
