I like numbers

This feels like a dirty secret, but it becomes very clear if you ever work with me. I like numbers. 

I count things, some more relevant than others. I usually know how many times someone called me a 'guy' on a particular day, because it's my calming mechanism of having to deal with that annoyance. I know I delivered 23 external conference sessions in 2020, I know my team is about to do release number 24 before the year ends, I know we've processed a little over 300 Jira tickets (which I consider a few full weeks of wasted time) and I know how many items of value I would say they actually include since I reverse engineered value out of the work we did this year. 

The numbers are information. They are only one type of information. But they are information easily used to impress people. And they are a tool for getting my message across stronger than most people. 

For past few jobs, I have done end of year numbers. Since anything I work on right now is kind of secret, I'll just go traveling back in time to show what type of numbers I collected in the two previous years, to run an annual comparison.

• Team size. How many people delivering the work I was analyzing. 
• Monthly Active Devices (MAD). The first number I was curious on is how many customers the team was serving with the same people. Being a DevOps team meant that the team did development and deployment of new changes, but also provided support for a ever growing customer base we calculated in impressively large numbers. Telemetry was invaluable source for this information. It was not a number of money coming in. It was people using the product successfully. 
• Work done represented in Jira tickets. I was trying hard to use Jira only as an inbox of work coming from elsewhere outside the immediate team, and for most part I succeeded with that and messed up all my changes of showing all work in Jira ticket numbers (I consider this a success!). About a third of visible ticket work done was maintenance, responding to real customer issues and queries. Two thirds were internally sourced. 
• Work coordinated represented in Jira tickets. Other teams were much stricter in not accepting work from hallway conversations, and we often found ourselves in a role of caring for the work that others in the overall ecosystem should do. Funny enough, numbers showed that for every 2 tickets we had worked on ourselves, we had created 3 for other teams. The number showed our growing role in ensuring other teams understood what hopes were directed towards them. It was also fascinating to realize that 70% of the work we had identified for others was done within the same year, indicating that it wasn't just empty passing of ideas but a major driving force effort. 
• Code changes. With the idea that for a DevOps team nothing changes if the code (including configurations) changes, I looked around for numbers of code going into the product. I counted how many people contributed to the codebases and noted it was growing, and I counted how many separate codebases there were and that that too was growing. I counted number of changes to product, and saw it double year over year. I noted that for 4 changes to the product, we had 3 changes to system level test automation. I noted code sharing had increased. Year over year numbers were delight: from 16% to 41% (people committing to over N components) and from 22% to 43% (more than M people committing on them) on the two perspectives of sharing I sampled. I checked my team was quarter of the people working on the product line, and yet we had contributed 44% of changes. I compared changes to Jira tickets to learn that for each Jira ticket, we had 6 changes in. Better use the time on changing code than managing Jira, I would say. 
• Releases. I counted releases, and combinations included in releases. If I wanted to show a smaller number, I just counted how many times we completed the process: 9 - number that is published with the NEXTA article we wrote on our test automation experience. 
• Features pending on next team. I counted that while we had 16 of them a year before, we had none with the new process of taking full benefit of all code being changeable - including that owned by other teams. Writing code over writing tickets for anything of priority to our customer segment. 
• Features delivered. I reverse engineered out the features from the ticket and change numbers, and got to yet another (smaller) number. 
• Daily tests run. I counted how many tests we had now running on a daily basis. Again information that is published - 200 000. 

So you see, numbers are everywhere. They give you glimpses to what might be happening, but at the same time they are totally unreliable. If you have a critical mind and good understanding on their downsides, looking at them may be good. 

Going back in time even more, I find my favorite numbers: how I ended up having to log less and less bugs as the team's tester. From impressive numbers showing I found 8,5 bugs for every day of the year to having almost none logged as I moved to fix-and-forget and pairing with developers on fixes give a nice fuzzy feeling that turning my work invisible was a real improvement. 

Ask your question, and numbers may help. Numbers first - or comparable numbers between different teams - usually cause havoc. 

So like numbers, like me. But be careful out there. Even if they say you get what you measure, it is only true sometimes. We can talk about that another time. 

The box with Christmas Ornaments

There is a fascinating way of coming to the idea that the problem is almost always testing. Here's a little story of something that has happened to me many times in many organizations, and was recently inspired to think about. Maybe it is because it is almost Christmas. :)

Speaking in metaphors, the box with Christmas Ornaments inside. 

Once upon a time, there was a product owner who ordered a Box with Christmas Ornaments. As product owners go, they diligently logged into Jira their Epic describing acceptance criteria clearly outlining what the Box with Christmas Ornaments would look like delivered. 

The Developers and the Testers got busy with their respective work. Testers carefully reviewed the acceptance criteria that was co-created, and outlined their details of how testing would happen. Developers outlined the work they need to do, split the work to pieces, and brilliantly communicated to testers which pieces were made available at each time. Testers cared and pinged on progress, but when things aren't complete, they are not complete. 

The test environment for the delivery was a large table. As pieces were ready from the Developers, their CI system delivered an updated version into the middle of a table. The Box with Ornaments was first a pile of cardboard, and everyone could see it was not there yet. But as work progressed, the cardboard turned into a Box, without the Ornaments. As per status, pieces were delivered (and tested), but clear parts of the overall delivery were still undone.

Asking the status and wanting to be positive, Developers would report on each piece completed, and the Box on the table looked like it was there. It was there quite some time. Asking status from testers on testing, they would learn that testing was incomplete, and it was so easy to forget that there are scenarios that required both the Box and the Ornaments to make sense of the final item, even if we could and had tested to learn about each individually. 

The product owner, equipped with their Epic in Jira looking towards the table concluded: 

Things get stuck in the process. They are long in an intermediate stage. It feels like they don't care about delivering me my package, they just leave it lying around for testing. 

It's not like they ordered the Box without Ornaments. Yet they feel it looks ready enough that putting the Ornaments in is extra wait time. 

To achieve flow of ready to the hands of whoever is expecting, optimizing developer time between multiple deliveries really does the negative trick. Yet we still, in so many cases, consider this to be a problem with testing.

I know how to fix it. Let's deliver it as soon as developer says so. No more Testers in the place you imagine them - between implementation and you having that feature at your hands. 

A better fix is to deliver the empty box all the way to the customer as it is ready, and carefully think if the thing they really wanted was the Ornaments, and if another order of delivery would have made more sense. 

RED green refactor and system test automation

 In companies, I mostly see two patterns with regards to red in test automation radiators:

1. Fear of Red. We do whatever we can, including being afraid of change to avoid red. Red does not belong here. Red means failure. Red means I did not test before my changes can be seen by others. 
2. Ignorance of Red. We analyze red, and let it hang around a little too long, without sufficient care on the idea that one known red hides an unknown red. 

I don't really want either of these. I would like to work in a way where we bring things back to a known (green / blue - pick your color) baseline, but seeing red is invitation to go dig deeper, to explore, to understand why. 

Red being associated with failure is a problem. With Red-Green-Refactor cycles, we want to start with red. Red is an essential part of the process. Trying to make progress with occasional fails is better than not making progress to avoid all fails. 

The red allergy is allergy to failing. And allergy to failing is an indication that we don't have a mindset of appreciating learning. 

Instead of fearing red, we should fear green/blue. Red calls in for action, green/blue accepts a status quo. W can only trust green/blue if we have, in the process of creating the tests, seen red. Make it fail, make it pass, clean it up. 

I think back to the projects I've watched with system test automation, and red is not an infrequent visitor. Red invites work. And with making decisions, we still wish we'd have less red, and faster analysis of the red. 

From appreciating the red, to working to get the right red to right hands sooner - that is my call for action. And with a system of systems being automated, the granularity of feedback is a fascinating problem to solve.