Testing fast and slow

A few days into yet another monthly release, this one with team developer's feeling less motivated as the work was analyzed for them and they were asked not to think too much. End result is a nervous tester in the team, when bits and pieces are developed in isolation without much consideration of what purpose they're supposed to serve. There is no spec, and there's a atmosphere of denial for all issues that I may point out. Not a normal case, fortunately. Pressure is on for others as well, I think.

Feeling the schedule pressure makes me test in a different way. I try to do a lot in a short interval of time, and I report bugs quickly without isolating them in detail. I realize I do this with the chance of buying time - with enough relevant issues that we're remotely aware of takes away the nice fuzzy feeling of "all is well since we did not see any problems" but turns us to another nasty, unusual case of having lots of issues were unable to fix because we can't yet reproduce them. And  not having the repro steps is a way of dismissing the info too.

I just realized, looking at the differences in how the business owner reacts, that we really need to talk about the difference in testing fast and testing slow. Changing the pattern here is justifiable, but quite confusing when done without a warning.

I'm not sure which one is worse:

• releasing with hints of problems we cannot reproduce (yet)
• releasing with a few more fixes (due to nicely isolated bug reports) but not knowing what the users will experience

Seems with the time left I can't have both.

Introducing Test-Fix-Finalize week monthly

About three months ago, being the only tester in my team of 8 developers all finishing their work in the last minute to release, I was thinking about possible solutions I'd go about suggesting. I started discussions with a project manager and my personal manager with the idea of introducing agile planning and following remaining work so that testing would be included. For various reasons, right when we started talking of this I realized this is not about to go through: the project manager was not ready. So I changed what I suggested and went with the "need time in schedule for testing to happen" and "one of me can't do all the checking & exploring on the last week alone so team needs to help".

The idea was formulated a bit more and soon I found myself in front of a steering group allocating the whole team to "testing" for the last week of each increment we decided to call Test-Fix-Finalize. It's very much a waterfallish approach to leave testing (some testing, not all though) in the end and it's really not what I see us aiming for. But, with a team with no automated unit tests and a culture of "let the others' do the testing" this seemed like a smart next move.

For the Test-Fix-Finalize week we identified our team would be working on brain-engaged testing (sometimes pairing up), fixing whatever was found and needed to be fixed and for any time we could, on developing the unit tests further.

We have now two of these Test-Fix-Finalize weeks behind us, and I feel puzzled.

On the first one of these, we tested to find nothing that needed to be fixed. Yet, there has been few issues in found in production we missed. This doesn't surprise me, though - there's quite much width and depth to what might need to be tested, and the our best bets for choices did not pay off in the optimal way. Most of the team did not work on the release testing though and looking at the issues the added investment wouldn't have been worth it either. Some of the developers worked on unit tests and refactoring to make them possible, and others chose to do manual testing of a major refactoring that looked awful lot like fixing and developing. One opted to help me with the release testing, and while I found something yet nothing that was newly introduced, he just found nothing (yet the area he works in cannot be touched without finding something).

I talked about the experience to realize three things:

• The developers can get better at checking, but their style of manual testing may still long be just confirming their designs - lots of paired work may be ahead while some already is done. Having them do work they don't know how to do (and sometimes feel like don't want to do) will not provide the results we seek.
• It might be a better strategy to use all weeks, including the final week on some test automation stuff since that approach wouldn't leak much more than the current one but might get better over time. We could take the risks while development still is as careful and fast to fix reported issues as they are now. 
• The themes in this particular increment were such that there was not as much to test as had been when I introduced this week. The cycles in how things come in had made the team focus on fixes and enhancements and not major features. 

Along came the second Test-Fix-Finalize week. In this cycle I had all developers focus on unit tests, even the ones who wanted eagerly to do manual testing that looks like fixing. We talked and I suggested they'd use the week as an opportunity to learn and try things out - quality over quantity. This month we had significant new features coming in, in the extent that at the start of the month we were already discussing how we'll cope with the last week at all, to learn that we could set up schedules and order of work now - finally - in the order where we considered testing timeframe outside the last week. So to my surprise, two of the three major change areas for the month were tested before the last week. This time with less testing hands on the last week than before, we found two issues that needed addressing before release (out of all the ones that need addressing at some point). One of those issues was such that it alone was worth the effort. Then again, being realist, I know that it would have been caught by the next layer of our internal product management trying things out too - it was way too visible to miss.

End results I'm still processing. There were significant individual differences on whether people could grasp the unit testing and a fair amount of disbelief it will help us finding the things that end up breaking due to their nature looking back at the time I've had the pleasure of being there. We're also noticing that introducing new developers (which we've needed to do) is tipping the scale to another direction than the seasoned product expert developers on the amount of unintended side effects.

One good thing on these experiments, though. I have something to show on the results, and the project manager may finally be ready to bend towards us planning the release together in more of an agile-style planning.

All It Takes is One Wrong Bug

I feel to urge to make a note on the other side of what I posted yesterday on the value of having bugs in the product - to build a customer relationship through quick fixes and avoid spending time finding them.

First of all, the bugs that I mention as samples in my "since April" timeframe are a small portion of all bugs logged and even smaller portion of all bugs fixed. These are a special category of things that when found - by real end users, and important enough for them to find the hard route in their organization to complain about it to us - will be addressed differently. We talk of them as branch fixes, basically just indicating we'll do single changes in two different versions separately to fix them.

I'm not happy with the amount of branch fixes. Our product managers are not happy with the amount of branch fixes. And most definitely, our developers are  not happy with the amount of branch fixes. All I'm saying is that from the product manager's side, they did not actually see the valuable side of these in building the customer relation before they were asked about it. And there are other ways - while my theory still is that maybe not as effective - to get customers to answer the calls when sales people call and build a customer relationship on mutual successes.

Not all of our bugs are quick fixes. We've just been lucky that the ones customers have complained about (perhaps the ones they have understood well enough to complain about) have been that.

Just yesterday we were fixing something that was relevant for the users (internal users complained about it) that wasn't so easy - a browser specific issue on a functionality not working when clicked one way vs. the other. There was a workaround, not everyone would hit that but definitely, not an easy fix. I recognize from my team that if this would be one of those branch fixes, they'd probably first hide it and take more time for the actual resolution.

In addition to only mentioning, intentionally, one category of bugs we deal with, I did not mention that the timeframe I'm looking at is incremental feature addition to an existing architecture. The sister project I also work with is in the early stages of building the frame, and much of the issues there take longer to resolve. The frame for this product has, with all its limitations, been in production for almost two years, and I imagine (and have read from the bug database) that the earlier phases of the product lifecycle were different.

Eventually, all it takes for customers to  not recover from finding a bug and getting it resolved quickly is one bug, that is relevant enough. Relevant enough to consider moving to competing products (assuming there was an option...), relevant enough to just make a move where the compensation for the customers losses would be something they seek.

One bug could still bring down important customer relationships. So for the bugs that we'd want customers to find, I still argue that we'd like to be informed - through testing - that none of them is the business-killer -type.

When Bugs Have Positive Business Value

There's a theory I'm working on analyzing in my current project. For customer value experience in total, it may be good to not find bugs but let customers find them for us.

As anyone reading my posting should know, I'm a tester. I believe in finding and fixing relevant problems before release. I believe we shouldn't steal the user's time with the annoyances and unworking features, when they actually buy the software to do something with it they should actually  get to do what they expect.

My current project is making me question my beliefs.

First of all, if we don't put so much effort into testing, we are able to produce more in quantities. The quality that we produce isn't great, it doesn't live up to my expectations and perhaps not other people's expectations. But it isn't that bad either. You can usually do things with the software. It doesn't stop you right at the footstep, without letting you see in. There's scenarios that fail, but often there are also optional scenarios to the same thing that don't fail.

Second, when customers find bugs and report them to us, we are quick to fix them. This is the core of my theory: there's actually a lot of value for the customer-relationship on these little shared successes. Customer finds an issue, annoying enough that they call us. We take it seriously, and fix it quickly. Customer, comparing this experience with some others where the issue is logged and it may come some time in some hotfix delivered in 6 months, gets really happy with the fix delivery. And as end result, the customer relationship is stronger, and it may even be that the second call back to the customer telling the fix is available includes also expanding the use of features for another project / feature area - sales happening.  

So far I've realized this approach is vulnerable, and it's really still only a play in my mind:

• If we get too many fixes in short timeframe, we wouldn't be able to keep up with the quick deliveries of fixes - but our quality, as limited as it may be, has not gone to this level yet.
• If the customer profile changes so that they'd end up contacting us on different issues on the same days, this would also ruin our ability to react quickly.
• If the software delivery mechanisms changes so that the servers are not quick-and-easy to update, that again would destroy it.
• If the development team members change, it will eat away the quickness of fixing, as there's more analysis and learning to be done to do a successful fix.

I'm thinking right now, that the work I do as the teams's tester, might actually currently decrease the value for the customer. While features may work better, they may work better in ways the users did not find relevant. At least the testing I (and the team) do means that we deliver less features with the same amount of effort.

The bigger value in quality is about the work that the team must do. It's not much fun to actually fix issues come in later, having forgotten about the details of implementation by that time. It's not fun that you can't make (and stick) to a plan for more than half-a-week, because you always need to be alert to implement the quick fixes. The bug-time is away from implementing new features.

Quite an equation we have here. After this quick note of it, I need to actually spend time breaking it down to something I can investigate for this particular context.
 

Mention an issue as action

Some time ago, I learned the words mimeomorphic and polymorphic actions from Michael Bolton. The concept of actions that machines can mimic and actions where you expect a different thing feel relevant for testing, so I went for more information and just got my copy of Shape of Actions by Collins et al. 

Just reading the first chapters of concepts, I notice I'm already thinking about my actions as the team's tester. I noticed a pattern of what I intentionally do that I did not pay much attention to before: action of mentioning issues.

This team's developers are onsite only Mondays and Thursdays, so those are naturally more communication / problem-solving oriented days. In this product, I have the habit of logging into Jira every morning as my first thing, to see what new comes in - from customers, through product managers or from product managers. I also read the comments people make on issues of others with the intent of learning about the product.

Yesterday two issues were particularly interesting related to the mentioning them -action.

The first one was a comment on an issue that could not be fixed since it did not reproduce and as per error message was "external". Reading the comment I could not help but laugh out loud, as this was the first time I saw something like this in this team's comments. The team is usually so well in system ownership, that this is out of ordinary. As my amusement already disrupted the peace, I could go right into mentioning what I just thought of. As reaction, the developer next to me went to the code mentioned in the issue and came up with a solution we can do. By end of the day, there was a solution to the external problem, that was easy to dismiss by someone else - a newcomer to the team. What especially stuck with me was the close-by developer who suggested the actual fix saying "we here have learned that the external issues will come back to us, so it's just better to handle them right away".

The second one was an issue that was noticed in production, and had probably been there for a while. There was no owner assigned for the issue automatically, as it was in an area with somewhat unclear ownership. So instead of waiting for those responsible for allocating the work to find someone appropriate, I just asked about the issue. I asked if we had done any changes that could trigger this problem. Two developers found it interesting problem, and within the next hour did some paired investigation of it, resulting in a fix.

I realized that whenever I chose something to mention, things like these would happen - fixes would be created. Mentioning too many would ruin it for all of us, but mentioning just the right ones helps make the product better and the team feel small successes. Yesterday's two mentions reached out to our product management too, and I was delighted to see an email this morning from the head of product management telling she "buys this fix" with an emphasis that something good had happened.

Reminded: some things are easy with code

As part of my usual routines, I reviewed problems found recently in the versions that have been released to production. I noticed one in particular, where it was evident that this one is a recent introduction of a problem in something that used to work.

In particular, we introduced in the latest version a feature for our combo boxes, to make a selection of all vs. none with one click. The change wasn't very large in development size, and as it was not considered very risky (still isn't), I tested it by sampling some of the combo boxes, focusing on ones the developer was more unlikely to check as they were not in his own area of responsibility.

The issue found in the production version by one of our product managers - not by the customers who don't seem to complain about things more relevant than this. The issue was that one of the combo boxes in his area showed more items than the count would tell - the count was always one lower.

From seeing the bug report, it was quite easy to tie the connection. We just changed the combo boxes, and then one of them doesn't work as before.

I talked about that with two developers, one who changed the combo box and one whose area had the combo box that was causing the trouble. We looked at that piece of code, to notice that the call for the combo box was not implemented as instructed. A simple one-liner to fix.

My first reaction was thinking how we could have tested better, to realize this particular combo box was different. Right after that thought, I realized that there was nothing that told us that this one was the only different one. So, already next to the code, I asked if this was elsewhere. With few clicks, I was told there was 686 mentions of the method in the code, and a comment to just give five minutes and I could have my answer.

Five minutes later, I had learned that that was actually the only one left with that particular problem. There had been another one identified during the month, and fixed. But no one asked the question if this was also elsewhere, so we failed to learn all the things we could / should from a sample.

This reminded me of two lessons I seem to keep forgetting:

• Some things you shouldn't test for with the software, there are just easier routes to find the information you're looking for. 
• Encouraging people to address a family of problems instead of the mentioned problem is something that needs emphasis.

When I shared this story to a more distant colleague, the first reaction was that test automation could have caught this. I think not - we wouldn't have automated a check for all the 200 + combo box instances for this particular problem. Instead, we could easily ask if there was something other than testing with the software that would help us understand if it could fail. And a combo box that has always been a problem because it's sensitive to exactly right way of calling was a piece of information that we would have been able to use, if we worked more as a team and less as individuals each doing their separate responsibility.

Finding problems before and during development

Recently, I've been thinking about specification by example. For the last two days, I've been thinking that with the datapoints I'm seeing at my work, it may be an investment not worth the effort we'd put there.

In the last two days, I've logged plenty of issues on a particular feature area not working as it should. It actually works as specified, or within the limits of how it's been specified, but just doesn't serve its purpose.

I realize I could have - if I even looked at the thing earlier - told about some of the issues we'll be facing later on. Now that I tell of them, they're concrete behaviors (or lack of thereof) in the implemented system.

But what surprised me is that there seems to be very little rework, but more of extra work that just hasn't been done yet. And for now, it seems that the risk of regression isn't that big a deal either - with the type of system, unit tests seem to do a decent job on their own.

I reserve my right to rethink this. But I just find it hard to digest, that using more effort before development than less during development would be the way to go.