All It Takes is One Wrong Bug

I feel to urge to make a note on the other side of what I posted yesterday on the value of having bugs in the product - to build a customer relationship through quick fixes and avoid spending time finding them.

First of all, the bugs that I mention as samples in my "since April" timeframe are a small portion of all bugs logged and even smaller portion of all bugs fixed. These are a special category of things that when found - by real end users, and important enough for them to find the hard route in their organization to complain about it to us - will be addressed differently. We talk of them as branch fixes, basically just indicating we'll do single changes in two different versions separately to fix them.

I'm not happy with the amount of branch fixes. Our product managers are not happy with the amount of branch fixes. And most definitely, our developers are  not happy with the amount of branch fixes. All I'm saying is that from the product manager's side, they did not actually see the valuable side of these in building the customer relation before they were asked about it. And there are other ways - while my theory still is that maybe not as effective - to get customers to answer the calls when sales people call and build a customer relationship on mutual successes.

Not all of our bugs are quick fixes. We've just been lucky that the ones customers have complained about (perhaps the ones they have understood well enough to complain about) have been that.

Just yesterday we were fixing something that was relevant for the users (internal users complained about it) that wasn't so easy - a browser specific issue on a functionality not working when clicked one way vs. the other. There was a workaround, not everyone would hit that but definitely, not an easy fix. I recognize from my team that if this would be one of those branch fixes, they'd probably first hide it and take more time for the actual resolution.

In addition to only mentioning, intentionally, one category of bugs we deal with, I did not mention that the timeframe I'm looking at is incremental feature addition to an existing architecture. The sister project I also work with is in the early stages of building the frame, and much of the issues there take longer to resolve. The frame for this product has, with all its limitations, been in production for almost two years, and I imagine (and have read from the bug database) that the earlier phases of the product lifecycle were different.

Eventually, all it takes for customers to  not recover from finding a bug and getting it resolved quickly is one bug, that is relevant enough. Relevant enough to consider moving to competing products (assuming there was an option...), relevant enough to just make a move where the compensation for the customers losses would be something they seek.

One bug could still bring down important customer relationships. So for the bugs that we'd want customers to find, I still argue that we'd like to be informed - through testing - that none of them is the business-killer -type.

When Bugs Have Positive Business Value

There's a theory I'm working on analyzing in my current project. For customer value experience in total, it may be good to not find bugs but let customers find them for us.

As anyone reading my posting should know, I'm a tester. I believe in finding and fixing relevant problems before release. I believe we shouldn't steal the user's time with the annoyances and unworking features, when they actually buy the software to do something with it they should actually  get to do what they expect.

My current project is making me question my beliefs.

First of all, if we don't put so much effort into testing, we are able to produce more in quantities. The quality that we produce isn't great, it doesn't live up to my expectations and perhaps not other people's expectations. But it isn't that bad either. You can usually do things with the software. It doesn't stop you right at the footstep, without letting you see in. There's scenarios that fail, but often there are also optional scenarios to the same thing that don't fail.

Second, when customers find bugs and report them to us, we are quick to fix them. This is the core of my theory: there's actually a lot of value for the customer-relationship on these little shared successes. Customer finds an issue, annoying enough that they call us. We take it seriously, and fix it quickly. Customer, comparing this experience with some others where the issue is logged and it may come some time in some hotfix delivered in 6 months, gets really happy with the fix delivery. And as end result, the customer relationship is stronger, and it may even be that the second call back to the customer telling the fix is available includes also expanding the use of features for another project / feature area - sales happening.  

So far I've realized this approach is vulnerable, and it's really still only a play in my mind:

• If we get too many fixes in short timeframe, we wouldn't be able to keep up with the quick deliveries of fixes - but our quality, as limited as it may be, has not gone to this level yet.
• If the customer profile changes so that they'd end up contacting us on different issues on the same days, this would also ruin our ability to react quickly.
• If the software delivery mechanisms changes so that the servers are not quick-and-easy to update, that again would destroy it.
• If the development team members change, it will eat away the quickness of fixing, as there's more analysis and learning to be done to do a successful fix.

I'm thinking right now, that the work I do as the teams's tester, might actually currently decrease the value for the customer. While features may work better, they may work better in ways the users did not find relevant. At least the testing I (and the team) do means that we deliver less features with the same amount of effort.

The bigger value in quality is about the work that the team must do. It's not much fun to actually fix issues come in later, having forgotten about the details of implementation by that time. It's not fun that you can't make (and stick) to a plan for more than half-a-week, because you always need to be alert to implement the quick fixes. The bug-time is away from implementing new features.

Quite an equation we have here. After this quick note of it, I need to actually spend time breaking it down to something I can investigate for this particular context.
 

Mention an issue as action

Some time ago, I learned the words mimeomorphic and polymorphic actions from Michael Bolton. The concept of actions that machines can mimic and actions where you expect a different thing feel relevant for testing, so I went for more information and just got my copy of Shape of Actions by Collins et al. 

Just reading the first chapters of concepts, I notice I'm already thinking about my actions as the team's tester. I noticed a pattern of what I intentionally do that I did not pay much attention to before: action of mentioning issues.

This team's developers are onsite only Mondays and Thursdays, so those are naturally more communication / problem-solving oriented days. In this product, I have the habit of logging into Jira every morning as my first thing, to see what new comes in - from customers, through product managers or from product managers. I also read the comments people make on issues of others with the intent of learning about the product.

Yesterday two issues were particularly interesting related to the mentioning them -action.

The first one was a comment on an issue that could not be fixed since it did not reproduce and as per error message was "external". Reading the comment I could not help but laugh out loud, as this was the first time I saw something like this in this team's comments. The team is usually so well in system ownership, that this is out of ordinary. As my amusement already disrupted the peace, I could go right into mentioning what I just thought of. As reaction, the developer next to me went to the code mentioned in the issue and came up with a solution we can do. By end of the day, there was a solution to the external problem, that was easy to dismiss by someone else - a newcomer to the team. What especially stuck with me was the close-by developer who suggested the actual fix saying "we here have learned that the external issues will come back to us, so it's just better to handle them right away".

The second one was an issue that was noticed in production, and had probably been there for a while. There was no owner assigned for the issue automatically, as it was in an area with somewhat unclear ownership. So instead of waiting for those responsible for allocating the work to find someone appropriate, I just asked about the issue. I asked if we had done any changes that could trigger this problem. Two developers found it interesting problem, and within the next hour did some paired investigation of it, resulting in a fix.

I realized that whenever I chose something to mention, things like these would happen - fixes would be created. Mentioning too many would ruin it for all of us, but mentioning just the right ones helps make the product better and the team feel small successes. Yesterday's two mentions reached out to our product management too, and I was delighted to see an email this morning from the head of product management telling she "buys this fix" with an emphasis that something good had happened.

Reminded: some things are easy with code

As part of my usual routines, I reviewed problems found recently in the versions that have been released to production. I noticed one in particular, where it was evident that this one is a recent introduction of a problem in something that used to work.

In particular, we introduced in the latest version a feature for our combo boxes, to make a selection of all vs. none with one click. The change wasn't very large in development size, and as it was not considered very risky (still isn't), I tested it by sampling some of the combo boxes, focusing on ones the developer was more unlikely to check as they were not in his own area of responsibility.

The issue found in the production version by one of our product managers - not by the customers who don't seem to complain about things more relevant than this. The issue was that one of the combo boxes in his area showed more items than the count would tell - the count was always one lower.

From seeing the bug report, it was quite easy to tie the connection. We just changed the combo boxes, and then one of them doesn't work as before.

I talked about that with two developers, one who changed the combo box and one whose area had the combo box that was causing the trouble. We looked at that piece of code, to notice that the call for the combo box was not implemented as instructed. A simple one-liner to fix.

My first reaction was thinking how we could have tested better, to realize this particular combo box was different. Right after that thought, I realized that there was nothing that told us that this one was the only different one. So, already next to the code, I asked if this was elsewhere. With few clicks, I was told there was 686 mentions of the method in the code, and a comment to just give five minutes and I could have my answer.

Five minutes later, I had learned that that was actually the only one left with that particular problem. There had been another one identified during the month, and fixed. But no one asked the question if this was also elsewhere, so we failed to learn all the things we could / should from a sample.

This reminded me of two lessons I seem to keep forgetting:

• Some things you shouldn't test for with the software, there are just easier routes to find the information you're looking for. 
• Encouraging people to address a family of problems instead of the mentioned problem is something that needs emphasis.

When I shared this story to a more distant colleague, the first reaction was that test automation could have caught this. I think not - we wouldn't have automated a check for all the 200 + combo box instances for this particular problem. Instead, we could easily ask if there was something other than testing with the software that would help us understand if it could fail. And a combo box that has always been a problem because it's sensitive to exactly right way of calling was a piece of information that we would have been able to use, if we worked more as a team and less as individuals each doing their separate responsibility.

Finding problems before and during development

Recently, I've been thinking about specification by example. For the last two days, I've been thinking that with the datapoints I'm seeing at my work, it may be an investment not worth the effort we'd put there.

In the last two days, I've logged plenty of issues on a particular feature area not working as it should. It actually works as specified, or within the limits of how it's been specified, but just doesn't serve its purpose.

I realize I could have - if I even looked at the thing earlier - told about some of the issues we'll be facing later on. Now that I tell of them, they're concrete behaviors (or lack of thereof) in the implemented system.

But what surprised me is that there seems to be very little rework, but more of extra work that just hasn't been done yet. And for now, it seems that the risk of regression isn't that big a deal either - with the type of system, unit tests seem to do a decent job on their own.

I reserve my right to rethink this. But I just find it hard to digest, that using more effort before development than less during development would be the way to go.

Thinking through what sticks with me from Rapid Testing Intensive

I participated end-of-October - as an usual student of testing - on a training, that I was really not sure what exactly to expect of: Rapid Testing Intensive by James Bach in Helsinki. The course outline said we'd be testing something for two days and learn from this mini-project. Someone tweeting about this gave me an impression this would be RST with just one software, seeing how it works in real projects, so I wasn't really sure what that meant. It could mean, we focus on the bunch of heuristics and trying those out in practice, or we focus on the doing testing and understanding the flow of it. I feel latter was closer to what happened.

To sum up what I remember we were taught:

• Testing actions recipe to cook from: intake, survey, analysis, setup, deep coverage, closure - where any of the types of actions could be a focus of a session, there could be mix of two, and the actions could take place in orders other than assuming there is an order.
  Over the two days tried out different types of activities: asking about the project, finding out what the product could be about by testing it, using help files in testing,  making a test strategy to focus our testing with risks and specific testing activities we could do, tried working with generated data, and reported our status and results. 
• Talking about testing, the difference between a professional (focused for the audience) and a comprehensive test report, and in particular the three levels of reporting testing: status of the product, show it has these bugs (what you did and didn't test), why the chosen testing and how we know if it was good.

By the end of the course, I was more inside my head thinking of how to structure the mess better than with the activities people could imagine they view from the outside. I stopped to think about how I think and feel, and how my choices and somebody else's choices would be different. I realized that I personally dislike the concept of "emergency testing" of something I have no previous experience on. When time is way too little for doing anything professional, I have the tendency to focus on playing time - just finding something that would delay the release. And when I feel that there's nothing, for this particular context that would buy time, I noticed I realize what I should have done too late - when we're already out of time.

We tested XMind, a released version. While the course is half make-belief of an end emergency testing situation, I couldn't help but thinking that this is what they have already released. Would any of the bugs we currently have in production actually made a difference to timing - perhaps not. And if not, what's the rush?  Remembering which parts of context are imaginary for course purposes and which parts would actually be things happening with that particular product and its release decision got me confused a little.

Since I did not want to miss out on my notes of what was said and what we were told, I used a lot more of our testing sessions wondering somewhere else in my thoughts, and actually testing the product like it was real project. That was my choice, to take time from learning and digesting. I probably went to find our a bit different things than others, my main point of curiosity was not towards how would I test better, but how do others teach this.

A great course, all in all. So much like my exploratory testing work course, expect that personal preferences make us focus on quite different aspects. It was like comparing coaching styles with a true expert - without saying that out loud. Only thing I regret is not making the point of being there with my team's developers - they would have learned so much.

Funny how my mind works

As I split my testing time between two projects / systems at work, it's fun to notice the differences in the projects. The other project is a new product, originally with very little features but as usual, growing with time. It' not yet in production, but will be in about a month. Being there on-time has been a great experience.

Since I started with the project early in the development, we talked of what information would be most relevant, and that guided what I reported. I could always use more time on this project, but the other project with already-in-production setting also needed attention. So I'd just try and do the best I can with the time available. One of the things that was deemed not so relevant was certain types of typos - in database contents. I learned to skip them without being too much annoyed.

Then with the new features being added, in came a reporting feature, that would take the contents and show them in a deliverable that is clearly intended for our customers. Previously the interfaces I had were for internal use, but this one changed the game.

I talked with the project manager about the typos, and after the first sentence, he said they're not relevant. I smiled, and told him that I find it funny how my mind works: it was natural to pass them, while they were visible on the internal interfaces, but now that my mind is set up for the idea that we give THIS to our customers, I have this feeling that they may think that typos, especially a large number of them, is sort of unprofessional. I could see from the project managers face that something clicked, and he continued, not with a "but" but with "and" - and it would be so much easier to fix those before going live, as there will be dependencies on stuff created when we get this out.

In just a few sentences we changed from "not relevant" to "good idea". And really, my mind works so that when I recognize the type of the user, I notice different types of issues.