Tuesday, April 26, 2016

How would you count bug reports you never get?

There's recently been a significant transformation on a piece of software we work on with my team. In the last month, we've done more releases of that piece than we managed to do in the year before it. There's a clear feeling of progress. And the progress has included the amount of code for the piece dropping to a quarter of where we started off with the transformation. The final frontier of siloed development has joined team ownership and our drive to clean up things to be understood.

Looking at this piece of software, I'm observing an interesting aura of mysticism on how users approach it. It's like I'm moving into the past where I four years ago surprised our product manager with the idea that software can work when he tries it out - something that should be close to obvious in my books.

I started to think about this seeing a tweet:
The worst piece of code my team delivered probably had also just one or two bugs in production over the last year. But for us, the reason was that the users would seek workarounds rather than ask for fixes, or that the bugs just did not bug the users enough for them to speak up.

Since we moved the piece of software into the frequent release cycles, we've fixed a LOT of bugs users never complained about. A lot that users had not even run into. Looking at the  bugs we've fixed, I find it nearly impossible to believe that they could have been using the software all this time. But they did, they found their ways around everything that bugs us (and slows them down).

When you get little to none in bug reports from production, do you assume all is well? My sample says I shouldn't. Even asking the users directly might not give you a realistic picture.

And on the other areas where feedback is more frequent: if it bugs a user, it's a bug. And we have loads of things that bug the user, that we rather frame as feature requests, some of which we have no intention of acting upon. Counting them seems more of counterproductive. We could count how often we get interrupted with a quick request to change something in production that reorders our immediate priorities? But most of those wouldn't count as bugs in any traditional sense.


  1. Hey Maaret,

    Two ideas spring to mind:

    1) Maybe there is insightful monitoring in place (with complimentary analysis!) on some of these products?

    2) Maybe usability testing is in place and 'bugs' appear as feature requests/stories on the teams backlog?

    In both instances that might explain why teams like Mr Zuill's seem ok with 'zero bugs'? Because bugs are detected/reported outside the typical 'bug report' mechanism?

    Does your team do either of those things (or something else that would provide a channel for bugs to be 'reported')?


    1. Yes, my team does both of those mechanisms - now also for the piece of software that puzzles me on how bad it can be.

      I'm just noticing that I get more bugs out of every software (with real and imaginary user collaboration) than companies who keep thing they have no bugs in production. Lack of exploratory tester makes me suspicious of how critically the product has been looked at.

  2. This blog awesome and i learn a lot about programming from here.The best thing about this blog is that you doing from beginning to experts level.

    Love from

  3. This is something I struggle with in discussions with my developers around bugs, to fix or not to fix.

    We are always behind the ball in terms of work, so new functionality is always prioritised over bug fixing unless the bugs block the new functionality, are a critical bug (crashing, affects financial data etc) or a user has complained about it.

    This however leads to a great many "minor" bugs that simply stagnate at the bottom of the backlog. The reason being, that if they're important they'll be raised by the users. If not, they obviously don't care.

    But this argument doesn't work when we know that the act of complaining/sending in feedback is more trouble to some people than them just finding a workaround, or tolerating the bug. But if we have no visibility of the above cases, I can't feed them into my bug advocacy... and the bugs will continue to be "at the bottom of the list".

    1. I've struggled with the same. I found a few strategies that have helped me.
      1) I've learned to fix some bugs myself, in particular the types that take longer to report than to fix.
      2) I've cultivated relationship with one (or more) devs who will pair up with me to fix stuff that they wouldn't fix by themselves
      3) I've brought in "real users" to be listened to.