Thursday, May 1, 2014

Optimizing efforts into bug reporting

A colleague from another company contacted me  asking about numbers on how many bugs testers find per a time unit. I remembered checking that number for our organization last autumn for a presentation I gave, and now checked again to see if had changed. It's not our goal or connected in any way with how we set goals, but I find it occasionally an interesting trigger for thinking through the results. The number had not changed at all - we log 5,8 issues for a day. It was the same when I was the only tester, it was the same when the second tester started learning and it's still the same for the two of us combined. Somehow it's more of a number that reflects the point when we prefer doing something else than logging issues.

We also had a paired testing session for the two of us. During the two focused hours, we covered rather shallowly a new area we had not had time for otherwise to find about 20 issues. As the area was in the other tester's product (current split of responsibilities is that I only manage on this product's team) I left all the issues to be logged for the other tester. Later in the afternoon she mentioned that she will need to leave the logging of issues for next week, as there's resolved issues to work on and two days of out-of-office ahead for her. This triggered my favorite concern, the amount of time wasted on good and detailed bug reporting out of habit. And after all, that's what testers are taught. I replied to say I will quickly put one issue with all the bugs I saw, just copypasting my notes in that one issue. The reasons for unclear summary style reporting are many in this case:
  1. Copy-paste of notes with the unclarities takes me 30 seconds, while clear repro steps in step-by-step style easily take 5 minute each. With 12 in the queue, that's an hour of work!
  2. Unclear reporting triggers the devs to talk to me. And they do, often. The positive impact of those discussions have been significant. 
  3. The oneliners tend to be already enough to get the bug in the first place - most of the time. Not writing for the audience but on template wastes effort. 
  4. If I reported now, the devs could already fix them. And some of those most likely will before there is the time in calendar that would allow the detailed logging. 
  5. The rare skill in our team is to see problems. There's plenty of developers to isolate the problems if they're hinted there is a problem.
There's a huge difference in the reporting style for me and the other tester. If you've taken Cem Kaner'sbug advocacy or read articles on clear bug reporting, the other tester does all that.  But the other tester also must invest more time in logging. I write onliners describe just the relevant data, and let pictures talk for me. If my current reports were showcased to externals, they would not be clear enough. But it seems, asking the developers, that most of the time they are sufficient. And they can always trust me to demo things for them. 

Imagine the amount of wasted effort on the detailed reports if the detail is not necessary: 5,8 bugs every day for two 25 months with 20 working days. 242 hours if 5 minutes per bug was enough. 

I'm a big fan of thinking all activities - including the selection to log issues imperfectly or leave them unlogged completely using your judgement - in the frame of opportunity cost. With another choice of allocation, could you have done something more valuable? Not just the cost of testing, but the costs in development. Bug reporting in the way most books describe is not a best practice. My style isn't either. But my choices seem to fit this particular context - for now. And I can ask, as the test manager, the other tester to start experimenting with a new, quicker style of reporting that will take her probably out of her comfort zone. 

Later addition: Asking the developers on the preference I learned two things. They find the step-by-step descriptions wasting their effort - they need to learn long stuff to get something they could get in more concise form. And they have been annoyed - without ever saying a word - that all bug reports are logged in evenings, as in they are kept in store for the whole day when there is a chance to do something about them. Know your audience. Ask, try different things. Pay attention to things that people don't say.

Yet another later addition: I seem to have offended the contracted tester's manager by making a claim of "all bug reports are logged in evenings", when the data shows that only a majority but not all of bugs are logged after developers leave office at 15.00. Just to be clear, I added a the previous report of verbally transferred information, and whether it is all or some, it is a feeling that was was stated. I reported that not to offend, but to note that the pacing of how we choose to report with regards to when we find the issues also has an impact the developers notice.

Choices of my word may be incorrect as in all vs. majority, but they are well meaning to describe recent events that reflect stuff that I have also learned over the years. I don't do safety language that well, but don't intend to stop writing because of it.


  1. Hi Maaret!
    Interesting post about very important tester skill.
    Here is my feedback on it:
    2. What if unclear reporting triggers discussion that lasts for 1 hour? But if you spent 10 minutes on issue report with more details, conversation would not be needed.
    What if developer just dismiss issue, without any conversation? Context matters. For example when tester works with new team, in the begging it is important to gain developers credibility, and one way of doing it is to advocate the bug issues. In later phase, advocated reported issue could be in your notes form.

    Why do you think that Bug advocacy course favourites writing issue reports that require a lot of time? You referenced some testing books that describe bug reporting that is not "best practice". Could you list which books did you have on your mind?


    Regards, Karlo.

    1. If unclear reporting _sometimes_ causes confusion, that is not a reason to always pay extra in effort to avoid a confusion that may not happen. Whenever quick reporting causes confusion, you can stop and think what you could learn about that. Sometimes the confusions end up being great glue for the team of devs and testers, sometimes the confusions end up isolating the tester. There isn't one rule on how the practice goes, that is what I'm saying.

      On the case of working with a new team and gaining credibility, logging your issues (making developers look bad) may not be the best case. How about finding bugs that they too feel they want to fix, without the concern on logging? I personally find that small issues (that devs don't care about) eat so much of my energy when I test that it may take a lot of focus to find something they do care about. Going past my natural inclination of the order of things is something I've needed to learn to do.

      On matter of Bug Advocacy course and the "how to report bugs" chapters in various books: I think they all are children of their time and context. When we work in separate teams of devs and testers and rarely see each other, the written communication plays a different role than when we're a very small colocated team with a common goal. I love the Bug Advocacy course, and find clear step-by-step bug reports something that I value often. But I notice I value it more on the side of making sure you can reproduce than the steps description. And it depends on the complexity of the issue you're reporting how relevant the steps are. Unfortunately, many bugs with our software are not that complex.

      I think there are no best practices - only good practices in contexts. Something that works well in my teams may not work well for other teams. I'm just saying we should, in my experience, be aware of the opportunity costs and not think of even bug reporting as a straightforward "we know how this is done best" practice.

    2. Hi!

      I agree with first paragraph.

      What troubles me in second paragraph is this:
      "On the case of working with a new team and gaining credibility, logging your issues (making developers look bad) may not be the best case."
      Is this based on your testing experience?
      If this is true, you are trying to solve your REAL problem (issue makes developer bad) in wrong way.

      Also, I can not see the reason why your notes filled in as issue can not be aligned with bug advocacy course concepts? And what is the unit of short/long issue report metric? Where is the boundary between short and long?
      I understand your context, but I am sure that practicing bug advocacy will make you very effective in writing better issue reports.

      I agree with you that context always matter.


    3. With 17 years in testing with different teams and developers, I'm happy that that is not my current experience as developers welcome feedback and don't find logged bugs intimidating. But it has been one of my experiences over the years that not all developers are like that, in particular if organizations like to measure success on bug counts / lack of thereof. So I think I'm still trying to solve my current real problem, which is to optimize the very limited time available with requiring a deeper level of skill in deciding for each bug the level of effort on reporting, instead of following a template.

      On the second note, I must have said something wrong if the impression is that I think adjusting level of reporting to lower effort considering audiences is against the concept of bug advocacy. What I try to say is that bugs and audiences whom you write to are significantly different, and part of advocacy is to consider the effort you put on fighting for each. Some bugs and teams need the detail, the generalization and all the works to get the proper emphasis. Some teams take a bug as a task and don't dismiss it just because it does not follow a template of excellent bug reports.

      On short/long, the unit was minutes. I don't look for a general rule there, but short in this particular case was that I could report 12 in one minute as one-liner notes and get them into fixing queue sooner. With the template of what they should look like with steps to repro and nicely worded explanations, it would have taken an hour.