Test Documentation and Mindmaps

I have a thing with documentation - I like it. I like to read books that deliver me new concepts, teach me basics of how to do certain things and make me feel inspired. I read articles, and have a soft spot especially for experience reports that don't tell how things should be done elsewhere but outline how (and perhaps why) they were done in one particular place. I like the fact that people put their ideas on paper - with pictures and text - to open a discussion in software projects. And I really enjoy working to identify the core of information that needs to be written down.

I'm turned into a fun of avoiding test documentation. In my main project, I write the end user online help to remember things I might have put in test documentation because it makes the document shared. And it appears people actually do read it. If I have details to write down, I'd prefer to have them written down as comments and implementation into automated tests. And for difficult information on "how was this supposed to work", I regularly volunteer to add stuff on the specifications if the information isn't end-user oriented.

I create test documentation too. I create mindmaps at a point when I know the least. Mindmaps are a cheap form of documenting, they allow me to change my mind as I learn, and they support the learning that happens. When I think I've learned, I transform my mindmaps typically to two documents: a checklist that supports going through the test ideas as the software lives on that usually is in a spreadsheet format. And a short summary document with pictures and text of central concepts I need to quickly remember when I've forgotten all about this area working on another one. And I write reports on issues I notice, and categorize them on their relevance.

As testing is learning, I've noticed over the years that I need to throw away my early documentation structures that I create when I start to test. In the early documents, I put in the stuff that I'm told / read from other sources that exist. Stuff that usually isn't very insightful or complex - the complexity starts to build on top of those. The insights start to happen at times I write bug reports. I learn that things were not as I expected and I log a bug. The insights, if they get collected, get muddled with all the mundane information. And as I come accustomed on my structure of where I write what, I no longer refactor the information for other audiences. The document I started creating works well for this feature / change right now, but in my experience it doesn't serve the needs of those who come later.

Quite a long time ago, I learned to think of test documentation as an output of testing, not input into testing. Test documentation that I'd like to see created serves me when I need to get up to speed with an area other professional tester has tested before me. It delivers some, not all, of the learnings in a condenced format allowing me to get up to speed faster. I can't accept that every tester would start from scratch, there must be ways to accelerate and take the fast lane. Similar documentation I need when I work on an area I haven't had time for in a while - I forget and need a quick reminder when coming back. I believe it's professional to leave things in a better shape than what they were when you entered and care for the one that comes after you, even if no one explicitly tells you so.

This week, I was working on a feature that I realized is similar to something our teams' other tester has worked on. So I needed the summary of the feature and core lessons from that feature already being tested. I digged in for the documentation to find three mindmaps that don't make sense. Surely they list features, but why three? Why they are split this way? Why isn't there just one that I could work with, that would have the up-to-date view. And when I looked at the details of those, there was nothing insightful written down - insightful to me. I also looked into specifications, to notice those do exist, but were out of date. Even if I update specs regularly, I have not emphasized that it could be done by others too. And they were long and boring, with anything insightful that I as a tester could use somewhere between the lines. I know how to read to see things that are not said, but I would love the two page summary instead of the 44 pages.

With this behind me, I tweeted:

I'm starting to seriously dislike mindmaps as test documentation. It gives me a feel of 'unable to write to audience' when all is mindmaps.
— Maaret Pyhäjärvi (@maaretp) May 10, 2014

I'm sure I'm attacking mindmaps just because I'm disappointed in the quality of the ones I looked into in comparison to my expectation at that time. It's hard, if not impossible, to guess the future needs. But in this particular case, looking at the mindmaps again and again, I'm sure the trouble is the mindmaps describe structures from the time we knew the least (before testing) not the time we know the most (when we've tested). I would expect us to have the discipline to spend a moment in the end to make sure that what we leave is - critically looking - the best we can with the latest information and a reasonable amount of effort to balance value / cost.

So I looked around more to realize all documentation, unless I specifically have commanded an other format, is a mindmap. Even the low tech dashboard I've adviced to create is a mindmap. Which gives me the impression - unconfirmed - that there may be a liking of a tool instead of actively thinking of the format in place. Mindmaps are a tool, that brings in a format for the documentation you create with it. I'm not sure if they would work well to capture stories. They seem to work well for listing features and subfeatures, and their relations. And when they have lots of detail, using them to keep track of what you covered just isn't what you could do with a checklist format.

I appreciated what James Bach pointed out:

@vipulkocher @maaretp The hidden point of the mindmap is often to de-hype bloated traditional docs.
— James Marcus Bach (@jamesmarcusbach) May 10, 2014

I think it should not be a hidden point. In this particular case there never was the bloated traditional documentation. There never was a constraint that stopped us from creating the best imaginable documentation for future - except our imagination. The imagination was lacking in thinking of audiences and times of use, focusing just on what I need and can create right now, while testing. I think a big part of the reason why this happens is that there's an over-reliance on mindmaps. The hidden point being the main point might help with the choices.

In the information I was looking for, in all honesty, I could have used a mindmap. It would have just needed to be a mindmap that structures knowledge from the time after test when you know the most. It would have had different contents. But I could also have used a picture of the process embedded into the feature. No text would suffice. And if I would start testing that area, a checklist would be nice. Mindmaps are a poor format for checklists. A checklist, for me, is something that usually has multiple times to tick it off that I want to keep  track of: builds, environments and such.I would not have used step-by-step instructions on how to test, but the ideas of why and what to test, and how to make sense of it all quickly.

The fact that I can rework all the existing documentation again to learn the basics just isn't good enough. I want a better mix of things.

Volunteering - an active approach

On my walk to the office this morning, I realized that my tendency of volunteering has played a significant role on how my career gets built.

Most recently, I volunteered to improve a design - not by stating how it was incorrect or risky (testing) but by coming up with several other suggestions. I ended up owning the design of the feature and the driving the discussions to conclusions and agreement.

On numerous occasions, I've volunteered to deal with something everyone seems to be avoiding that needs doing. I've volunteered to speak out with a personal risk when I know something needs  to be said that others would want to say but don't. Volunteering is my way of contributing. I'm very seldomly assigned something to do, instead I'm actively looking at what needs doing and what I should volunteer for. I don't need to be a manager to be an active player in the direction I'm heading and we're working towards together in the team.

I realized volunteering comes naturally for me. I volunteer for boards of non-profits that I believe in. I volunteer to run a kids computer club for 1st graders to teach them about versatile, collaborative creating with computers instead of programming. I volunteer to organize meetups to meet great people that give me extra energy with their ideas and enthusiasm.

While at university, I volunteered for many student organizing activities. I held several financial responsibility positions organizing events with large audiences. So many of those lessons have been valuable, defining moments on what I do now.

Going all the way back to when I remember volunteering for the first time: my sisters needed me. I volunteered to teach her Swedish so that she would pass a class without meeting a teacher she did not get along with. She aced the exam and was barely allowed on the next class level. I felt useful.

There's so many things that would not have happened without volunteering. It's rewarding and it drives me forward allowing for all options. I suggest everyone should try volunteering. I've been surprised on how much of a difference that makes.

Optimizing efforts into bug reporting

A colleague from another company contacted me  asking about numbers on how many bugs testers find per a time unit. I remembered checking that number for our organization last autumn for a presentation I gave, and now checked again to see if had changed. It's not our goal or connected in any way with how we set goals, but I find it occasionally an interesting trigger for thinking through the results. The number had not changed at all - we log 5,8 issues for a day. It was the same when I was the only tester, it was the same when the second tester started learning and it's still the same for the two of us combined. Somehow it's more of a number that reflects the point when we prefer doing something else than logging issues.

We also had a paired testing session for the two of us. During the two focused hours, we covered rather shallowly a new area we had not had time for otherwise to find about 20 issues. As the area was in the other tester's product (current split of responsibilities is that I only manage on this product's team) I left all the issues to be logged for the other tester. Later in the afternoon she mentioned that she will need to leave the logging of issues for next week, as there's resolved issues to work on and two days of out-of-office ahead for her. This triggered my favorite concern, the amount of time wasted on good and detailed bug reporting out of habit. And after all, that's what testers are taught. I replied to say I will quickly put one issue with all the bugs I saw, just copypasting my notes in that one issue. The reasons for unclear summary style reporting are many in this case:

1. Copy-paste of notes with the unclarities takes me 30 seconds, while clear repro steps in step-by-step style easily take 5 minute each. With 12 in the queue, that's an hour of work!
2. Unclear reporting triggers the devs to talk to me. And they do, often. The positive impact of those discussions have been significant. 
3. The oneliners tend to be already enough to get the bug in the first place - most of the time. Not writing for the audience but on template wastes effort. 
4. If I reported now, the devs could already fix them. And some of those most likely will before there is the time in calendar that would allow the detailed logging. 
5. The rare skill in our team is to see problems. There's plenty of developers to isolate the problems if they're hinted there is a problem.

There's a huge difference in the reporting style for me and the other tester. If you've taken Cem Kaner'sbug advocacy or read articles on clear bug reporting, the other tester does all that.  But the other tester also must invest more time in logging. I write onliners describe just the relevant data, and let pictures talk for me. If my current reports were showcased to externals, they would not be clear enough. But it seems, asking the developers, that most of the time they are sufficient. And they can always trust me to demo things for them. 

Imagine the amount of wasted effort on the detailed reports if the detail is not necessary: 5,8 bugs every day for two 25 months with 20 working days. 242 hours if 5 minutes per bug was enough. 

I'm a big fan of thinking all activities - including the selection to log issues imperfectly or leave them unlogged completely using your judgement - in the frame of opportunity cost. With another choice of allocation, could you have done something more valuable? Not just the cost of testing, but the costs in development. Bug reporting in the way most books describe is not a best practice. My style isn't either. But my choices seem to fit this particular context - for now. And I can ask, as the test manager, the other tester to start experimenting with a new, quicker style of reporting that will take her probably out of her comfort zone. 

Later addition: Asking the developers on the preference I learned two things. They find the step-by-step descriptions wasting their effort - they need to learn long stuff to get something they could get in more concise form. And they have been annoyed - without ever saying a word - that all bug reports are logged in evenings, as in they are kept in store for the whole day when there is a chance to do something about them. Know your audience. Ask, try different things. Pay attention to things that people don't say.

Yet another later addition: I seem to have offended the contracted tester's manager by making a claim of "all bug reports are logged in evenings", when the data shows that only a majority but not all of bugs are logged after developers leave office at 15.00. Just to be clear, I added a the previous report of verbally transferred information, and whether it is all or some, it is a feeling that was was stated. I reported that not to offend, but to note that the pacing of how we choose to report with regards to when we find the issues also has an impact the developers notice.

Choices of my word may be incorrect as in all vs. majority, but they are well meaning to describe recent events that reflect stuff that I have also learned over the years. I don't do safety language that well, but don't intend to stop writing because of it.

Another day, a design concern

I tweeted a Robert Martin quote from Clean Coder -book yesterday:

 “It is the worst kind of unprofessional behavior to simply code from a spec without understanding why that spec makes sense to the business"

As a reply I got a kind note:

@maaretp And same goes for writing the spec :)
— Juha Heimonen (@evilbubu) April 28, 2014

Thus it seems appropriate that I've spent most of today working on a spec, and feel I need to share a few experiences on that. There surely can be a lot of work embedded in the quote of avoiding unprofessional behavior.

There's a new concept we are supposed to work on by the end of this year. As concepts and our experience goes, it tends to be a good idea to play with the concept before jumping into implementing it, to avoid the "implementing 2-3 times just because we did not focus to learn early" -syndrome typical for us.

As usual, two people have had vivid discussions on building the ideas of what the concept / features would be about. They've drawn some pictures that help discuss it. Nothing is set in stone - except the fact that the business representative's calendar is very full, allowing us easy access to discuss with him once next week and then again in a month.

From my initiative, two other people start looking into the draft specification: myself in whatever role I end up with, and a UI designer in as large a role he feels comfortable taking. The quick guidance is that we should discuss after we've read it - same assignment for the two of us.

I read the document to realize that I would not advice implementing any of the stuff the way they've been sketched. The core concepts this builds on are concepts used elsewhere in the product for a completely different scope and meaning. Technically they seem to be suitable (filtering is a general idea), but the user visible concepts have implications that would make things difficult. And there's a new stick-it-on-top-of-the-others view just for this feature - while other features for same users would be elsewhere.

I hint about the problems for the user interface designer, to learn that he read the document, but did not formulate his own view on that. He has a few ideas of how to present the described concepts in the user interfaces sketched, but had not noticed the concepts would not fit our overall idea of what the product is like. We agree from discussion that he'll take another look at the document based on my heads-up.

I'd like to find a way to do two things:

1. Build the skill of noticing when things don't fit the concepts we have in place. I think of this as seeing the system, and testing is a great place to build an overall view of what there is and how things belong together. 
2. Learn to discuss concepts before they give the appearance of some higher-level decisions  are done. As much as I try, the specification by example -type of discussions don't seem to (yet) find their spot in the way we work. And there's more to understanding concepts than just the given-when-thens. 

Just thinking about the amount and scope of questioning my dear developer colleagues would need to go through to qualify as professionals seems a lot to ask. So there must be a context-specific choice in play on how to distribute the responsibilities over the skills we have with a nice stretch that grows our skills by taking us out of the comfort zone.

Developer skills and the need of testers

The past few weeks have lead me to increased focus on thinking about programmer-developer skills. Whatever I write about the topic does not fairly represent the multifaceted mix of skills and personalities in the real teams I reflect on, instead includes a fair bit of subjective emphasis to make some points I feel like making.  To start this, I need to mention that I absolutely love my colleagues and respect their contribution and all the positive surprises they come up with regularly, even if I feel frustrated on occasion.

I work in an organization with a long history of hiring programmer-developers. Hiring for this role is understandable, as there appears to be progress to be made with writing code. Surely to have the valuable features, a very practical transformation of ideas to code must happen. We also have had a very strong separation between the value of the idea into a concept or design without coding it and the actual coding of it into a group of product managers and programmer-developers. And product managers can test - if other work allows the time.

It turned out that focus this particular case was lacking is testing. Not the amount of it, but the quality of it. It seems that these groups did not end up with a working solution that delivers value but something that works if you don't use it for real-life scenarios it's intended for. It seems to me that a major contributor to the end result was separating the concept and design thinking from the 'just do what you're told' coding. To fix the situation, someone decided to hire a tester - but just one, as the limited budget is supposingly best used in people who write code. Later one became two, while the minimum need is four.

Just today, I opened a discussion about yet another feature that we would reimplement because it did not match the needs of the users. I listened to the arguments saying that it's not a developer job to question what the users would actually need, our job is to do what they ask and accept that they try again later - their organization pays for these mistakes that are theirs alone to make. I tried making a point that we had every chance of talking with the internal users before implementing and failed to question what was said, what we understood and what was actually needed. That discussion would be a core for us to improve as a team. Clean Coder by Robert Martin books puts it nicely: "It is the worst kind of unprofessional behavior to simply code from a spec without understanding why that spec makes sense to the business". Same goes for writing the spec or testing - any activity that requires you to think.

Another discussion I had recently was on the need of having more people who actually see how value is generated and notice issues that threaten that value. That discussion ended up with fixed budget of these people and the idea that one would need to leave before another, with a missing skillset, could join.

So I roughly categorized closest programmer-developers in the effort and need of testing they create:

• Architect-developers with experience. I love working with these types, and wish all devs were like this. They own the architectural design choices and strive to understand what is needed. And often are given a position that enables them to perform as in talking directly with the customers. Time taking responsibility over same product / similar solutions show in the end result. It doesn't always work, but the surprises that issues bring forth are considered in scale, not as individual symptoms. 
• Ones with potential. These types tend to be young ones that have not yet learned an unproductive role of giving up on improving. They might not know how to best learn, and try to deliver what was asked - obediently. As a tester with these types, there's often many surprising connections of features that you get to show. Instead of teaching these types to expect that from a tester, getting to their potential it may be important to teach them how to learn, together. 
• Ones without a system/value view. These types take what is asked and focus on implementing. If the product doesn't help with it's intended purpose, it must be someone else's problem. These types think it's normal to implement the same feature three times  just so that you don't have to talk with people with a different mindset. They accidentally waste a lot of effort but refuse to see better ways they could themselves contribute to. Testing for these developers is critical pre-implementation, to catch the expensive mistakes. And there's a fair share of post-implementation testing too but it appears the connections are not made and the code degenerates as fixing progresses.
• The sloppy ones. These types seem to be programmer-developers because they can write code, but not because they're good at it in the criteria of good I use. I tend to associate structural code within the object-oriented paradigm or the infamous 'whole program in the catch-clause of try-catch' -types of choices here. But the worst part is with fixing to hide symptoms - to generate more problems. As for testers, these guys make you feel needed as without you it never works. But testing here is just an odd choice if no learning happens. Perfect work generators for testers.

With people like this, 1:10 ratio of testers and developers seems off. Then again, fixing quality should perhaps start with making the unwilling willing again - stop limiting smart people with artificial role boundaries - and especially making the unable able. I'm sure people as smart as these developers would be capable of more. 

I could still use a little bit more of skilled testing in these teams. Empirical evidence is powerful also in organizing for the support people need to grow with developer skills.

I might also hope that the ones who are like taxi drivers who can't find even the sightseeing locations without a map reader (thanks to Michael Bolton on the metaphor)  would get paid significantly less. As it comes to pay, I find that some should pay for the trouble they cause instead of getting paid regardless of output that is executable code. Measuring value and contribution would be something I'd like to work on.

Nominated for the Finnish Tester of the Year -candidate

It's the time of the year when nominations for candidates for the Finnish Tester of the Year -award are out, and I learned I'm on the nominees list: for the 8th time. Yes, the award has been  given seven times before, and I've been nominated as candidate every year. And every year someone else has won.

The competition has a rule that you are ruled out of nominees if you have won. And I feel sometimes just a little frustrated on the post-vote comments that many people assumed I must have already won once, since I deserve to win.

I wanted to roughly translate what was said about me by an anonymous this year as basis of nominating me:

"Maaret Pyhäjärvi brings forth testing nationally and internationally. Last year e.g. in EuroSTAR program committee, as the architect of Helsinki Testing Day, organizer of several application development seminars, Agile Finland board etc etc etc. If activity was the measure, Pyhäjärvi would have deserved to win the title every year since 2007. This woman breathes testing!" 


That's beautifully said about me. I feel honored, even though I did ask all people close to me not to put me on the list again, I missed someone or this was written by someone who did not take my request seriously. And it was someone who put an effort into writing an updated description of what I've been up to.

The rules say that a Finnish Tester of the Year candidate is someone who:

• has inspired colleagues or other organizations in increasingly better testing
• brought fourth ideas and trends from the world into Finnish testing
• has positively impacted the birth of a testing culture in their own organization
• has influenced the results of testing activity (test coverage, issues found etc) in own organization or in the community
• has done test-related innovations, rationalizing improvements or created new ways of doing testing
• has had impact on the birth of testing as a profession in Finland
• has influenced finnish testing culture and tester profession development positively
• OR in other ways improved the abilities to do testing

You could vote for me to help me get away from this honor of being nominated every year: http://digiumenterprise.com/answer/?sid=1167464&chk=BQAVX7AV

Hands-on Testing is a Valuable Test Management Practice

In my previous post, I emphasized the non-testing activities I do to contribute in my team. Reading it through a day later, I realized it contributes to the idea that many people around me have on not understanding what testing is. When I emphasize the other stuff, I implicitly leave out the testing work other than bringing in the mindset of critical thinking and value-orientation.

I wanted to share a story of how it is important, in my experience, to do hands-on testing and not just hang out and contribute by talking and helping others.

At a point of my career, I was working in a customer-contractor setting in a multimillion project. I was assigned the role of a test manager, which meant a lot of meetings and a mix of planning and testing. As I was on the customer side, I had a tiny part-time acceptance test team to work with, on a huge and very complicated system. After all, this was the last bit of testing after all the contracted layers.

With reading the specifications and trying out a couple of scenarios, we learned that the system could possibly work as specified, but not fulfilling its purpose. It was a data processing system that gathered data from various external sources and put it together to sum up a financial decision, with very complex logic. If the financial decision was incorrect, the system was of little value.

As a test manager, I was at a choice point I did not realize I was at. I implicitly chose to talk with stakeholders more, to find a way to communicate the first lesson and with the context at hand and the structures of the contractor, it was a major effort. While I chose to invest my time on the communication aspect, I chose not to test so much personally. I could not be at two places at once. My tiny part-time team tested and I supported them, but as I was not testing myself, the hands-on testing effort was very small.

I remember many combat-like scenarios of discussions in the various committees with each party finetuning the arguments. But there was no common goal. The contractor goal was to deliver in schedule what had been specified. The fact that specification would be incorrect would not slow down that train that would pay them the money. And there was no better specification - as I learned later, acceptance testers needed to hands on test how the system would behave to know what they needed to specify. One experience in particular was such a scene I still laugh thinking about it: I had two colleagues in test kicked out of a board meeting with a minute warning time because they were employed by other contractors, on the premise of "business secrets" - just as we were working on an expensive decision that would not be as positive for the contractor if the test managers would be included. The politics drained a lot of energy.

Looking back at all that energy, it was wasted. With an explicit decision of myself testing - with little budget on testing, the hands-on empirical information would be more valuable than the politics game. The hands-on empirical info would have helped in transforming the politics into something we could actually act on. I know it would, since someone else, with a different resourcing model, had both parts, and the testing results part changed the game.

The lesson I learned is one that Elisabeth Hendrickson emphasizes with the phrase "Empirical evidence trumps speculation. Every. Single. Time". The real information from testing is important. Theories are only theories. Regardless of the test role I feel I'm assigned to, it's still testing, not theorizing about testing that could be done given enough resources. There's always a choice and it comes down to me making the choice.

I learned to choose to test. Hands-on, yet incomplete is better than the plan and communication. Testing to get actionable examples of things we should know of is valuable and game-changing information in the projects.


That's what I spend most of my time on. Knowing from personal experience what works and what not. All the other stuff I can contribute on in the project comes from that core: disciplined hands-on testing to transform theories into empirical evidence.