An evening eye-opener

For Saturday fun, I'm reading Gojko Adzic's book Spefication by Example. Not for the first time, but I thought this time I'd try read it in order intended, instead of the usual jumping to the most relevant first.

On the very beginning of the book, describing key benefits of Specification by Example, there's a quote. Paraphrasing: In other organizations in the end testers find something wrong with the product and developers need to continue, and in ours they don't. The issues that the team have are about making a test (automated) go green, not later feedback surprises.

I got stuck on this particular out-of-context quote, namely because it, unintentionally, resonates with something I'm experiencing. Not specification by example and hitting the right target, but being in an organization where the issues used to be "tests not becoming green" and now there's definitely more churn - going back and forth.

To illustrate my story, I'm sharing a picture from our Jira that I've been looking at recently, trying to figure out what to do next.

Something happened a while back. Something that makes our software development look a lot less effective. They got their first tester that focuses on testing. They had at some point before a tester who focused on test automation, and with that, the wrong kind of test automation, so much of what I'm showing come as surprises.

It's not that you couldn't use the product. Apparently you can, since there's customers using it. And apparently they can't be very unhappy or they would complain in ways our people would understand. I'm not quite sure what's happening there.

But I know what is happening testing-wise. I started testing and logging my issues a bit before out latest release. And the stuff I'm finding for addressing is more than the fixing / development work of our whole team.  The problems are not about missing requirements per se, but about surprising side effects of two supposedly separate things. They are problems of shortsightedness about things that can go wrong and believing you'd be luckier than the others. And not knowing hides it all.

I had organization, who could have no exploratory testing and no churn. No real information either. Having e.g. me work on anything like specification by example & test automation now would seem like the wrong choice, as they need quick eyes open effectively reported -testing first. But I still kind of hope to get to that too, to turn the trend around for better.

Why would we improve the way we work?

I had a short, but inspiring discussion at work today with some of my team's developers. Since it's apparent someone for some reason wants things to improve, they were asking for set goals and better understanding of where (and why) we're heading. Apparent for the reason that they just recruited me, but still not so clear since while they may now invest a little on "tester", they still haven't invested visibly into fixing.

I gave a short answer as I see it: there's a tradeoff in the way they develop. When we deliver "more short-term business value", we take "quality/technical debt". We're approaching a point where the debt and the interest of the debt are threatening our ability to deliver features, and we are unwilling to scale the team size but look for other solutions. Our challenge is not about us wanting to do things in ways that make sense, but about the effort and timeframe we need to invest to get where we want to be.

Another one of the developers who wasn't part of the discussion as it started, suggested from his cubicle that this could actually be what we'll clarify and discuss with the product management team. Didn't test that yet - will soon. This is a very simple and basic approach, but today reminded me how many times it has been effective. And most likely, I will learn more layers on what we actually find relevant for our product to become a better tester for it when talking about this.
 

If it looks too easy to be missed by developers...

On my first weeks at new job, I've had the pleasure of reporting bugs again. I find this particular result of testing to give me the feeling of achievement. The more relevant the problem, better.

There was one bug I reported on Monday, that just looked too easy to be missed by the developers in my team. As I originally reported it, the problem was that when logging in with one of our three main browsers, there's a highly visible error message. And that this seems to happen only with the recent builds, not in production version.

In the end of the week, I quickly asked in passing the developer whose component was failing, if a fix is available in the next weekly build. He seemed puzzled: what problem, what fix? I checked our Jira, and the issue had not been addressed - which is quite normal. He took a quick look at it, and came back with "I didn't change this for ages" with some details.

I started testing the issue more with the information from him. With fresh eyes, I realized I entered the program from a bookmarked link - something I hadn't mentioned in my original report. I also realized, that I had different addresses bookmarked in the other browsers. So I had missed a relevant bit of info I provided now.

Bottom line: if it looks too easy to be missed by developers, it may be that they didn't test, but in this case, I missed relevant factors that are needed to get the bug visible.  Talking sooner than later to the very busy developers is still a good idea.

New product, new team, new practices

For a bit over a week now, I've been wondering where I ended up in my quest for hands-on testing work. With hard choices on the way, I'm now working for Granlund, a civil engineering company, with a product that handles building-related data. The domain is something I have little idea as of before, and I'm looking forward to learning a lot on that in addition to tuning and changing whatever I can with my testing skills. We have a small team of less than 10 people, and I'm the first and only tester. Most of my colleagues in development seem to work remotely, but within a week, I've had a change of learning they're just as much fun to work with as I expected.

I start off with a redesigned version of a product that has been around for quite a while. The redesigned version is also out in production, new versions of it going out once a month. With customers actually paying for the product, they must be doing something right, even if they never had testers around.

After reading up on what the product is about with a shallow scan of its documentation, I've worked on:

• Setting up test management based on sessions with Rapid Reporter and csv-note scanning tool to show metrics I will create - as I won't create test case counts
• Learning the product's capabilities (and quality) with doing exploratory testing on its main feature areas
• Existing test suites and redesign of test documentation
• Redesigning a consultant-suggested testing methodology that I just can't believe would provide added value (unless faking testing is considered that to someone I did not yet meet there)

There's two strong first impressions:

1. I've got a likely case of "asking for testing, when you should ask for fixing" ahead of me
   I find it somewhat funny that so many people equate testing (quality-related information service) with fixing (making the quality right) and don't think of the dynamics that the added information will bring in. Then again, understanding the business impact and meaning of the existing technically oriented issues is a service I think I can help with. 
2. As there's not enough rational testing examples around, it's easy to take the basic book on what's a test case and try replicating it without thinking enough
   I've enjoyed reading the attempts to design tests in per-feature-area test suites of varying sizes, but all with step-by-step test cases repeating most of the steps again and again. I took on of these documents, 39 pages with 46 documented test cases and read it through in detail to make a mindmap of mentioned features (I do need a feature list to support my testing). While reading and using the product for learning it in practice (a couple of 1,5 hour sessions), I came up with one-page mindmap mentioning 88 things to test and four dimensions that cause repetition on significant amount of testing that should happen, like different browsers, user rights, and such. Out of the 39 pages, came out 3 things I could not directly deduct from the user interface with little information on the actual product. While doing this stuff, I marked down some (quite many) issues I would write bug reports on - if it wasn't the area we're about to rework on in a significant manner right about now. 

Looking forward to all this - and the chances it provides for writing stuff and providing examples of something that is doable to "just a tester".

Looking for a new job

I feel excited and puzzled - I'm actively looking for a new job. I've spent last 2,5 years at Ilmarinen, and enjoyed the lessons learned there greatly. We've succeeded, with my projects and teams, to get projects, where the contractor does their share on quality-related work and acceptance testing phase is actually just that: testing, not fixing. We have great business specialists in acceptance testing, they've learned something relevant about testing with me and work with contractors is less cumbersome than before.

So why I'm changing: because I watch out contractors to get to do the stuff I'd want to do with the limited time I have for work. They actually create the quality, their testers enhance the productivity of the teams they work in. They do testing for finding problems so that customers like us don't have to. And I want to be doing that: work close to development, actively making a difference in quality and productivity for the company I work for, and the end users of the products our teams are creating.

So I'm looking for a senior testing specialist position, that would enable me to do hands-on testing work. I'd love to take a spin at test automation, from the extending exploratory (not manual, but brain-engaged!) testing point of view, doing more and using ideas that are not possible without tools. With test automation, however, I know I'd be stronger with a team where I could provide practical ideas of what we'd test, instead of trying to do that by myself alone. The reason is that I have developed a practical way of developing and grouping ideas, and focusing on automation alone may prove to be difficult as the list of stuff that would need to be covered may take more time than first assumed.

My ideal position would be:

• working with a system or product with relevance: either successfull business-wise or relevant purpose, like with Ilmarinen: there's no pension paid without the success of the chain of systems that make it happen
• working with a development organization: either product company or company providing software development projects on contract. Essentially, working with people who create the software. 
• enabling me to do hands-on exploratory testing, develop exploratory testing related practices on smaller and larger scale.

I'm considering both work in Finland and relocating somewhere in the world. But, since life is short and I really want to do more of hands-on testing work than I can in my current position, I would prefer to move quickly. I can work either as a self-employed contractor or as employee. If you happen to know anyone I should talk to, give me a hint.

A tester at heart, always learning - that's what testing is about. So I think I've gotten pretty good at learning to be useful quickly.

Manual Testing Radiator

Two days ago Finnish Association for Software Testing - that I just happen to be the chairman for - had an Exploratory Testing Dinner. The idea is to eat and talk testing, with some volunteering to talk on a topic, and then let others join in.

One of the three talks this time was by Petri Kuikka from F-Secure. He's my testing idol, someone I respect and admire, having worked with him for a few years at F-Secure. He shared a story on a manual testing radiator, that I just need to share with others even before they manage to put the system open source as they've planned.

In the years I worked at F-Secure, I brought in borrowed idea of a testing dashboard, adapted from James Bach's materials. I introduced the idea of area architecture based reporting on two dimensions, Quality and Coverage. We used traffic lights for the first, and numbers 0-3 for the latter. In some projects, I used a third entity, best-before commitment, which basically was an arrow showing if we'd need to say I'll change my mind about quality due to changes by tomorrow / a little longer / relatively stable. To collect the values, I used to collect people into rooms for thumbs up / down voting, making the other's opinions visible and just learning to report feelings based data in a consistent way. I learned that power of a crowd outweights any metrics-based reporting in delivering the message.

The problem that I never tackled but Petri and other current F-Secure employees have tackled, is keeping this style of reporting up to date, when you have continuous integration, with changes introduced from many teams, for areas separate and common.

For the continuous integration build system, they have a status radiator for automated tests coming directly from the tool of their choice. If a build fails, the visual radiator shows which part is failing, and when all is well (as per test automation being able to notice), the whole picture of parts remains blue.

The manual testing radiator does the same, but with a facebook-like thumb-voting icons. If a tester clicks thumbs up, that's counted as a positive vote. Positive votes remain valid only for a limited timeframe, as with continuous changes, if you did not test an area for a while, it's likely that you really know little of the area anymore. When several people vote thumbs up, the counts are summed and showed. Similarly, there's possibility to cast to kinds of negative votes. With the thumbs down icon, the tester gets to choose if the status is yellow ("talk before proceeding, concerns, bug report ID") or red ("significant problem, here's the bug report ID") . If a negative vote, even one, is cast, the positive votes counting goes back to zero. All the votes are saved in a database, this is just the visible part of it in the radiator.

Another type of a radiator they use is showing progress on statuses on regular intervals. Like in battleship game, they mark down where they feel they are with product releases and end-of-sprint versions. This is to show that not tested does not mean not working, but it means we really don't  know. With scrum and automation, the amount of exploratory testing is still significant and guided by time considerations.

Petri was talking about having the automated and manual testing radiators, on visible locations at office, side by side. I would imagine that causes positive buzz towards the results that exploratory testing can deliver, that are hard for the automation - assuming that there's stuff that exploration finds on areas that automation claims that is working fine.

Falling into a trap - talking of Exploratory Testing

I attended the public defense of dissertation for Juha Itkonen today. His doctoral dissertation with the title "Empirical studies on exploratory software testing" is a topic particularly close to my heart, since it was the topic I started with but failed to continue striving for in the academic sense. Juha started in the same research group after me, took the relevant topic and made all the effort to research and publish what he could in the limited timeframe (of nearly 10 years).

When Juha's opponent started, he went first for the definition of exploratory testing and distinguising exploratory testing and non-exploratory testing. This, I find, is a trap, and an easy one to fall into.

Juha explained some of the basic aspects of what he has summarized of exploratory testing, but struggled to make the difference of exploratory vs. non-exploratory. He pointed out that learning is essential in exploratory testing, and changing direction based on learning. And the the opponent pointed out that typically people who test based on test cases also learn and add new tests as needed based on what they learned. Juha went on explaining that if you automate a script and remove the human aspect, that is clearly not exploratory. And the opponent pointed out that a human could look at the results and learn.

What Juha did not address in his defense, or in the papers I browsed through, is the non-linear order of the tasks. He still works on the older definition of simultaneous activities, and thus, in my opinion, misses a point of tester being in control what happens first, after, again and for how long each of the activities endure to achieve a goal.

A relevant challenge (to me) is that all testing is exploratory. There's always a degree of freedom, tester's control and learning when you do testing. Trying to make an experienced tester follow a script, even if such existed, is not something that happens - they add, remove, repeat and do whatever needs to be done, including hiding the fact that they explore. All real-world approaches are exploratory testing to a degree.

I still have a problem to solve: if all testing is exploratory, what words should I use to describe the wasteful, document-oriented approaches, that take a lot of effort for creating little value. There's a lot of real-world examples where we plan too much, what we plan is the wrong things as we do the planning at time we know the least. And what's even worse, due to be belief in planning, we reserve too little time for the hands-on testing work and related learning, and with the schedule pressure, fail to learn even when the opportunity supposingly is there.

Low quality exploratory testing is still exploratory testing. Getting to the value comes from skill. And with skill, you are likely to get better results with some planning and enough exploration-while-testing, than skipping the planning. And yet with skill, you could use too much time on planning and preparing, due to logistics of how the software arrives to you.

Ending with the idea from the opponent for today's dissertation: the great explorers (Amundsen - south pole / example coming from a norwegian professor) planned a lot. What made the difference between those that were successful and those that were not, was not whether they planned or not, but that they planned holistically, thinking of all kinds of aspects, and keeping their minds open for learning while at it - making better choices for the situation at hand and not sticking to the plan like carrying back stone samples at cost of one's life.