Orchestrating System Test Automation for Visual Thinkers

Working on a sizable system, feature after feature I found us struggling with timeliness of completing testing. Each feature kickoff, testerkind showed up to listen, and start learning. While development was ongoing, more learning through exploring took place. And in the end, after feature was completed, tests were documented in test automation, and just in case time taken for some more final exploring. 

It seemed like a solid recipe, but what I aspired for is finding a way where testing could walk more in sync with the development. The learning time was either half-attention, or elsewhere occupied, and it resembled a mini-waterfall for each feature. 

I formulated an experiment, with the intent of learning if being active with test automation implementation from the start would enable us to finish together. And so far, it is looking much better than  before.

The way I approached test automation implementation was something I had not done before. It felt like a thing to try, to move focus from testerkind listening and learning to actively contributing and designing. In preparation for the feature kickoff, I draw an image of points of control and visibility, tailored to the specific feature.

As I don't want to go revealing any specific of what we build, I drew a general purpose illustration of the idea. Each box on the main line were touch points where development for the feature would happen (pinks). Each box on the side was a touch point already existing in the system, found by analyzing the system for what was relevant combination for this feature (greens and oranges). I used two colors to categorize existing system touchpoint on closeness to the things we were developing (greens) and necessity for end to end perspective (oranges). Finally, for each touch point, I identified the type of handles: visibility and control, existing or missing. 

To make the abstract a little more specific, let me give you a few safe examples from my years in testing:

• ssh to a command line in a remote computer is a touch point of both control and visibility
• reading a file in filesystem is a touchpoint of visibility
• reading a system log is a touch point of visibility
• calling a REST API and verifying responses is a touch point of both visibility and control
• clicking a button on the UI and entering values is a touch point of control 

Same things can be both control and visibility. Making these choices is how I think about design for testability. And I prefer thinking it visually. 

Having agreed the touch points, we could do test automation a touch point at a time. Our understanding of what touch points that were not about to change we could already work on, and what touch points were depending on the changes. We could talk about how order of development enabled testing. And most importantly, we could talk about which touch points I identified did not make sense for the system scale, because we could address risks on unit and component tests. 

The current popular thinking in the testing community is to paraphrase testability into something wider than visibility and control. This little exercise reminded me that I can drive a better collaboration test automation first, without losing any of exploratory testing aspects - quite contrary. This seemed to turn the "exploring to learn randomly while waiting" into a little more purposeful activity, where learning was still taking place. 

In a few weeks, I will know if the original aspiration of starting together actively to finish together will see a positive indication all the way to the end. But it looks good enough to justify sharing what I tried. 

Stop paying users, start paying testers

At work when I find a bug, I'm lucky. I follow through during my work hours, help with fixing it, address side effects, all the works. That's work.

When I'm off work, I use software. I guess it is hard not to these days. And a lot of the software recently makes my life miserable in the sense that I'm already busy doing interesting things in life, and yet it has the audacity of blocking me from my good intentions of minding my business.

Last Friday, I was enjoying the afterglow of of winning a major award, bringing people to my profile and my book, only to learn that my books had vanished from LeanPub. One the very day I was more likely to reach new audiences, LeanPub had taken them down! 

After a full excruciating day of thinking what was it that I did wrong to have my account suspended for authorship, LeanPub in Canada woke up to tell me that I had, unfortunately, run into a "rare bug". Next day I had my books back, and a more detailed explanation of the conditions of the bug. 

If I felt like wasting more of my time, I guess I could go about trying to make a case for financial losses.

• It took time of my busy day to figure out how to report the bug (not making it easy...) 
• It caused significant emotional distress with the history of one book taken down in a dispute the claimant was not willing to take to court
• It most likely resulted in lost sales

But overall, I call my losses, and turn it into a sharable story. Issues with software are something to pay attention to, and I can buy into the idea of this being rare in the sense that it was probably hard to catch and they were quick to react on it. 

Next day, I decide its time to buy a new domain name, and I go to Hover, where all my services are hosted in a nice bundle. I own a dozen domains. That is how I start new projects. Sometimes, the annual process of deciding if I want to still pay for a domain is my best way to manage hobby projects. 

I try to log in. And I fail with a 404 error. Like a good user, I refresh, and continue my day without reporting. I don't want to spend time on this, I just want to get things done without all these hiccups. 

These two issues are functional bugs, meaning bugs that inconvenience the user, one more significantly than the other. They have very little impact on the company, and the better companies are like LeanPub, making the bug go away fast when it is actually blocking. 

There is a third functional bug that I stumbled upon, in the very same way as all these other functional bugs. But this functional bug is what they call "exploitable". That basically means that it can hurt the company, not just the user. Sometimes exploitable bugs are about money, like this one was. Sometimes exploitable bugs are about secrets, like Vastaamo problem was. Exploitable bugs are of special interest to folks in security, and they are bugs with a particular priority to folks in software testing. 

Just like with the two other bugs, I was minding my own business, not testing in anyone's production. I wanted to buy food delivered, and I used the Foodora app. 

First time I saw this bug, the symptoms were subtle to me. I received two confirmation emails. I received a phone call from the pizzeria I ordered from asking if I ordered two meals. I had a pleasant conversation with the pizzeria guy about this happening often that orders were duplicated, as pleasant as a conversation I don't want to be in can be. And I received one meal, just what I ordered. Because while I was a user, I'm also a tester and a particularly observant one, I knew what I had done to trigger the condition. I was not planning on doing it again. So I tweeted a wide characterization with no repro steps to vent my frustration on software getting in my way. 

Ordering takeaway food as much as I do, it was only a matter of time before it happened again. This time I had my whole extended family over, and it was a fairly large order. Yet this different restaurant apparently did not know the issue, and delivered two full sets of food at my doorstep. With yet another inconvenient discussion with someone I did not want to meet at my doorstep, the driver took the other set of food away with him. I was inconvenienced, more than the previous time. I tweeted about Foodora needing to fix their bug, and searched their web page to contact with no luck. So I let it be. Someone asked me for steps to repro, alerting me to the idea that I might have to waste my time on this eventually. 

Third time was the charm. I still don't want to reproduce this bug, and despite my best efforts, it happened AGAIN. Already upset at the idea of having to have the conversation of what I ordered and didn't, I just took the food they delivered and complained they delivered me two sets of my order billing only one. With the proof of order numbers and payments, I assumed they could track it and get it fixed. Their loss, they should care. I mentioned that if they did see my feedback and needed help, I would be happy to tell tell them exactly what causes it. I didn't feel like writing long step-wise instructions into the void. I spent more of my energy in establishing credibility in knowing that this is a bug than on anything else. Because, that is what testers find they need to do when they report. Dismissing feedback is regular. 

Weeks passed, they contacted me. I wrote them the steps. Weeks passed, they contacted me. They said thank you, that they had the issue addressed with their tech folks, and gave me, the user, 10 euros. 

I write a shorter version of the story on my LinkedIn wall, only to learn how the cyber community (some outskirts of it, at least), misrepresents what I say:

• Someone claimed I was "testing in production" because through my profession, I couldn't be a user. 
• Someone claimed I was "testing without consent" because I wasn't part of a bug bounty in finding this
• Someone claimed that I was breaking the law using the software with a vulnerability hitting the bug
• Someone claimed I was blackmailing Foodora on the bug I had already reported, for free, expressing to them I was not doing this for money in our communication 
• Someone claimed I was criminally getting financial benefit of the bug
• Someone claimed the company could sue me, their user, for libel in telling they had a bug
• Someone claimed I was upset they did not pay me more, not on the fact they didn't pay a competent tester in the first place (I know how to get to the bug, I would have found it working for them - exploratory testers couldn't avoid it, automation only strategy or test cases could)
• Someone claimed I was eating on the company's expense, when I was reporting on 200 euros of losses (for food they had thrown away) 

You may already sense that I think these claims are poppycock. But these types of claims are very typical for anyone with a larger follower base - I'm great platform for both security marketing effort but apparently also restaurants telling me they still want me as their customer. 

Today I listened to Turvakäräjät podcast, only to learn I made appearance as a "reputable tester" on this security podcast in Finnish. They really got their alternative facts straight: eating on others' expense (not that I had to pay for every order I made, there was no free meal ever, only duplicate to go to waste) and being upset over 10 euros bounty. 

Let's reiterate this one more time. 

There is no money in the world that I would be willing to sell my free time on to test their software with a zero sum contract that only pays when I find problems. I think bug bounty programs and crowdsourcing are unethical, and would not take part in results-only payment models. I struggle with ethics of companies like Foodora that don't treat their drivers as employees, but I use them because I have nothing better in times I don't want to leave home for food. My ethics are important to me. 

There is no money in the world that would compensate stealing my time on bugs, exploitable or not. The bugs are not the users responsibility, even if sometimes users hands are tied, and waiting or leaving are not real options.

There is no ethical obligation for me to report even exploitable bugs as a user. The ethical obligation I take on me is not to share a vulnerability further. Knowing impact does not mean knowing the mechanism. And I write about the impact vaguely to protect the buggy party. I delay big visible posts to times when they won't have extra harm on my account. 

I say don't pay the user, paying users 10 euros is an insult to the users. Hire a professional, and note that the professionals don't work for peanuts. 

Asking Your Users Perception

A colleague was working on a new type of application, one that did not exist before. The team scratched together a pretty but quick prototype, a true MVP (minimum viable product) and started testing that on real users. 

Every user they gave the app to, gave the same feedback on a detail they did not like after first experience of use. 

Fixing the thing would take a while, but since the feedback was so unanimous, the team got to work. A week later, they delivered a new version. 

Every user that had the app before, hated that there was a change, they had grown to like the way it was. 

Every user they added for first time experience, hated how it was different from the usual way things are done. For the first day, until they learn.

I shared this story to my team at work this morning, as my team was wondering if we should immediately put back the "in review" state to our work board in Jira. The developers were so used to calling "in review" their done, but also refusing to call it their done and moving it to the done column. They used to leave tickets to this in review column, and no one other than themselves was finding any value on it. 

We agreed to give the change two weeks before going back to reflect on it with the idea that we might want to put it back. 

Users of things will feel differently when they are still adjusting to change, and when they have adjusted to change. Your needs of designing things may be to ease getting started, or ease the continued use. Designing is complicated. Pay attention. 

The One Thing That Turns Your Testing To Exploratory

I worked with a product owner, who liked seeing all of us in the team put down very clear tasks we are doing, with estimates of how long does it take to complete it. Testing never fit that idea too well. 

Some tasks we do, until we are done with them. 

Some tasks we do, until we've used time we're willing to invest. 

Some tasks, through learning become very different than what they originally were and writing it down may get you doing the wrong (planned) task, instead of the right (emergent) task.

Test cases and manager saying things like "I expect to see 20 test cases completed every work day", or "add one case to automation every day" give testing the feel that instead of learning and exploring, we're squeezed on time.

Thinking to the explorers of territories, finding something new and interesting was probably not likely if the orders were to take the main road, stick to it and make it to destination in an optimized schedule.

The one thing that turns your testing to exploratory testing is time. Take time, and think about using time time in a smart way. You won't know all things in advance. But without time, you wouldn't be able to be open to learn. 

This insight comes from last Friday and me trying to get to a conference stage. I left on time, and turns on the way lead me to a completely different talk than I intended to give - the one I intended to give would not have worked out for that stage. 

Things that can only happen when isolating...
I was about to leave for downtown to record a talk for #MimmitKoodaa. As I stepped out the door, I realized we have snow. I had no winter tires. Wait, it gets better (1/n).

— Maaret Pyhäjärvi (@maaretp) November 20, 2020

What Exploratory Testing Is Depends on Who You Are

Size of the software industry double every five years, says an old folklore. I don't have the energy to go fact-checking, but whether the numbers are correct or not, I definitely have the feeling that in scale, half of my brilliant colleagues have less than five years of experience. 

Less than five years of experience, provided that we use every single year in learning more and more about our applications, our methods and our industry, means just that - less experience than what I have nowadays. 

Depth of experience shows up as connections

Today, we took part in a #MimmitKoodaa Not a Webinar, a virtual event broadcasted from a studio downtown. As I was watching the event, two other colleagues were also watching. One of them pinged me to say that they didn't enjoy the emojis filling up the chat window, the chat was full of interesting people explaining their ideas and experiences and a mass of clapping emojis made it harder to see the relevant. "Someone should code a feature that separates discussion and reactions" - they explained. As a more seasoned online conference goer, I gave a few reference platforms that did exactly that. The third, most new of us, had not yet reached a feeling strong enough for them to act and express a will. 

My first colleague has decades of building a professional general-purpose self-confidence that gave them the platform of expressing the thing they missed. 

I had experience in using that elsewhere, and could compare to what I knew already existed. 

Our third colleague is new, and would not have raised that discussion out of a session of testing. 

The discussion was great. It lead us to understand, yet again, that skills are multifaceted. In testing, we work on many layers, all invisible, at the same time. We think of what is there, and what could be. We think of technical and business at the same time. We make connections in the moment, and with the reflections we do together. 

What This Has To Do With Exploratory Testing?

Some of us do exploratory testing with less experience. All of us do exploratory testing with different experience. Instead of thinking of Exploratory Testing as one thing, it varies based on who we are today, as we are doing it. 

When I started off as a new exploratory tester, I had not yet found my feet as a polyglot programmer that I was. Somehow I managed to get through multiple completed programming courses avoiding any glimpse of programmer identity. I could code, but I could choose not to. It wasn't even hard to find other relevant skills I needed to strengthen. I put significant effort in my observation skills, and found great joy this week in our 15-yo new colleague in loan tell our common manager that I was particularly good at spotting things based on us pairing. I put significant effort in my creativity around test ideas. I put significant effort in my skills on self-managing a learning centered activity.  

Five years ago, at 20 years of my career, Ensemble Programming woke up the hibernating polyglot programmer in me, and I have been writing code, and automation code since. The exploratory testing I now do looks very different than the exploratory testing I did before.  

Now I explore the application to learn about it, how to test it, and what to pay attention to. 

Now I document my lessons as I go adding test automation that runs for me for any and all changes. 

Now as I attempt to document as automation I end up looking into things with detail, like it was a magnifying glass to anything I used to test. 

Now I create little scripts to do testing I want to do faster, and I throw it away after using it. 

Now I write test automation scripts, and try them with different values.

Now the existing, running test automation calls me in to see when things change. 

I still do exploratory testing. All the work I do is exploratory testing. I just use automation within it, because I have the skills to do both. 

But everyone doesn't have my years of experience. So I've started figuring out how early in ones career can we teach people to have both skills. It turns out, first week of work is just fine for it. Having actionable knowledge on both testing and programming for testing purposes grows a day at a time. We need space for both. Sometimes we choose to speed things up for teams by splitting the learning that must happen to two different people. 

If we have someone who automates but does not think like a tester, they alone do bad at testing. If we have someone who goes search for learning and problems, but don't automate, they are soon overwhelmed in a devops continuous delivery type of an environment. But if we have both, and they collaborate, that works well. 

Testing all in all is a huge field. Early access to programming skills, in my experience now, accelerates the people rather than provide a threat to having eyes on the application.

But framing this as exploratory testing - with automationist's gambit - helps. Automation is a part of winning the game, but so is creativity, dept in test analysis, finding the right ideas, observing, operating and putting this all together so that people doing it are not overwhelmed with having to do too much, too soon, too alone. 

Let's face it: of all types of programming, programming for testing is specialized towards some of the simpler learning needs first. Putting this programming together with understanding that makes it active testing is important. Being a good test automation programmer with their own ideas to implement takes a lot of work. I call that growing into Exploratory Testing. 

Canvassing for improvement experiments with a canvas

If all the world is open to you, why do things like they were always done? Throughout my career, I have held one heuristic above all others for exploratory testing: 

Never be bored.

Obviously, I have been bored. But I have never accepted it as a given, or a thing I cannot change with internal locus of control. I can't change the world around me (always), but I can change how I look at the world. And I choose to look at it with creativity and playfulness, and being just enough of a rebel that things can be different. 

With enough rehearsing on ideation on both how I would test and how I would frame the system I test in (not software system, the system of people in organizations), I've collected ideas that I can combine into new. I find myself being one of those people who show up to a retrospective with ideas of what we could try, and with increasing comfortability in the idea that to learn the most, we should have half of our ideas be too far off so that they should fail, and other half take us further through successes. 

I'm quite happy with some of the experiments that stuck with us. The "no product owner". The "vanishing backlogs". The "start from automation". The "zero bug policy". The "wait for pull". The "stop reporting bugs". The "rotating responsibility". The "power of network". I could go on listing things personal, team and organization, and notice how much more I enjoy telling the stories of successes. Let's try X could become a personal mantra.

This week, I decided to experiment on how I share my experiments. I'd love to see people propose ideas from left and right, but while that isn't happening, I can model that through what I do. I wanted a visual describing what change goes on with me. The visual - a improvement experiment canvas - is a tool for canvassing - seeking support, asking people for opinions on things that are easily invisible. 

I ended up creating this: 

Improvement Experiment Canvas

I fill it in using Mural, but also have a pdf version I already passed forward for people without the tool. 

The fields explain themselves well enough, but the little icons don't. Criticality is about how important this hypothesis is for our success. Timeframe required is about time required for the experiment, from days to weeks to months. Test cost is about how much effort doing it will take. Data reliability is about generalizability of results. 

No product owner experiment was months, and we could not do all the follow-up learning in days or weeks. It didn't take us any extra effort, just a mind switch. Data reliability in the organization was anecdotal, and we did not need it to be more. 

I also consider two other templates. A very straightforward one: 

And one based on Toyota Kata, where focus is more on themes rather than individual experiments. 

The first two experiments I described this week. One of them is duration of a few weeks, from start to finish of a feature. Another is one month, to see if a major change in how we track our work feels good or not. Defer commitment. Defer scale. First learn. And learn continuously. 

Automationist's Gambit

Last few weeks, my twitter feed has seen increasing amount of people recommending a series on Netflix called Queen's Gambit. The series is awesome, highly recommended, and focuses on two things: substance abuse and chess. My inspiration for testing comes from the latter.

Watching the series and doing a bit of research, I learned that queen's gambit is a chess opening: one of the first moves a chess player starting with white can open with. queen's gambit is popular, has attacking prowess, and puts pressure on opponent to defend correctly. It offers a pawn as sacrifice for control of the center of the board, and the sacrifice is made in search of a more advantageous position. 

Gambit - a game opening - is a word I was blissfully unaware of until I watched the series last weekend.

You may already guess, my interest in chess is superficial. My interest lies with exploratory testing. With the conference season at it's peak, my twitter timeline also displays anti-automation takes from prominent people in testing. Talking about a dichotomy of exploratory testing and automation is anti-automation. Because really, the only reason these two are sometimes considered separate is (lack of) skill. Skill in testing (for the programmers) and skill in programming (for the non-programmers). Skill limits what openings for the collaborative game of testing in software development you have available. 

So I call for a new opening move for exploratory testing: the automationist's gambit. Learn every day about both testing and programming, and treat them as mutually supportive activities embedded in the same, growing individual no matter how many testing years they have under their belt. 

You can't automate well without exploring. While you create that code, you will look at what goes on around you and acquire understanding, and report things you consider wrong. 

You can't explore well without automating. If you don't have automation to document with, to extend your reach with and then throw away to avoid maintaining it, your reach is limited. 

Failing test automation in an invitation to explore. Exploring provides you ideas of what to document in your automation, or where to reach that you could not by hand. Automation serves as a magnifying glass for the details you need to check to know what works and what only appears to work. Automation serves as the web of the spider to call  you in to see what it caught this time. 

Exploring without programming is common but it is not the only way. So I offer you consideration of learning the automationist's gambit. Test automation makes you a stronger explorer. And this option is available for those who are just starting their careers just as much as it is available for seasoned professionals. Try using the anti-automation energy to learning the skill you are missing and see if useful automation has an increased chance for emerging. It has for me. 

Automationist's gambit is like queen's gambit. It's not about becoming an automator. It is about enabling the automator by sacrificing something that is on the way: your need to warn people that automation isn't all encompassing. Trust me, the managers already know and you are not helping them or yourselves presenting as anti-automation person.

Doing is one thing. Doing together is another. I'm all for collaboration skill that enables us in performing the automationist's gambit as a pair that is stronger than an individual could be.