Wednesday, October 28, 2020

Staying with Problems Longer

 One of my favorite quotes is:

It's not that I'm so smart, I just stay with the problems longer. --Albert Einstein

This idea has been central over the years as I have worked on understanding what it is that we call "Exploratory Testing". It's not a haphazard smart clicking thing, but it is learning deeply while testing.

We do our best work in Exploratory Testing when we stay with the problems longer. When we follow through things others let go, and investigate. When we dig in deeper until we understand what those symptoms mean for our projects. 

Luck - or rather serendipity, lucky accident - does play a role in testing. As we spend time focusing on coverage, we give our systems a chance of revealing information we did not plan for. Sometimes, but rarely, that happens as soon as we get started. Sometimes, but rarely, that information is so valuable it is worth our entire year's salary. That's a myth, and this post is motivated by a colleague who is using that myth to sell a training course I now categorize as snake oil. 

In a profession where we seek what is real and what is not through empirical evidence, falling into promises like "I teach you how to find a bug worth your entire year's salary in an hour" are just as bad as the excessive promises tool vendors do on test automation. 

In words of a super-successful golfer:

The more I practice the luckier I get -- Arnold Palmer 

Spend time to be lucky in Exploratory Testing. A gamble-based approach to exploratory testing is off and encourages the idea that thoroughness isn't the goal. Serendipity is real, and it becomes real by staying with the application longer and using test automation to extend your reach in all kinds of directions. 

And yes, I too have found a significant bug in an hour after joining a project. That says NOTHING about me and my abilities as an exploratory tester, and a lot about the project. Let's change projects so that they make us need to be a lot less heroic and a lot more investigative. 

Tuesday, October 27, 2020

The Vuln, the Breach and the Ransom

 A week ago, I was reading the news in Finland to learn that a major psychotherapy service provider, Vastaamo, had received a ransom note from someone in possession with their patient database. I could guess I would soon find myself a victim, and a few days later on Thursday, that's exactly what I was told. The event unfolded some more when on Saturday I, like apparently tens of thousands of others, received a marketing-style personalized ransom email asking me to pay. 

I'm lucky - whatever discussions I have had there have already seen the social media and just filing in a crime report on the ransom was a no-brainer. 

My first reaction was to be upset with Vastaamo for doing a crappy job protecting our information, as the criminal's messages implied that the reason they had the information was that the database was left online, with root:root access credentials. An open door, yes, but not an excuse for stealing something private, and even less of an excuse to blackmail folks. 

My fascination for this case comes from being a professional tester. With the 25 years of working, I have been a part of reporting and getting fixed hundreds, most likely thousands of vulnerabilities. Even the problem of weak password for relevant data in production, there's been more of that than I care to count. There's been protecting admin interfaces by thinking a secret address that we only know would protect it. There's been great plans of security controls, that turn out to be just plans but not turned reality. Well planned is not half done, it isn't even started. 

Bad protection shouldn't happen, and I would love to say you need folks like myself, aware of the issues around security and keen to follow through to practice to not leak through something this stupid. I even made the claim that this is level of protection for *health records* is against the law as I filed that complaint last Thursday on Vastaamo. But bad protection happens, and all it takes is, like the now-fired-ceo of Vastaamo claims, a human error. And perhaps, deprioritizing work that would cover at least the basics of security controls. 

As time passes and news unfolds, my focus turned on my annoyance on how the news reports on when the company knows their data was stolen. 

We need to separate, on a timeline, a few concepts. 

  • The Vulnerability is the open or insufficiently locked door 
  • The Breach is the moment someone walked through that door
  • The Ransom is the moment when they used the data illegally in their possession for further steps

Separating these three, we can collect statements of what we know.
  • The vulnerability was fixed in March 2019 and this is how they know data after that haven't leaked (for this particular incident)
  • You can't fix a thing you don't know is broken. So they know of the vulnerability even if they don't know of the breach. 
  • The ransom requests were reported to police in September 2019 and this is how we know when the company knew they had been breached for a fact. 
  • The breach could have happened any time the vulnerability was there and we have been given two points of when the data was accessed. We are told the latter is something the company figured out in their security audit activities (which lead to fixing the vulnerability). We don't know if the company knew of November 2018 breach before the ransom request. 
The timeline of these will become very important for the CEO of Vastaamo, as the new owner is interested in whether they were sold a company knowing the breach. But knowing a vulnerability is not knowing a breach. They are separate and we just don't know yet. 

With the hundreds or thousands of vulnerabilities I have been part of, the number where I am aware of a breach is less than one hands fingers. Sometimes we don't know because knowing requires going back and analyzing. Sometimes we don't have the data to analyze, but more often we end up looking into future. Similarly, with the hundreds or thousands of vulnerabilities,  I can still cope with my fingers on calculating how many times we have told we had a vulnerability that we fixed to our customers. 

We find vulnerabilities through analysis and testing.
We learn of breaches through logs monitoring use and contacts. 
We tell of vulnerabilities to customers when we have identified they were almost certainly breached, and most certainly now protected. 
We fix vulnerabilities in secret to not invite more breaches. 
I don't like that the news are passing such one-sided perspective on an upcoming court case on the Vastaamo CEO that will define timing of the vuln, the breach and the ransom. Knowing one is not knowing the other. 

Monday, October 19, 2020

Fix by symptoms, fix by causes

Working with a new tester is both refreshing and inspiring as they go through things I've been through and where my seasoned nature makes things different. One of those things is related to how we communicate on bugs. 

The team took upon themselves a change where some information currently stored in a local database in our own objects now move to being stored in a 3rd party system, with a well-defined REST API to get to the information. The developers would do their bit, and as their flow of pull requests was dying down, the new tester raised the question: is this supposed to be ready? Having already tested it throughout the steps, they knew of many things that didn't quite work, and having had conversations on those with the developers, they expected the developers did too. Yet, the state of functionality was not where done would reside, and the tester was confused.

With the confirmation of the problems being unknown, the developer insisted on writing separate bug reports on every symptom. And there were quite a few. What had happened is that while the previous local object to REST API had included read-only information after the change locally, no one remembered to discuss that this one was asymmetric and didn't. There needed to be both the information from the REST API and the local object, and as when you confuse a principle like this, data was getting lost in quite confusing ways. 

Discussing the symptoms the tester was seeing made us all feel a little puzzled without the connecting story of why the change was failing. And with failing change and many reports, the disconnect in communication between us all was clear. 

As soon as I worked together with the new tester, we figured out what was wrong. And the new tester turned their experience to a metaphor.

It was like we had been building a car, and as they sat on the driver seat, they could experience that they no longer saw through the windshield  like before. They could experience the steering wheel was hard to use, and that reaching pedals was getting next to impossible. They just could not tell why they experienced all this.  They could not tell it was because the seat position had shifted. Not because the change was to change the seat position - the change was to install new car mats on the floor. They were describing symptoms, creating bug reports of each symptom. And the worst part was that they were getting fixes for their symptoms, not yet knowing the reason they had to see those symptoms and a nagging feeling that the fixes they were getting may not address why they are seeing those symptoms.

When they need the chair moved back, they instead get a wider windshield, a pedals adapter and had to live with the bad experience with the steering wheel. 

As testers, we are taught to describe what we observe, the symptoms. Yet the way we are able to frame those symptoms triggers memories and ideas of what the right, proper solution is with the developers. And my new colleague is very right that them doing their job on reporting is not sufficient. They can, and should, work on setting the relationship right so that instead of fixing by symptoms, we fix by causes. 

Wednesday, October 7, 2020

The Difference of a Test Idea and a Test Case

Year after year, organization after organization I join in anticipation of not having to see test cases other than those that are automated, and through continuous execution guide the process in keeping themselves up to date. 

Yet year after year, organization after organization I learn people still write test cases. Those things where there's a title and steps. Those that set out a flow through the application that needs to be verified, and steps you can choose to disobey because you are not a robot. 

The way I look at it, ideas are cheap, and we don't care much on how well they are documented. The ideas on post-it notes I find often difficult to decipher in a month, but they are critical notes when I'm learning to structure things in a way that I can recall later. Test cases are something we may want to keep around for later, they are more than ideas. They have structure that supports executing them, even if they were checklist-like. They often include steps and an idea of an order of execution. Test cases are better considered output of testing (and automated!) than an input to testing.

The thing is, some of our worst experiences will never get published, because we can't talk about them while we are in the middle of them. Inspired by conversations today, I go back to an experience I could not talk about back when it happened, but I can today, with examples.

I was working with a product / consulting company, and the consulting side I was representing was doing quite well - so well in fact that it was hard to make space for any of our testing experts to do testing on our own product. The consulting paid so much better. The management pulled one of us consultant out to test release 1, another to test release 2 and so forth. Eventually, I was next in line. 

Thinking about it 20 years later, it is hilarious to realize I already was presenting as the resident testing expert. Going in to test the product, I intended to do well - don't we all. I asked what the ones before me had done, and was pointed to a test management tool, by the proud colleagues who had made sure they documented their testing. 

These are actual samples of what I received. 

These were considered test cases. They are a lot worse than some of the better test cases I have seen over the years, but even the better ones come with the universal stigma of not being useful for good testing. 

These test cases were awful. They still are awful. Time did not make them better, only worse. They were step by step instructions, with a lot of effort in faking testing by describing elaborately the same steps just so that the tester using them would know they've for example moved a box up, down, right and left. They had magic values defined that completely miss the point of why those values were selected, but I can only guess the choice of data in these reflected naming by concept, not naming for likelihood of finding problems or even pointing at ideas where problems might be. 

On my round, the first thing I did was throw these away. These were done by my respected colleagues, and they did the best work they thought of with the situation they had at hand. I could only hope they reflected test cases, not the testing that was done. 

I moved testing to exploratory. No more test cases. Closest we would get was expecting a checklist of features and ideas after it was all learned and structured through multiple rounds of rehearsing through actual testing. 

While what I run into in organizations nowadays is generally not this bad, it isn't much better either. I've shown it again and again that dropping test cases has improved testing. The controlled experiment of four weeks where we used pre-designed cases for two finding zero problems and using freeform exploratory testing with prepared test data for other two finding all the problems there was to report from that acceptance test. The removal of 39 pages with 46 "test cases" where 3 pieces of information were something I did not know joining a new company on week 1. The others where I did not do a public presentation I can go dig out years later for numbers I was sharing. 

I wish the world was ready for good testing, but it still isn't. Automating and working through ideas of better and good seem like our best hope. And I'm delighted that test automation is merely a smart way of documenting your testing. 

Saturday, September 26, 2020

A Step-Wise Guiding into Automation

Looking at a group of testers struggle with automation, I listened to their concerns:

    - Automating was slow. When automating, it would easily take the whole week
      to get one thing automated.

    - Finding stuff to automate from manual test cases was hard. It was usually a step, 
      not the whole thing that could be automated.

    - It was easy to forget earlier ideas. Writing them down in Jira was encouraged, but 
      Jira was where information goes to die. If it didn't die on its way like often happened.

I'm sure all their concerns and experiences were true and valid. The way the system of work had been set up did not really give them a good perspective of what was done and not, and things were hard. 

In my mind, I knew what was expected of the testing they should do. Looking at the testing they had done, it was all good, but not all that was needed. Continuing exactly as before would not introduce a change we needed. So I introduced an experiment. 

We would, through the shared test automation codebase, automate all the tests we thought we could document. No separate manual test cases. Only automated. We would split out efforts so that we could see coverage though the codebase adding first versions of all tests that were quick to add and then add actual automation into them a test at a time. Or even, a step of a test at a time if it made sense.

We would refactor our tests so that mapping to manual tests to automated tests was not an issue, as all tests were targeted to become automation. 

None of the people in the team had ever heard of an idea that you'd create tests that had only name and a line of log but agreed to play along. Robot Framework, the tool they were already using, made this particularly straightforward. 

Since I can't share actual work examples, I will give you the starter I just wrote to illustrate the idea on documenting like this while exploring, using prime from eviltester as a test target. 

*** Settings ***
Documentation Who knowns what, just starting to explore

*** Test Cases ***
Example test having NOTHING to do with what we are testing but it runs!
[Tags] skeleton
Log Yep, 1/1 pass!

This is already an executable test. All it does is that it logs. The name and log message can convey information on the design. Using a tag shows numbers of these in the reports and logs. 

Notice that while the Documentation part of this identifies my test target, there is actually nothing automated against it. It is a form of test case documentation, but this time it is in code, moving to more code, and keeping people together on the system we are implementing to test the system we have. 

As I added the first set of skeleton tests and shared them with my team, they already were surprised. The examples showed them what they were responsible for, which was different from their current understanding. And I designed my placeholders already in a way that can be automated. I had placeholders for keywords that I recognized while designing, and I had placeholders for test cases. 

Finally, at work, I introduced a four level categorization of "system tests" in automation:

Level 1 is team functionality on real hardware. 

Level 2 is combining level 1 features 

Level 3 is combining level 1 for user relevant flows

Level 4 is seeing the system of systems around our thing. 

The work on this continues at work, and the concreteness of this enables me to introduce abilities to test the team may have assumed they can't have. Also, it enables them to correct me, a complete newbie to their problem domain on any of the misunderstandings I have on what and how we could test. 

The experiment is still new, but I am also trying it out with the way I teach exploratory testing. One of the sessions I have been creating recently is on using test automation as documentation and reach. With people who  have never written any automated tests, I imagined browser tests in Robot Framework might do. They do, but then it takes the time from testing to tool learning. Now I will try if the first step of creating skeletons enables a new group to stick with exploration first, and only jump into a detail of automating after. 

Introducing Exploratory Testing

 "All testing is exploratory". 

I saw the phrase pop up again in my good friend's bio, and stopped to think if we agree or disagree.

All testing I ever do is exploratory. All testing that qualifies as good testing for me is exploratory. But all testing that people ask from me definitely is not exploratory.

I worked at one product company on two separate batches of about 3 years each, and with 10 years in between. If there is a type of company that needs good, exploratory testing to survive, product companies are this. The whole concept originated 35 years ago from Silicon Valley product companies Cem Kaner was working in. Yet when I first joined, exploratory testing was something we did after test cases.

We wrote test cases, tracked running of those test cases, and had some of the better tooling in continuously moving test target version with our Word-Excel in-house tooling. What made the tooling better than any of the ones I have now in my use was the in-built idea of primarily needing to understand when you last verified a particular test idea, since every change effectively cancels out results of all the things you've done before. 

On top of test cases, we were asked to do exploratory testing. We were guided to test our tests so that we did not obey the steps, but introduced variance. We were guided to take a moment after each test to think what we would test because of what we had just learned, and do it. We were guided in taking half-a-day every week in just putting the test cases aside and testing freely. 

It was clear that all testing was not exploratory testing.

10 years later, there was no test cases. There was one person who would do "exploratory testing" meaning they would follow user flows to confirm things without documentation, without any ability to explain what they were doing other than rare bugs they might run into, missing out a lot of problems. And then there was test automation that codified lessons of how to keep testing continuously, discovered through detailed exploration finding problems. 

It was clear that testing they called now exploratory testing was not exploratory testing. It was test automation avoidance. And the testing they called test automation was the real exploratory testing. 

I get to go around companies and inside companies, and I can confirm that we are far from all testing being exploratory. We have still tyranny of test cases in many places. We will have test automation done without exploring while doing it, taking away some of its value. 

We have managers asking for a "planned" approach to testing, asking for "requirements coverage". They ask those as a proxy to good testing, and yet in asking those, often end up getting worse testing, testing that is not exploratory. 

Opposite to exploratory is confirmatory. It does not seek the holes between the things you realized to ask for, but only things you realized to ask for. And you want more than you know to ask for. 

So I keep going to companies, to introduce exploratory testing. 

I bring down test cases, I move documenting tests to automation. 

I convince people to not care so much for who does what, but work together to learn. Developers are great exploratory testers if only they let go of the recipe where they do what they were told, and own up to creating a worthwhile solution. 

I break ideas of exploratory testing as something you do on top of other things testing. I introduce foundations of confirming what we know being true and learning, and then building figuring out unknown on top of it. 

We read requirements, we track requirements, we use them as testing starters. But instead of confirming what is there, we seek to find what is not. 

All testing will not be exploratory testing while we write and execute tests cases. All testing may have an exploratory angle. But we can do better. 

Monday, September 21, 2020

Exploratory Testing and Explaining Time

If you've looked into exploratory testing, chances are you've run into two models of explaining time.

The first one is around the observation that there are four broad categories of "work" when you do exploratory testing, and only one of them is actually taking your testing forward, and thus visualizing the portion of this may be useful. In that model, we split our time to setup, test, bug and off-charter. Setup is anything we do to get ready to test. Test is anything we do to amp up coverage from just starting to getting close to completion. Bug is when we get interrupted by reporting and collaboration on results of testing. And off-charter is when we don't get to do testing but to exist in the organization that contains the testing we do. 

The broad categories of work model has been very helpful for me explaining testing time to people over the years. It really boils down to a simple statement: Getting to coverage takes focused time, and if we report the time on it, you may have an idea of testing progressing. Let's not measure test cases or test ideas, but let's measure time that gives us a fighting chance of getting testing done. 

The three other categories of time use outside "test" are set up as the possible enemy. Setup takes time - it's away from testing! Finding many bugs - not only away from testing, but requiring to repeat all testing done so far! Off-charter - you're having me sit in meetings! They can also be set up as questions to facilitate a positive impact on the "test", as in investing in setup that makes test time be sufficient, or investing in pairing on bugs that make future bugs less frequent. 

The second model making rounds includes a more fine-grained split of activities happening within exploratory testing sessions, that people could even use to explain their sessions for things like daily meetings. Instead of saying you are doing "testing" day after day for that big login feature, you could explain your focus with words like intake (asking around for information and expectations), survey (using the software to map, but not really test), setup (infra and data specifically), analysis (creating a coverage outline), deep coverage (get serious testing done), closure (retesting and reporting). 

If we map these activities to the four categories, there's a lot of explanation for setup here: intake, survey, setup, analysis and closure are all mostly setup - they don't really build up coverage, but are necessary parts of doing testing properly. 

While the first model has been valuable for me in years of use, I would replace the latter model by finding the words that help you communicate with your team. If these words help, great. If these words silence your team members and create a distance of them not understanding your work, not so great.

The words I find myself using to explain how I progress through a change / feature related exploratory testing are:

  • what am I investing in: in the now, or for later; getting the job done quick vs. enabling myself and others in the future
  • what kind of outputs I'm generating: story of my testing, bug reports, mindmap, executable specifications 
  • what kind of output mindset my work has: generative or completion-oriented; some work generates more work, some gets stuff done
  • whether you see movement: working vs. introspecting; some work looks like hands on keyboard, other look like staring at a whiteboard
For me it is important to add more words to "test" too: mapping, acquiring coverage, completing as well as for "bugs": isolating, documenting, demonstrating. 

Looking at the way I work, I explain very little of testing in daily meetings and don't write a report of any kind. Documentation I leave behind is automation. For transferring deeper knowledge, I pair with people, and to get new people started, I write a one page playbook of testing that sets the stage. The people I explain testing to are ones I pair with for either automation or for doing particular testing. 

The real question of explaining your time is: Why are you doing it? Do you grow others with it? Do you explain yourself to people who hired you for them to trust you? Or maybe you explain it to yourself  as introspection, to figure out how things could be different for you tomorrow.