Security, Testing and My Place in All of This

Where I work, we have this constant struggle of us (the good guys - cyber security specialists) and them (the bad guys - in their various forms). The struggle is kind of fascinating to look at from a tester perspective.

We know software always has bugs. The good ones of us are good at catching them, and catching some of the relevant ones before we ever deliver our software out. And we catch some of them after release, listing carefully to signals and confirming suspicions. As a tester, I've come to the ultimate conclusion that what matters is our ability to change. It matters because we will miss something, regardless of our best (and improving) efforts. But it matters more because we have an adversary that plays the game from a different angle.

The people who create malware are software developers, just like us in many ways. I find it fascinating to think if they have the same needs of investing in testing. Any software introducing the right kind of bugs can give an attacker a route into a system they shouldn't be in. Does it matter if the software using an exploit is fine-tuned and tested?

I had a chance to talk with an old tester colleague of mine who now uses his analytical skills in working with security after something bad happens. Something bad these days could be that a company has ended up with ransomware, lots of critical data unaccessible as it has been encrypted. You could try restoring from backups, but what if your incremental backups don't sum up to full? Apparently you can also use your testing skills, testing the malware and finding bugs in it to be able to open the encrypted files again. The joy on my colleagues face on finding the bug and using it to open up all the files was the same joy we feel when we catch relevant bugs in our software. The skills used are same or similar, the target of testing is just completely different.

Working in a security company makes me wonder about the role of testing. A lot of the bugs I routinely handle have a lot to do with lost time, lost services, inability to do what needs doing. But some bugs, the security ones, have to do with lost access and lost data.

We're starting to see the value of security as data is crucial. I wonder if we will ever see the value of people's time, or if other solutions to free up time over delivering well functioning (tested and fixed) software will win out.

We live in interesting times. And today I stop to appreciate that what I do for *this* software, I could do for any software. 

Reading with rose-colored glasses

As a tester, I specialize in feedback. I both find things that no one else was bringing to the table, and amplify things someone else did so that the feedback gets the attention it needs. One of my favorite sayings is from Cem Kaner's book Lessons Learned in Software Testing.

I got to think about this today, as I had a pair of people to give a piece of positive feedback on. 

I approached the first, using a phrase of "waking up a security bear" to emphasize that something they did resulted in a positive outcome of identifying and addressing a vulnerability. The positive feedback was taken at face value. 

I approach the second, explaining a little more context of why this was important. And while I thought I was still trying to say "well done", I got into a spiral discussion of what was wrong with what they did. Reflecting the interaction, there was *one thing wrong* - the immediate response that the bug had been previously reported and dismissed then. 

The first one approached the feedback as something positive. The second approached the feedback as something negative. I ended up with two completely different interactions for the same message: job well done, I would like to turn up the good and this was good. 

The whole experience took me back to a one-liner from my boss: "I want to talk with you". My immediate response was "bad or good". I chose to wear my rose-colored glasses and assume positive intent. 

Putting these two things together, I realize that wearing the glasses of good intent is the single most relevant thing I have done to feel happier and more successful as a tester. The world is filled with good, and even the (negative) feedback we are bearers of is positive from a constructive angle. 

I hate it when the rules change

I have a blessing and a curse in feeling empathy at office. It is a blessing in the sense of being able to help people do emotional labor and eventually be happier and more fully themselves at work. It's a curse as the emotional labor isn't free but drains on my energies, and can easily distract me from the so called technical content.

Sometimes other's feelings are not hard to pick up at all. One of these days, I arrived to my place of work seeing angry pacing and hearing swearwords. Understanding that someone is upset wasn't hard at all. Offering myself as someone to listen to what was wrong wasn't hard at all. Comparing the vocal expressions in privacy of our room and what ended up being said in writing for a pull request wasn't hard at all. And realizing there was a difference in the unfiltered and filtered expressions, I found ways of seeing how this discrepancy could manifest in quality as we experience it.

Sometimes feelings can be really hard to pick up, as some people hide it better. And I confess to the possibility of mirroring my own feelings to people. Me hating rules changes could as well be my feeling projected and understood through an experience someone else was going through.

There was a pull request, introducing a new test into test automation. There was nothing special about it. It was business as usual. Until the rules changed, without a warning.

If things flowed just like they always flow, we could have expected to get two comments. There was a mistake in the logic of what gets verified, and these typically get caught and corrected in someone else reviewing the code.  But this time, things were different. 

I sat on my computer, looking at this piece of code and invited a developer to look at it with me. I asked about things that bothered me on it, expressing uncertainty and need to learn what was our standard. Instead, I learned that our bar was sometimes low. The comments would be the minimal to make the pull request acceptable. Not my idea of actively fixing and putting things in the right order. 

Just talking about it started a change of rules. What used to be enough no longer was. But on the other side of that pull request now getting 12 comments and basically advice to completely rewrite it was a real person, who was not part of changing the rules. 

If this happened to me, I would be devastated. They turned quiet and worked on other things. 

As I expressed the  need of going and checking on they felt on receiving end, to be surprised by the idea that others considered change of rules like this business as usual, and wondering why I might be concerned.

The whole experience just reminded me that while we like to think of feedback we're giving as striving for improvement and objective, there's always a person on the receiving end. 

When rules change, having a system in the middle may hide how the other felt. It does not take away the fact that the other one struggled. Addressing this face to face, person to person is a much  nicer way of changing rules for the better. 

Plans make us slower and worse

Over time, I've had the pleasure of working with different kinds of projects we like to call "agile".

There's ones that I absolutely love, like what my "home team" does. We frequently talk about business directions, needs and visions and contribute to those equally from a technology perspective. But we don't have a plan. We don't know what we will do next. When we are starting something, we try to figure out how we could make it smaller. When something is started, then there is an idea of a plan. And we don't do estimates. No story points, no t-shirt sizes, no hours to work on it. Again we ask if we could make it smaller.

Then there's ones that require "visibility" that I don't particularly appreciate. They start with the idea of someone outside "approving and following a plan". The plans, as flexible as we try to make them SHOULD fail in the face of reality and adjusted. But making the plans was painful. So we end up rather faking we work to a plan. We reject doing the right things that emerged, because the plan suggested an order of things. For each part of the plan, we refine and we estimate. We refine estimates. We monitor what was done and what was not. Every ceremony discourages doing the right things over doing the planned things.

The first one succeeds when you learn to deliver parts that take you days, not months. And these are parts of valuable functionality, not "an api I could use later on".

A lot of times, plans just make us slower and worse. The time you use on planning could be used on doing.

Test Automation Smells so Obvious Anyone Could Notice

Like so many exploratory testing enthusiasts before me, I can easily find things to do with my time so that I don't have to go dig into the realms of test automation. But every now and then, unlike so many exploratory testing enthusiasts, I go and take a dig at automation anyway. It reads mostly like English anyway.



Here's my list of things to work on inspired looking at one set of tests today.

Tests that don't test only tour

So you have some kind of structure with your tests. You have test suites / sets somewhere that bundle things together. You have some tests going into those suites /sets. And you have some libraries you can use. Great start. But take a look at the things in this that are considered tests. Can you find any that haven't got a single verification, asserting that something must be true? When you do, these are tests that don't really test, they tour. They are a path to getting to a place where you actually want to do some testing. Don't muddle them together on the same level as things that actually are checking things.

Tests inheriting tests over libraries

Inheritance can make things very muddled for  your random stroller. You're trying to read steps of what happens, and for sake of similarity, someone came up with the great idea of inheriting and overriding to create the differences. Same things would appear like they could be done with libraries, instead of test cases. Do you really have to conceptually mix up test cases with inheritance?

Randoms inside a test

You see a test that says TestABC. Great, it looks like this is a test that tests ABC. You continue reading, and a call to random pops in front of you. Whenever this test gets run, it always does ABC but there's a few different ways to do ABC. Someone decided to leave which of the ways you get to faith, making sure every time it fails you need to go and check which of the options was broken. Isn't there enough lack of deterministic behavior in the tests without adding randoms in?

Avoiding passing values

Reading the code, you start realizing a whole lot of method calls look like they're the same yet they are not - quite. There's some little detail on each that differs. Your method could take an argument, and you could do some magic with the argument. But for some reason it seems it was a better idea to create a number of separate methods to call, without argument but each including what looks awfully lot like a thing that would have belonged in the place of an argument in its name.

Duplicating to my corner

There's a directory there somewhere, that uses a label you recognize as a concept meaning ownership that draws you in. And you find a nice cozy corner of clearly laid out tests. Reading through their names, you start to feel like you've seen some of them before. Searching confirms - there's a well organized little corner there somewhere what everything is concisely together. There however is a lot of duplicated code elsewhere. But that must be someone else's problem.


A Fool's Coverage

As an exploratory tester, you start realizing how much coverage there could be if the collaboration between what you learn and know would better turn into automation. You identify some of the English in the tests that makes sense in the world of using the application for exploring it, and realize how little of your ideas have ended up encoded into the tests. Whatever is in, gets continuously monitored. Running the same thing a thousand times isn't really yet that much in the coverage - it's a version of a fool's coverage.



Here's a thing: anything I can name and recognize, I can fix. And while these feel obvious to me today, they clearly have not been equally obvious when they were introduced. What does your "let's fix these" list look like?

Learning Programming Test-Driven

I've been delivering a test automation course, in a mob format. I get the whole group together, to work on test automation activities. We first get an experience that we share, and then we talk about what we learned and need to know about automation.

We've typically spent a couple of sessions on Selenium web driver, first just creating a test that runs, but soon after refactoring the tests so that we'd have more of a page object pattern in use. We spend a session on ApprovalTests, just to figure out what could be tested if we approached testing with creating and comparing to a "golden master". We test an API in a "fill in the blanks" kind of style. And finally, we spend a session starting a program blank, seeing where Test-Driven Development takes us.

The Test-Driven activity has turned out to be my favorite. I love how the tests grow with the application. I love how small steps end up taking us to a coverage we wouldn't create later on if we tested after. But what I love the most is the way the group - usually full of people who don't identify as programmers, some people who have never written code in their life - reacts to it.

We usually do very simple problems like FizzBuzz. Getting the 1st test out of the group asking for a single example is often the hardest part. Turning the example to code some syntax needs to be revealed, but none of the cruft around classes, just a single test. And when most of the programming in the IDE is learning to use the automatic generation of methods, people come out with a very different feel to programming than if they had to know how to write all the magic words around it.

We've done this in Python. We've done it in C# while I still had Visual Studio. We most often do it in Java. And there is little difference in these two when the full editor support is available.

The reactions of the learners I've worked with make me want to experiment with this more. What if we could teach programming so that we did not go through the usual moves of naming concepts, but we could teach test first and experience first. Naming what you did can come later. Even much later, when you've seen several examples of how some things are done or used.

I'm an awesome tester who also happens to be woman

At a test automation conference, I started talking with another participant. We exchanged details of the craft through name dropping and content appreciations. We talked about his upcoming talk, and I shared some ideas on the same topic. Excited on the discussion we were having, he decided to suggest I could do a lightning talk in the evening.

It was great that he suggested - I was not aware there was one, let alone that I could still get listed and that the voting was going on throughout the day. I immediately made up my mind.

But he continued: "we need women speaking".

I said nothing, more like shrugged. But decided to address it when speaking.

I introduced myself. I shared that I was here doing a lightning talk because another participant encouraged me to so that there would be women speaking. Needless to say, I was the only woman on the lightning talks round. I also mentioned that while it feels that my main reason to be here would be my gender, I'm here just as a person with credentials: that this is talk number 353 that I'm delivering, and that Kent Beck said yesterday he loved my book. I made a joke about it, because it was a better option for feeling uncomfortable than being silent about it.

After the talks, I talked again with the other participant. He apologized and invited ideas of how he could have better expressed his excitement on what I had potential on sharing. I suggested that reminding me on my gender (any woman on their gender) could be a pattern to avoid. They're already painfully aware.

The conference had no women in organizing committee.
There was one woman speaker out of 9 talks.
There were LOTS of women in the audience.

There's no absolute "best" in speakers. Speakers tell stories that are based on the life lived. We need diverse voices. What I said today to open my lightning talk was something that just happened to me: all three details on my 'credentials'. I bet no one encouraged any of the men in a gender-based fashion.

I make lists of awesome testers. Notice the list never mentions "awesome women testers". Because the qualifier of gender isn't relevant. They are awesome, just as they are. Yet looks like no one can share and promote that list without mentioning gender.