Fear of technical and unknown

I've been a tester for 20 years, and it has been (and will continue to be) quite a learning journey. With this post, I wanted to take you back on my memory lane of something that really spoke volumes to me, enough that I remember to keep going back to it 15 years later: Bret Pettichord's old article on how testers and developers think differently.

Here I wanted to address one of these four dimensions in particular: Need of Mastery. The reason I want to discuss it is that I delivered a half-day workshop on API testing last Thursday, and looking at how the group responded to the work lead me to recognize a common pattern in people facing exploratory testing in course settings: fear of the technical, fear of the unknown. Paralyzing fear.

For longer than I care to remember, I've learned to tell myself that as a tester I'm expected to be able to work with partial information and understanding, and still be useful. While I definitely am not a product expert on day 1 of a new product, I take pride in being able to provide results and being useful already then. It's about the choices I make on what to focus on next. And day 1 is always just a beginning, it opens up the chance of treating every day as a learning experience I walk out of knowing just a tiny bit more than I did before the day.

As a tester, I've learned to think that my job is to get up to speed quickly, and be brave about addressing stuff I don't know - to learn, to change what others know. Ignorance is important, it helps me work with the way a new user feels, but it is also temporary. I need to pay attention to to it before I lose it. And I need mechanisms to get closer to it again and again.

When facing a new application to explore, I feel the tingle of fear of me not being good enough for that particular application or domain. I fear there's no feedback I could provide on top of what is already known and what people care for. But the fear isn't stopping me, it's driving me. It reminds me that when facing fear, I can do something, anything, to learn just a little more. If I choose to plan a lot but do a little, I know a little. There must be a balance that ties my considerations into the actual doing of testing. I know that if I don't start testing while clueless, the time is ticking away and I'm not providing anything. I'm not learning or contributing. And those two tend to go hand in hand.

When faced with something we don't know, we can step away and say we need more info before we start. Or we can plunge in and accept that trial shows us something of what we know.

I'm realizing there is a particular skillset to test code I don't yet understand. I can feel I need to know much more, but still make progress. There's a similar skillset for programmers on working with code they don't yet understand - techniques and approaches that make learning fast while already doing the work.

Need to think of how I could explain the skillset better. If you have ideas, I'd love to talk. Ping me on skype: maaret_pyhajarvi if you feel like contributing on this.

Exploratory Testing an API

This week Thursday, I'm running a workshop at Booster Conference on Exploratory Testing an API. I decided to go for ApprovalTests as the test target because it's something I've been meaning to learn, and what is a better way to learn than to test. Also, having the creator close by seemed like it could be an advantage.

Before my prep of me testing it, I knew there was extensive unit tests. ApprovalTests are tested with ApprovalTests, and a lot of them. So my main interest is not on the stuff that is already being tested, but on stuff that tends to slip through the cracks when unit testing.

With a bit of personal brainstorming, I came up with three approaches to focus on.

1. Usability and Consistency of the API and functionalities
2. Environment
3. Inputs and outputs

It's been quite a ride for preparations. 

First I noted cosmetic bugs - typos in the interface. Within 5 minutes, those were gone, fixed. I may still be snarky on the backwards compatibility for the poor users who rely on the mistyped but these were in the interfaces that are not intended for external users, regardless of being visible. 

Then I pointed out a specific difficulty for the users of the API. ApprovalTests uses the concept of Reporters, that get introduced with something of this format:

    [UseReporter(typeof(DiffReporter))]

And there's a lot of reporters! But they are all over the IDE-provided list, as they have no means of grouping them together. And honestly, the idea of a user going to check what is available in reporters in the open source project just seems a little more than most users I can imagine will do. Although I'm first to admit that I might be even more on the lazy side on this stuff. 

End result: change request to rename all reporters to start with Report. And while at it, clarifying the names otherwise too. Some of the reporters have the purpose of running silently with the builds, while others pop up the best tools for analysis. 

I built up a nice list of purposes that the Approvals and Reporters serve, that will make a great foundation for further exploration with my workshop participant. And mapping those out through learning about the consistency (and inconsistency) might be helpful as documentation also later on, so I'll share that later. 

I concluded my prep with a research on inputs and outputs in the unit tests. The innocent request of "please run them" revealed that they had not been run with the last change and had tests breaking. Software & tests got fixed - and I got my confirmation that my hunch on environments being still troublesome is worth going for with line breaks in the files. We sampled different types of tests, and found groups of tests that had been accidentally excluded from the solution. Added some of them back too. 

I'm going to have so much fun with the people at the workshop revealing problems with this. The best of all of it: I don't need to waste my time on trivialities of simple bugs, because there's been a significant effort into the unit tests. They are pretty impressive, but showing those isn't what this is about: it's about learning things you don't learn with the unit tester glasses: flaws of your design logic, inconsiderations of your environment and things that just slipped through the programmer's eyes. 

Semi--automated tests, with an advanced version

Remembering back a few jobs ago, I was working with a great team of testers and we were trying to figure out automation. We put together weekly retreats on the other side of the company premises to find time away from the regular work and interruptions, working on adding incremental value through automation to the testing we were doing.

Back then, one of the leaders from test automation perspective introduced us a concept of semi-automated tests. There's things that are easy to have the computer do, and things that are harder to have the computer do. Tests that would take us to the point where human intervention was required and then just pop up a message allowing you to do your bit after all the leading into -steps automated was useful. We really needed something like that back then: a reminder that with "test automation", it's not about automating all things testing but some things testing - any that is useful.

Today I remembered this experience as I was exploring ApprovalTests with Llewellyn Falco (the main developer of this open source project). He was showing me a particular group of test, using fancy words like iExecutableQuery and at first, I was somewhat disengaged. Hard words of stuff that don't map out in my mind. He run the first test, explaining what was going on and I started connecting pieces. With third and different test, I realized the usefulness of what he was doing.

He had unit tests running on 3rd party software, and I was thinking these cannot exist, making me more disengaged than I should have been. The idea on how the tests were set up were interesting and useful.

His test would create a call to the 3rd party app (diff tools in this particular case) and save it into a file and launch that 3rd party app.  This would only pop up on failure. If the call had remained the same, he would only verify the call, but on1st time and later on failure he needed to see also that the 3rd party app did what was expected.

This was a much more advanced way of doing the semi-automated tests I found useful already a decade ago. This turned the semi-automated while testing once into automated for regression purposes. Surely it does not test "the real thing". But the abstraction it tests feels useful.

With all the tests I've been seeing, why haven't I seen more of this before? Have you?

Products as scrapbooks and modeling your way out

I spent a bit of time with my team's designer to help model a particularly complicated area of features. I knew from having tested it that it was not good, but the discussion and our joint efforts into making it better made me realize more than what I knew from past testing of it.

As I was listing concepts and features, and categorizing them while introducing them, patterns started to emerge. The assignment we were working on was to add one more feature to a complicated area. A feature described to us as "just add this dialog" made us talk about all the work we do to avoid building products like scrapbooks, and how hard it is sometimes to go beyond the obvious request.

With each concept we listed, a pattern started to emerge and we started scribbling on paper. At first the user is handling a massive amount of data by default, then filters some of it out. But also adds some more while filtering. And then again there's selections to add more. And again filter more. A lot more. We calculated 7 different scoping actions in a seemingly random order, and realized it was no wonder we had hard time following through the area.

From the first shape, it became obvious that this was not a good, clear and simple model for the user. So we draw concept shapes that would be clearer. We could start wide and consistently narrow down until we're where we want to be. Or we could start really narrow, and carefully widen up to be only at most where we end up wanting to be.

Drawing silly pictures comes almost instinctively from modeling. The pictures help us work out which of the straightforward routes we can get our design into, but also, they generated a lot of ideas for me on how to fool the current system to reveal complicated bugs.

Scrapbook of a product is a bug. But it's a bug that won't get fixed in an hour. 

Video: Styles of Pair Testing

After months of procrastinating how hard making videos is, I did one! Very proud of myself, and wanted to share the end result with you.

I already have ideas on how to make this better and how to learn new stuff on making videos. Now that the first step is done, it will be fun to work on an improved version. 

From identity to skill sets

A little over a year ago, I had a discussion on twitter that I look back to realizing I've made yet another 180 degree turn on my perceptions. I had a discussion with my co-creator for the European Testing Conference, Adi Bolboaca, a software craftsman on the topic of "testers are developers too". I remember very specifically responding in twitter and in following face to face discussions that for me being a tester is my identity. It tells me where I came from. It tells me what I'm good at. And it tells me how I'm special, after all the years of hard work on learning to become better at it every day - a job that never ends though.

Then various things happened. I started reading more about mindsets and went back to listen to the brilliant talk by Linda Rising that we also invited for European Testing Conference. I started to realize that what I really want to be about is learning, not testing. I've always considered testing to be learning, and that my focus on interests has been from value and business and systems, not the details of the code.

But when you've been learning every day for 20 years, limiting your learning becomes an idea that is no longer a necessity. As a new tester, I really needed all of my energy in learning about how to model the value, how to identify bugs and how to put the best of me together for any single day to produce useful results. As a seasoned tester, some things I've practiced atrophy, but others are almost muscle memory, things I have to think hard to even explain. And I have room for new skills areas.

This all leads into a discussion with a friend on identity vs. skill sets. Think of firemen. Being a fireman could be an identity. In the past, there was a bigger demand for firemen, but things have changed. There's early detection systems, sprinkler systems and the sort. Firemen wait to do the thing they exist for. There's even firemen who say they hope that there would be more fires, so that there would be more to do. More within that specific skill set.

When your entire identity is wrapped around a certain skill set, you want the need of that skill set to exist, whether is good or bad. If you expand your skill sets and find ways to be useful in a bigger scale, you're no longer dependent on your specific skill set being in demand.

Sounds a lot like what I've been going through as a tester. From identity to expanded skill sets. I'm still a brilliant tester. I'm also accepting that I'm just as good product manager, project manager and agile coach. And becoming just as good a programmer. One day at a time, learning every day, like before. 

Changing conversation from bugs

Today feels like a good day to celebrate the progress we've made in my team. I'm approaching four years at this job, and while I'm regularly thinking of moving on, I just love what we do.

Four years ago, one of the hallmark moments was a day few months into me being in the company, when the product manager came to my cubicle, started shaking my hand and told me first he owes me a beer. He had thought until that point that when software comes to him, he should expect it to be broken. I changed that by providing information to my developers they needed.

At first, there was a lot of pain. Our sprints were awful and always late. I convinced the team and product management to try out daily releases and things improved a lot. We also stopped estimating efforts to implement stuff, and focused our discussions on how we can make it small and whether it is the thing to do at all. All the energy that went into looking good in estimates could now go into developing ourselves and our product.

About a year ago when writing my CV as I was looking for opportunities to work in US (still looking, but I need to find my brilliant team/organization that sponsors the H1N visa). I looked at some of my results bugs and added this: "Personally reporting and getting fixed 2261 issues over a period of 2,5 years on Granlund Designer and Granlund Manager projects". I was shocked as  that means that on average I had reported 2,5 bugs every day. I had already worked on improving the skills of the developers in developing and testing, but I felt I needed to step up the game.

A year ago, I intentionally emphasized discussion and developer learning. I started pushing my team to practice together pairing (rarely, they hate it) and mobbing (more regularly, as they learned to love the learning). And the result is that I've logged 348 issues in the last year, but that is only  1,1 on a daily average. My big (and regular) successes are the times when developers get in touch explaining a feature, and just look at me and tell me what they now know is wrong with their implementation. They hit the mark better, and our relationship has increased in quality.

We release on average twice a week, and our end users rarely see problems they would complain about. The amount of test automation is ridiculously low, but the technical quality of our code (a thing I champion for regularly) is good and improving.

I feel I've gone through an intense year of changing the conversation from bugs to value and skills, and it's been worth it. For a second year in row my collaboration review with the manager focused on how much I've helped the team to grow to empirical feedback and actionable suggestions, not on finding or missing bugs.

This is why I love long-term commitment. I now know that apart from one troublesome individual (see the previous post), my team will excel without me when I leave.