For testers still living in the waterfall

There was a piece of feedback from a testing conference: "There's all these great ideas about testing in agile. We still live in the world of waterfall. Is there nothing other for us in testing than to move to agile?"

It's been a while since I've used my time to actively think about waterfall projects, other than for the fact that I feel so blessed not to work on those anymore. I wanted to write this post thinking of the colleagues fortunate in different ways than I am, with the ideas of what is there for you on the current themes of testing.

1. Testing is testing, agile is a context

This phrase was going around some years back, and it's never been more true. Testing, as in looking at a product with most recent changes available to use and find information on, is very much the same with agile.

Agile as a context suggests some ideas that work just as well in waterfall environment:

• If we shared the ideas of what and how we're testing, we'd get more done and better - think testing over testers 
• If someone is a specialist in testing, they will have a chance of bringing in diversity of viewpoints (testers and programmers focus their thinking differently!) and find problems the non-specialists struggle with. It's not magic, it's time and focus combined with continuously improving skills. 
• A professional, self-organized team that works on learning does better in delivering. 

In waterfall, it's still a day after the other. Every day is a chance of learning. The feedback cycles are different (slower, requiring different effort) but none of that stops you. 

2. A lot of it starts with understanding value

In the world of waterfall, we used to talk a lot about requirements. But they are really our interpretations of what would be of value. 

Why would anyone want to use your product? What are the core functionalities? What are the core risks, with regards to quality? What do you need to know and is it you who will be delivering that information? 

You can work out the core risks just in time. For some of them, the right time is to talk about your concerns before implementation. For others, the right time is to talk about them sneakily while implementation is ongoing, without overloading the stressed developers. And for others, the right time is to find the problems before production, in whatever testing phases you have scheduled.

3. Buffer and protect the timeframe

With waterfall, people will still end up using the testing phases (that are really fixing phases!) as buffer. You need to make room for enough rounds of testing to happen, for this to not ruin all your plans and commitments on testing. And whatever time you have available, the best you can do is to find bad bugs early. Bad bugs will buy you more time. Bad bugs will need the time to get fixed. 

But be careful. In protecting the timeframe, you might have to make choices of how you do that best. You might use your time on talking with people, when actually a better strategy could be to just test some more. Empirical evidence trumps the speculation. 

4. Types of testing and tools

The new and cool stuff applies to you just as much as your more agile counterparts. Need more hands on (shallow) testing? Consider crowdsourcing. Need a tool to help out automating with a specific technology? Follow what new comes in and try not to drown in the overflow of options. 

Do you still write test cases? Have you already considered moving towards session-based test management or some lighter-weight exploratory testing management frame? Or making your test cases, if you must have them, at least some of them, into BDD/SBE-style automation of executable specifications? 

You know, it's not that different after all. Other than the waiting. The powerlessness of introducing change without added politics. The inherent blaming of missing things because you're asked to do things at times you knew the least. 

In waterfall projects, your bug reports are against a fixed deadline. Choose wisely and work to help in any way you can to make sure there's not too much options for you to choose from. The dynamics are different if you release once a year and if you release once a month. 

When two testers meet

Last Saturday, I participated a Code retreat in Vienna, Austria. I wanted to share my favorite moment of the day: meeting another tester.

In a code retreat, we practice programming implementing Conway's Game of Life in pairs with Test-Driven Development and various types of constraints (like data structures, frequency of checking in required, style of pairing). A day fits typically 5 sessions and thus 5 pairs.

I had just finished my first session of the day and a morning break was called, when one of the three women amongst the 25 participants approached me. She had learned that we were both testers, and we set up to pair together on the problem in the next session.

I had great time programming with her. It was one of these experiences some folks tend to refer to as "Reese's pairing" where both parties have ingredients that make the result great but neither has the complete set of things.

** In case you don't know, Reese's are peanut butter + chocolate, an american candy that was advertised as a lucky accident of bringing two great ingredients together to make something even  better. I wouldn't know as I'm neither American or able to each anything with chocolate. 

There were pieces I could bring into the puzzle from past code retreat experiences. With her, I pushed for trying out ApprovalTests and I really liked the way our domain model ended up in the classes. I learned a lot I could use in the later sessions of the day too.

Later in the afternoon, we were sitting with a small group. We got back to the discussion about both of us being testers. My pair was a tester specializing in Java test automation. I was a tester specializing in exploratory testing. What we do for our work has little in common, yet we're both testers to others.  We also work in very different contexts as per ratio of testers to developers and resulting assumption of who would contribute what in the development work.

When two testers meet, it's good to remember that we're not all the same. And instead of us arguing on the essence of which one of us is a true tester, we can just add labels to explain our difference.

Insensible test automation comparisons

In an open space conference, one session was playing on playing with combinatorial testing. I missed the session, but heard about it on the hallways. The message was that they found a problem (failure mode) somewhere at thousands of tests, and kept going up with the numbers to have some impressive amount of different tests created, just in this hour-long session where random people just got together on a problem.

Finding the failure mode was cool in my books. The number of tests run, as impressive as it may be, wasn't.

I started thinking back to this, seeing a tweet from #BTDConf stating on a slide that "It takes at least three times the effort to automate a manual test."

#Automated #Testing is expensive #BTDConf pic.twitter.com/rVBj9DCFuF

— Belgium Testing Days (@BTDconf) June 16, 2016

Just as the thousands of tests was irrelevant information, this isn't much more helpful. I've seen again and again that when there is a nice seam (api with special testability in mind), it can be faster to automate a test than run a similar idea manually. Then again, what I run manually is never exactly the same. Creating the seam slows things down, but adding more similar tests evens out the investment often quite radically.

It just makes little sense to me to compare stuff on a very general level.

Could people just share some very specific examples, instead of the attempt to generalize (and scare with how hard it is)?

And another thing from my experiences: the thing that is expensive when I do it turns out to be very cheap when pairing with a great developer with specific experience.

Yes, learning is expensive. So let's get cracking on it. All I need is everyone to be just a little better every day. On something. Choose something  you like, something that challenges you. And keep trying when it's hard. You are not alone. 

The Squeezed Testing Problem

This post is inspired by two nicely timed incidents:

• a discussion with an aspiring speaker, who was enthusiastic about BDD as a way of testers being able to work continuously throughout the sprint on testing
• a tweet mentioning a common theme of tester complaints: “We’re agile. All the testing is squeezed into the end of the sprint.”

Let’s talk about approaches in tackling the squeezed testing problem. I find this important because, as popular it seems to be, the BDD stuff is not the only way forward. I’m sure the ways forward I’m aware of are not the only ways forward.

So, how could you organize the testing so that it does not end up all being squeezed into the end of the sprint?

Approach 1: no sprints

I find that Scrum and sprints is what people often start from, and having had 10 years of experience with adopting agile, it makes less sense to me now. So instead of starting with Scrum, what if you would go for per-feature “sprints”, except those tend to be called Kanban and Continuous Delivery.
Don’t fall into the trap of thinking you cannot do continuous delivery without test automation. You can!

Try thinking it this way. You have feature X you know is important. Even most important right now. You discuss the feature with your team, and chip away until you find the smallest possible value item you can deliver all the way to production. You then, as a team, do all the necessary work you need to get it delivered. And all the necessary work includes manual programming and manual testing – nothing special there.

The strategies you employ to test might differ a bit. You might pair up with a developer to be there testing as it is being built. You may have a working agreement that you test on a development machine before anything gets into source control. Or you may have a branching model where each fully integrated feature can be automatically built as a system from its own branch that you can test with the developer fixing things as you find them.

Who defined that a month of programming would need to be tested in a day? The whole thing could be a two-month thing instead. When you deliver a functionality at a time, it’s not a big trick to consider the thing done only after it has also been tested and fixed.

Just work on having small functional slices. You really don’t want 2-month feature projects if you can have half-a-day feature deliveries.

Approach 2: forget about the idea of testing before production

I find that a lot of testers are stuck on the idea that system and acceptance testing (that’s what they do) happen before you go live with your software. But you could also look at testing as something you do almost exclusively against your production.

If you plan on using this approach, your organization might be better off having some safeguarding mechanisms on how you roll back or do staged releases (not everyone gets hit with problems at once in large user base).

So when your team is done with the sprint, the system goes live. You test the squeezed testing for the half-a-day you can squeeze in (or can’t), but it’s really not your responsibility to attend to the fact if it works or not. Someone decided that it will get addressed during production use.

The changes are perhaps not that big, after all, they needed to fit into the squeezed development schedule of the sprint. Your testing really starts when the end users using starts – just with the idea that you will tell clearly when you run into problems and pair up with a dev to get the fixes done as soon as possible. The end users might not tell they have problems, and when they tell, getting them to express what they did and what is the problem is a lot of work.

There’s one big problem with this approach. If you need to find a lot of problems, the developers need to fix a lot of problems and they don’t make progress with the things they aspired to in the upcoming sprint. But if it works well, you will find missing value items, ideas to improve the user flow and “missing backlog items” that you can add to your upcoming sprints to improve your product based on the feedback. The stuff you find can wait a sprint. It’s like you’re an empirical extension to a product owner.

Approach 3: shift left

Shift left is the popular idea in agile that you would finally get a practical way of doing the thing waterfall always failed with. Your chances are up because of short increments – helping build the right product one small right thing at a time seems more feasible than hitting the mark on something big in waterfall style.

BDD (Behavior-driven development), SBE (Specification by Example), and ATDD (Acceptance Test Driven Development) all roughly mean the same idea: create examples of behavior in test artifact format, implement the automation while implementing the features and you’ll hit the mark better and need to do less of exploratory testing when what you were building was more clearly designed.

This approach would ask the tester to work on creating the test artifacts (with a product owner and the rest of the team) first as text, and then contribute to automating. When the test artifacts “pass” against the implementation, the assumption is that the exploring needed is small and fits the squeezed timeframe. After all, you were exploring already while designing the feature.

I find that the product is my external imagination, and my best attempts to tell, even for small features all the stuff in advance are limited. But limited is better than not trying to clarify what we’re doing.

Other approaches?

There are a lot of options of tweaking each of these, I’m sure. My main concern is this. If so many testers are struggling with testing getting squeezed to the end, why are so many testers feeling so powerless to do anything about it?

We’re in the business of providing information through empirical work. How about using the empirical evidence to change the status quo to something a little better, one experiment at a time? A good tester has a lot of power. Find the information that matters to the people who matter.

Battleship TDD and ApprovalTests

Last Friday after #DevoxxUK (a java conference), we got together with a small group to try out a TDD mobbing kata. With a bit of discussion on what problems we'd find fun to work with, we ended up with Battleships.

So we draw our first scenario with a Destroyer on a 4x5 board after the little tweak of not having a board that is symmetric.

We write our test in English and translate each line to code. The last line is to check the board. And as  I've previously been introduced to ApprovalTests (as opposed to Asserts), there's a nice flow from the visual representation of our game into an ascii art form of the game.

So instead of sampling assert at a time to represent core parts of the board we draw, we drew the whole thing at once as ascii art, saved into an .approved text file and then let the test guide our implementation.

I've been to different TDD sessions over the years, and I've really grown to like more the Approvals-based approach as more visual and more as a thing that represents world as I know it. I had not really thought it much, but it's an ascii art representation of the thing we're thinking of that we save to a file.

It's nice to be able to say in code

Approvals.Verify(); 

and move the rest of the describing end result into the text file over writing a pile of asserts.

Similarly, I was thinking back to one of my first unit tests with Approvals, where we generated combinations of credentials, dumped them into a file and added nice and clear explanatory test around it. I've appreciated the clarity of that piece of documentation since.

These just seem to map closer to something I feel at home with. So, I keep thinking about it. Maybe it's just me. But if it works for me, it might be of interest to someone else too. 

Database checks help us test

Reading around what people write on testing and test automation, I get the feeling that there are these two big camps of information. There's a lot of stuff on unit test automation, and there's a lot of stuff about system test automation, in particular things like Selenium. It could just be what catches my eye, but I wanted to dedicate a small piece of writing to my one current favorite of test code we run: database checks.


Four years ago, as we were starting our efforts with automation, we focused heavily on unit tests to fail with them in various ways. We could use a lot of time creating them, but with lack of skill we ended up doing tests that lock implementation not behavior, and a maintenance nightmare. In addition, the tests never failed for anything useful. So they vanished.

Two years ago, we then focused on Selenium. The tests found relevant things, and covered ground that developers found somewhat boring to cover. But as the amount of these tests grew, so did the troubles with brittleness. We then identified the subset of tests that would run so that we'd be able to rely on their results.

Less than a year ago, we moved our unit tests from Asserts testing more towards Approval testing. It ended up helping us check complex objects without a lot of maintenance work, and encouraged the team to look for better interfaces to test through.

I don't even remember exactly when the idea of database checks came up. There was just this recurring theme with stuff I'd find that revealed some of our functionalities broke the data. It was painful to test that, because you would only see the brokenness through other functionalities or over a longer time. So we started adding the rules of what the data should be, and made it automatic so that it would alert when I used features that messed up the data, or similarly, when users used features in production that messed up the data.

The tests weren't particularly granular. We could tell who used application in a way that triggered the database checks, but not what they were doing. We could tell the problem happened in last 24 hours, but in production there was always a delay.

Running checks in test environment with what the team was doing was more granular. But even the detective work needed to figure out what needed addressing wasn't impossible - since we knew the work existed with the checks.

Out of all the things we've done for automation, these have helped us the most. The little extensions to what a person has the energy to continuously observe on a database level finds relevant problems.


There is just one big challenge: discipline. Keeping up with the idea of considering detective work a priority, to address causes over symptoms. We're still working in untriggered volunteering of this work to share with the team.

Loving sustainable pace

We're getting closer to two years at work with moving from Scrum-like monthly releases to releasing daily. Back when I drove through the change into #NoEstimates and continuous delivery, I knew the theory that this would be different. Now I know the practice and wouldn't really want to live any other way.

I remember to appreciate the steady, almost uneventful pace we're delivering in because I met a friend yesterday, who reminded me that everyone isn't equally lucky.

My friend is a relatively new developer, with about a year of project experience behind her now. She mentioned how her first project was a waterfall project, and how she was struggling to keep up with the pace. Not because she was new, but because of big promises early on with little room to adjust. She mentioned how she felt that going for lunch seemed something you couldn't do, and that it was hard to leave the office on time, and impossible to make it to user groups like the one we met in last night.

She also mentioned that things had recently changed for her, as she joined a project identifying as agile. Small iterations and all of a sudden, the panicky skipping lunches turned into a steady pace of delivery.

I appreciate a lot her sharing her success with agile on feeling in control of her time again. It reminded me how much I value the sustainable pace and lack of fires around me.

Small batch size in delivery is great and helps create a pace that we can sustain.