Sunday, August 4, 2024

The Driver-Navigator in Strong-Style Pairing

This post has originally been published as Medium post on August 26th, 2018. Since Medium did not win on my choices of blogging platforms, I am including it here in efforts to consolidate my legacy.  This post has 12k views on Medium.

There’s no better way of learning programming or testing than working together with people, building on one another’s insights. Well, it does not always feel like the best way if the power dynamics are off and the skills and understanding of how to work well in a pair are not in place. Let’s talk about the two roles a bit.

In a pair, we have a driver and a navigator. Think of these roles as they are in driving rally. The driver is hands on the wheel, and in control of paying attention to the immediate task at hand. They are busy looking at the road ahead, optimizing the drive and not checking the map. Checking the map would be a distraction. That is where the navigator jumps in. Keeping track of goals on a larger scale, paying attention to other level of details they can help significantly in getting to better results. Navigator takes care of directions and instructions, and feeding them to the driver as needed.

This same distribution of tasks is what is the core of strong-style pairing. Instead of making the navigator decipher what is going on in the drivers head, the control is placed with the navigator. For strong-style pairing, we say:

“I have an idea, you take the keyboard”.

For traditional pairing, we’d say: “I have an idea, give me the keyboard”.

Getting Set Up For Pairing

For the pair to work well, both parties in the pair need to work well in their roles, and there’s specific skills and techniques you can learn to work better from get-go with a new pair in either one of the roles. Techniques, however, are just ideas until you apply them in practice. Only through doing you can learn to pick up the small hints on what would be right thing to do, as you both are equally responsible for your mutual success.

We may think of the driver as an intelligent input device. Intelligence means that while the driver takes orders on what to do and where to go, the driver can also guide to get on the right abstraction level or improvise within the box the navigator gives them.

The navigator is responsible for mining the to-do list and passing the next things to do with instructions on the highest possible abstraction level the driver can consume.

This all builds on trust. The navigator may be just one step ahead of you, and thus unable to give you a clear overview of what you’re about to do. The direction the navigator is going to is where the driver goes. When the driver has an idea, the roles can be switched keeping a basic rule in mind: for an idea to get on the computer, it must go through someone else’s hands.

Who Should Drive and Who Should Navigate?

You have a pair of people. You are different people, with different ideas and experiences. The two of you are stronger together because you build on each other, and also because each of you have unique qualities. Sometimes the qualities are more of a foundational or long-lasting kind, like experience that does not build up over night or transfer in a few days of pairing. Sometimes the qualities are more of temporary, like one having a rough day just today.

One of you will take the role of the Driver — the intelligent input device. One of you will be the Navigator, working out the direction you’re about to head to together. Which of the two of you should be which?

The first rule on being the navigator would be that whoever has the idea toget on the computer, should navigate. Remember: for an idea to get on the computer, it must go through someone else’s hands in strong-style pairing. This brings quickly the idea that perhaps the more experienced one should navigate. At least first. At least until it’s time to switch. You learn very different things in being a driver and being a navigator.

The dynamic of newcomer and a seasoned professional is interesting. Having the newcomer just drive is already a relevant service for the navigator, freeing you from typing to think on a bit higher level. The information for the newcomer sticks in a completely different way from doing it, hands on the keyboard, than watching what someone else is doing.

Whoever had the idea of what to do will navigate. When I started pairing on exploratory testing with a developer, I was the Navigator. When I started pairing on unit tests with a developer, I was the Driver. Over time with both tasks, I will take both roles. And I will grow towards navigating so that the first idea that I have that I could navigate us through, I volunteer to do.

There’s three trigger options on change of roles:

  • On time: you can just change roles on a regular interval. Start with shorter times and grow the time as you grow to work as a pair.
  • On task: you can change roles when some task is completed. Like a typical ping-pong style pairing, you change so that one creates a unit test and other implements, and both get to do both types of tasks.
  • On idea: you just change when the driver feels there’s a direction he’d like to navigate to. It’s like saying “hey, I have an idea, you take the keyboard now”.

Regardless of the role you’re in, the pair of you should remember to take breaks. Pairing can be very intensive work and outside getting the work done in a pair, you’re also responsible for building forward your pairing experience. And that usually works best by inviting for feedback, like taking a small walk around and retrospecting your most recent pairing experience together.

Driver: Things to Do

You are the intelligent input device. What is expected of you except for typing as you’re told?

  • Push back. Express that you need to modify the way the navigator is navigating you. You may want to change the box in which you work, make it bigger to give you more freedom or make it smaller to be clearer on what to do. You may realise you need to try something else, and express that in questions. Key is to be active even when driving.
  • Improvise. The navigator gives you a box within which you operate. You have choices on how you can do things. You choose what you believe to be best for context at hand, and driver gives you more detail if they disagree with your choices. Improvising is about adding your intelligence to the collaboration.
  • Switch level of abstraction. If you feel the abstraction level of navigating could be higher or lower, talk back to the navigator. If the navigator is using too low level, you can give them the higher level. “You can just tell me to run it”, when navigator is telling you shortcut in keystrokes. If the navigator is using too high level, you can ask “Tell me what to type?” or “Where’s that located?”. Change the level of abstraction and enable a common experience of learning to work together better.
  • Ask questions. If you feel something isn’t right, ask about it. You can simply go for “Are you sure?” to stop the navigator to think about what is going on. Keep your questions specific so that the answers can be short. You can try validating questions to clarify what is going on right now, “It seems we’re using zip-add-object to regulate temperatures, is that correct?”. You can check if the thing you did was correct against your understanding: “I thought the database only accepted one connection at a time. Why did we do two?”. You can suggest where to go, without deciding for the navigator: “Shouldn’t we do this first?”. Avoid general why-questions and work to prevent long explanations.
  • Initiate role switch on an idea. When you realise what should be done and think you could navigate this task better, initiate a role switch. You can switch roles on idea, without waiting for the timer. Or you could never use a timer and switch on tasks.

It’s good to remember that you as the driver are helping the navigator to navigate. Sometimes navigators will try a general avoidance technique of declaring tasks mundane, and great driver will volunteer to be around even for that task. Sometimes you are just literally trying to go against the excuse. Sometimes the two of you just need to get through the bad stuff together to get to the fun stuff together, to build the long-term relationship.

Underneath what we actually do as driver, there’s a bunch of attitudes to consider:

  • Trust your navigator. Stay in the moment. Be ready to work with partial knowledge. Your navigator is probably only one step ahead of you.
  • Just try it. You can always do it both ways and end up with a third that builds on top of both. Learn by doing. Learning while doing is just as important as getting the end result out of the process.
  • Constant self-reflection. Investigate what is going through you. Focus on learning. When following your navigator, you’re reverse-engineering. Navigator keeps telling you stuff. You get a very thin, narrow view into the system. You can start putting things together a lot and this gives you a chance to reflect on what you’re learning individually as driver with your pair.
  • Patience. Give the relationship time. It’s good on both sides, but it’s extra useful on the driver side who works with incomplete knowledge.

There’s two main pitfalls to being a driver.

  • Thinking. You think within the box or change the box. Thinking too far as the driver takes the navigator’s focus on bringing you back to the task at hand.
  • Silence. It’s not an open box to do anything you think of. If you want to take the lead, switch roles, but try not to run your own way with the silence.

Navigator: Things to Do

As navigator, you are in responsible for caring for your driver. If the driver is the intelligent input device, for them to operate properly, you need to care for the conditions of work. Being the navigator, you need to pay attention to your driver, to constantly know where they are going. And you need to enable them to go as fast as possible.

As a navigator, you have three main tasks you’re paying attention to:

  • Feed driver the next thing to do. You’re creating the driver the box they work in on as high abstraction level as your driver can handle.
  • Mine the to-do-list. Create an idea on where to go next. You might be just one step ahead, but being that one step ahead is important. Where to go to get your thing done?
  • Observe your driver. See where they are and where they are going, and correct if the direction does not match what you had in mind. Pay attention on how they are doing, and help them whenever they need it.

Three main tasks might sound simple, but there’s a bundle of advice to do them slightly better.

Programming style matters

The style of programming matters. Pair programming would seem to work a lot more fluently if the programming style is consume-first. With this the idea is that you start with an end result at first and then one by one build the things you wish you had in order to have the end result. Consume-first enables the navigator to go immediately, coming up with things to do while figuring things out themselves. And the end result and it’s division keep a visible checklist of what there is to do to get to the end result.

Some navigators prefer to work with bottom-up programming style. They build the image of the end result in their head, and feed the smallest possible pieces to the driver, one at a time. In this style, the navigator has a lot more of the information about what we’re trying to achieve, leaving the driver to interpret more of why the navigator is having them do things.

The third style is hardest, when there’s no ability to work in small pieces. If the navigator has to figure out the whole thing before feeding driver work, most of the time the driver is paused or participating in design discussions over taking the implementation forward. As a navigator, you’re supposed to care for your driver, not just having them type for you. Keeping driver waiting while navigator figures things out by themselves isn’t exactly the optimal way of caring. Working with partial information is essential.

The abstraction level dilemma

As the navigator feeding the driver the next thing to do, finding the right abstraction level to communicate with them is relevant, even key to making progress. For driver to go forward, you need to find the right level on which to talk. If you are using an abstraction level too high, you will see nothing happening at the keyboard. If you are using an abstraction level too low, you’re not harnessing the powers of your driver by keeping them on too short a leash.

The core of going as fast as possible is navigating on the highest level of abstraction. Talking in keystrokes when talking in concepts would suffice can feel insulting. So pay attention to raising the level of abstration.

  • Intent. What is the thing we want to accomplish now?
  • Location. Where does doing that start? Is the cursor in the right place or moving to the right place so that what we want done could be done?
  • Details. What exactly to do or type so that the intent is fulfilled, allowing for the drivers contribution while avoiding mistakes?

You notice if the level of abstraction is too high with the puzzled look on their face and the lack of action you expect to see. Drill in if needed.

I was particularly puzzled with the idea of using highest abstraction level possible, until I realised finding the level is really a listening and observation exercise. If you give instructions and following them is hard, you’re probably working on a too high abstraction level. Drilling down can be instant, just be more specific. If you’re not noticing the need to change level, you’re perhaps risking an experience of failing with tasks that lowers motivation in pairing, so it’s good to keep your eyes and ears open. If you start from a very low abstraction level, the driver can always correct you by telling you the level they are comfortable with. But if they have not yet learned how to, they might get a feeling of being talked down. And I found I am particularly sensitive to that, being the only woman and paying attention to being treated differently.

Mining the to-do-list

There needs to be an idea of where we’re heading. But the road ahead might be only clear for the next few steps, instead of all the way. An important thing for a driver is keeping track of the work ahead. There’s three main things to consider on this:

  • Timing: Find the right time to use an item on the list. What should be done first, what would make a coherent shared story in your pairing. What choices would enable you as the navigator to care for your driver in the best possible way?
  • Backlog: what is there on the list of things to do? Your backlog is best if it can somehow be part of the code and become a shared view — with consume-first style. But you can also make notes by scribbling on a piece of paper of a whiteboard. Whatever you need to keep track of things. Notes are disposable, code (including test code) is what remains when you’re done.
  • Prioritisation: Deciding what comes next and in what order to do things. It’s not just about the right time in long term, but the right thing right now. And it keeps changing as you learn.

Express a in-a-nutshell idea of what you’re doing

As a navigator in strong-style, you are not supposed to have to justify all your chosen actions to your driver. Sometimes the driver has little clue on where you’re about to be heading, and a great practice in these cases seems is to invite temporary trust saying you’d like to just go to this direction for like 10 minutes, and if they are still unsure about doing this, you can change then. The argument of what is right thing to do takes easily more time than that. And nothing stops the pair of you implementing both of your ideas and then deciding that a third, completely different option is what you should go for. With programming in particular, there’s a lot of uncertainty that unfolds only through implementation and experimentation.

If you need to express what you’re doing, you should learn to express that in a very concise format. Instead of all the rationale, pick only the part of the message that is absolutely relevant to care for your driver.

Immediate feedback

When you navigate, keep an eye on your driver. Double-check with questions what the driver does. And if the driver does something you consider a problem, help them correct it right away. Feedback belongs with the action the feedback relates to.

Closure Activities

Recognizing time to switch

Either one in the pair can suggest a switch of roles. But as a navigator, you are usually in a controlling position, so your position gives you a bit of extra responsibility to pay attention to when the driver would have ideas that she should navigate on. You want, in the long term, to build a pair where both can contribute. Switching pairs gives a chance to try out the other role, and only practice helps you get better.

Step back and think about what you’re doing

Sometimes you may feel you get bogged down into the details of implementation, and you might want to take a step back and think if you are spending your efforts in the right area. You may not notice without intentional step back to look at the bigger picture, because you’re going fast with support of your pair (and usually a nice bunch of unit tests too).

Retrospectives

Looking at your collaboration and how the two of you feel about it on regular intervals is also necessary. Not just looking at what you implemented and stepping back on that, but talking about what you learned and what you like and dislike about your collaboration, in a constructive tone.

Ending this with a word of warning

I’ve started to really enjoy looking into the dynamics of this style of pairing. But there’s a story I learned, that I also find very specific to this style.

Two programmers were pairing in strong-style and had the time of their life. There was a lot of talk, as all ideas must go verbally for two people to share. Sometimes the discussions were very enthusiastic and to an outsider they might have appeared even heated.

While the pair was enjoying, the environment that did not recognise the different pairing style was betting on which one of the two in the pair would quit first.

Don’t care just for your pair, care also for your environment. Verbalising your thoughts makes them available for more people than just the two of you. Take it as an opportunity to build relationships outside your immediate pair.

And it is not always that the pair is enjoying. Your pair not opting to work with you again is feedback that the way you’re connecting isn’t appropriate. A pair is two people, where both need to feel comfortable.

Styles of Pair Testing

This post has originally been published as Medium post on August 18th, 2018. Since Medium did not win on my choices of blogging platforms, I am including it here in efforts to consolidate my legacy. This post has 2.1k views on Medium.

You must have heard about pair programming and it’s testing sibling, pair testing. What you might not have realized that there are two significantly different styles of pairing. We call one traditional and the other strong-style. The selection of style becomes particularly relevant when there are background and skill differences in the pair — let’s make educated choices and learn about both styles!

Individual Work

Let’s first think of testing as individual work. Core to the work is the a skilled tester, with ideas of what to do to uncover information. They use the keyboard to drive the computer, and as ideas emerge and evolve, write down notes in a tool of their choice. There’s just one person for all the activities around the task, and no need for changing roles other than within the tester’s head.

Working alone leaves all work on one brain.

Traditional Pairing

In traditional pairing, we introduce a second person and split the work to do. We introduce roles and rotation, either on task or on timer. The work is split so that whoever is on the keyboard is in control of the testing session. The one on the keyboard has the ideas, and focuses on tactical implementation of those. The other one thinks more strategically, and makes notes. We often suggest that the one on the keyboard must think out loud to allow the pair into the insights of what goes on in the testers head. After all, very small portion of testing happens at the keyboard and can be visually followed.

The problem with this style of pairing is that the one without the keyboard is continuously decoding what they are observing from the other’s testing, building assumptions. It is hard to see what exactly is going on and this easily creates a bit of a disconnect in the pair. At worst, the disconnect shows as the watcher zoning out and not paying attention. Regularly, it shows as the notes not matching the testing that was done.

Traditional pairing: I have an idea, give me the keyboard!

Strong-Style Pairing

In strong-style pairing, we split the work differently. The one who is not on the keyboard is now the one with the idea. They must vocalize the idea for the one on the keyboard. The one on the keyboard asks for clarifications and can challenge the direction, suggest ideas and ask questions. However, decision power is on the one without the keyboard. Notes are also taken on the same computer, representing ideas and initiated by the one not on the keyboard. As they are typed in and reviewed, they become a shared representation of the idea.

The benefit of this style of pairing is that the one without the keyboard must vocalize all their ideas. This style creates a stronger connection in the pair. Speaking about what you want to happen on the keyboard is a skill of its own, and it takes a bit of practice to get used to. The person on the keyboard sets the pace in which things happen within the pair.

Strong-style pairing: I have an idea, you take the keyboard!

Different Skillsets, Unequal at a Task

Strong-style pairing becomes particularly useful and necessary when pairing amongst different skillset so that your level of skills and knowledge is very unequal at a task. The difference could come from having product and domain expertise, while the other one would usually test another product. Or it could come from pairing a tester with a programmer. With traditional pairing, the one with more knowledge on the task at hand would be on the keyboard. The skills difference makes following what goes on harder in general.

With strong-style pairing the one with more knowledge is hands off keyboard, and the less knowledgable sets the pace of how fast they can absorb the pieces of the task.

Strong-Style pairing helps distribute the work and the control so that both are active.

Ensemble Testing — Pairing on Turned Up

Strong-style pairing is also the foundation of ensemble testing (and ensemble programming). In this style of working, you have group of people working on one task, taking turns on the computer. Ideas come from the crowd not on the computer. This is about getting the best out of everyone into the work we are doing. Everyone takes turns at the keyboard, and if you have someone unequal at task, the ensemble helps out navigating at an appropriate level of abstraction.

Mob Testing brings in more people for the ideas side using Strong-style to communicate.

There’s a lot more on the dynamics of pairing and ensembling. Check out the Ensemble Programming Guidebook I’m writing, it is available on https://ensembleprogramming.xyz.

For me adjusting to strong-style is what made pairing turn from frustration to learning fun and something I’d volunteer for. Similarly, ensembling was the gateway to pairing making me comfortable in a group before being alone with each of my team mates. I encourage you to experiment with what works with you.

Exploring Gilded Rose

This post has originally been published as Medium post on Aug 12th, 2018. Since Medium did not win on my choices of blogging platforms, I am including it here in efforts to consolidate my legacy. This post has 2.8k views on Medium.

There is a lot of talk around testing — who will do it, when it needs to happen, boxes it needs to fit in — yet not enough on the actual testing. This is the first article in the series of looking at software to test, and figuring out how to test it. This article is based on the experiences I’ve had watching people test as I coach and teach them using these examples.

Introducing Gilded Rose

Gilded Rose is a refactoring Kata (practice) created by Emily Bache. The idea with Gilded Rose is simple. There’s an inn somewhere that has an inventory system implemented. They would want it extended for new requirements but that won’t be all straightforward. Don’t break anything that used to work!

I’m a tester, so I don’t naturally come to the problem as it has been given, but I come with the idea that after someone changes it I may have to test it. How would I do that?

Priming With Information Sources and Tools

At work, things don’t come to you with the full range of sources and tools readily handed in. For doing the exercise, I no longer drop people in cold to “just figure it out” but I give them a few starting points.

  • Requirements. Gilded Rose comes with a requirements specification. Would that be of use?
  • Code. It “works in production” now and you can look at the implementation. If Java isn’t your cup of tea even if I use it while I teach this, Emily has been nice to provide it with tons of other languages.
  • Unit test. The one unit test gives a starting point of how to execute the code so that you don’t need to figure that out.
  • IDE with visual code coverage. Code and unit test in an environment where you can start working on it, with a code coverage tool. I use Eclipse with Code Coverage as I just like the visual nature of it showing green, yellow and red for the branches.
  • Ideas of the domain. You have past experiences of what makes sense for a shop and inventory management system. You have ideas of what inputs are meaningful, and what could be confusing. Everything you’ve learned about what makes software worthwhile is with you.

To make the exercise slightly more tester friendly and approachable in coaching people who never work with code, I extracted a method out of the original unit test.

If you want to try things out before spoilers, pause here and go do the exercise. Figure out what your test cases for it look like and why they are the way they are.

Getting to Work — How Would I Test this?

We’re approaching the exercise with exploratory testing, and all of our options are open. What makes this exercise particularly exploratory is that I will rule out the option of going away to your cubicle to write test cases based on the specification without running a single test. I expect you to design your tests as you go and allow you to learn rather sooner than later.

Unlike for me right now writing this article after having done the exercise, you are now at a time you know the least. You know there’s two information sources and either specification or code could be your starting point. If you are a tester by trade, you are likely to lean towards the specification and if you are a developer by trade, you are likely to lean towards the code and coverage tools.

As the exploratory tester, you are on the driver seat. You get to choose which way you go. And any mix is possible. There is no recipe. But there is options.

Just Try It

You could just forget about the specification for now, as well as not read the code that implements this, and start playing with values you can enter into the CheckItem-method. It takes three inputs:

  • a name (String item)
  • a number of days we sell the item (Integer selling)
  • a number indicating how valuable it is (Integer quality)

If we didn’t look at the specification, deducting this much info out of the interface is unlikely. We would just see it takes text and two numbers. But that is enough to play with it! This brings you to the problem with least structure, highest chance of getting confused and highest chance of learning something outside the spec and the code.

You could look at the problem with the simple ideas around the inputs. What if the input is really really long? What if the number is really really big? Oo, negative numbers? Special characters? All the things forbidden? Hey, there’s numbers that are special: what if your input is 00100, is that 100 or something different?

Read the Specification

Exploratory testing does not mean you have to jump in without considering any of the sources. It means you are intentional about what you choose, and you combine things in ways that keep you engaged as well as ensure you do a good job tracking coverage and meaningfullness of your work. Reading the specification gives you one way tracking coverage.

The specification is full of claims. Some of them are clear. Some will turn out not so clear when you’re testing and seeing what the application does. Some lines are single claims, some are have multiple claims in them. Some lines stand on their own, others depend on other lines of text. Not all lines are meaningful at all.

Which one do you choose to start from? How do you know? Actually, you don’t know. So you sample something and hope to have made a selection that leads you to understanding rather than confusion. And that if it leads you to confusion, you get out of there with later samples.

Read the Code

You could also choose to read the code. You could choose to introduce some tests that enable you to step the logic through in a debugger so that maybe you could see some patterns in how it is implemented. Maybe you just read it without executing so much. Read line by line, or read paying attention to some aspects like variable names or values the code checks against.

Code is the ultimate truth, what is not in the code that now “works in production” isn’t working in production. The spec and the code can be in sync or not, but if they disagree, the code wins until it gets changed.

Think about the environment

The program you’re supposed to test is probably intended for some use by some people somewhere. That somewhere most likely isn’t the test machine you’re using now, and the end user interface most likely isn’t going to be the method you have your hands on now.

What would change in the way you look at the application of the environment was different? Would someone try running it concurrently? Would it need to speak to an external system? Could it be limited to run with miniscule amount of memory, or in an environment that does not allow Java Virtual Machine to be running?

The First Test

There is no absolute choice for the first test, and having tested this with a good crowd of people both individually and in mobs, some people still make a different choice for the first test in the setup we test in.

For me, the first test is to see we can actually test. Running the test that has been given to us as an example. The test that reveals our ability to run any consequent tests.

There has been days of going into doing the exercise where this test fails, because I accidentally cleaned up more than I should after the previous run through the exercise.

Generally, this test results in a green bar.


As we are exploratory testing, we learn that the interface provided is good for us to go further. We learn that the tool reports us with green when a test is given that passes. We might even read the test to figure out that it says that when we start with item named “Any”, no days to sell it and quality of zero, the quality isn’t changing anywhere.

The Second Test

With the second test, we arrive at the significant divergence of choices.

  • the freeform value exploration
  • the specification
  • the code through measurable coverage
  • the context of use

The real choice is actually to realize they are all different dimensions and for properly testing this, some work on all might need to happen. Some tracking of coverage on all of them would need to happen. Some discipline in not abstracting results of one to fully cover the results of another would need to happen.

What I often see with the choice of the second test is that some people choose to just pay attention to the code, and end up missing out on all the problems the specification leads you to uncover. Some people choose to pay attention only to the specification and report problems that aren’t really problems and fail to cover the code. And some people just don’t feel like they are empowered to add anything beoynd the given artifacts, which is detrimental to their ability to uncover yet another category of problems.

Reading the Spec While Aiming for 100% Branch Coverage

Let’s assume we intentionally, not accidentally, chose to approach this code and code coverage first, with the help of the specification.

Here’s tests from one group to get to 100% branch coverage. 18 tests. Took two hours in a mob.

At first they tried giving good names, but knowing the least to begin with, the names were not really good. Halfway through they got tired of trying to think of names and just gave up thinking and trying to understand other things and focused on getting through the coverage of specification and code.

In every single test their assert was on quality, and they never see a requirement around how sellin-number would behave in case of legendary items.

Power of the group lead them to pay attention to the specification and they found the discrepancies there:

  • The lack of input validation implied by many requirements
  • The shorthand of naming items in the specification in comparison to full names used in the code
  • The fuzziness around limits the rules defined that behavior would change at

By the time they were done, they had worked significantly. Yet they called done before it was time. No cleaning up the names, no looking for rules they might have missed.

Tools Ease The Exploration

To contract to the 18 handpicked tests over 2 hour intensive work, I’ll show you a few minute example of just covering the code with ApprovalTests-library.

This tool includes a possibility to pass a group of values and automatically generate combinations of those values. The generated tests are pushed into a text file where we can visually verify and approve them. Within the minutes version, I would use the principle of them all being correct because this is legacy code that Works in Production. Within minutes, I generated 41616 tests to get to 100% branch coverage, and to run them again to make sure nothing breaks it takes 1.028 seconds to run. The code to do that is four lines and I go with the ultimate lazy of not even hand-picking relevant integer values but using all between -1 and 100.


If you miss the copy-pasteable example of this, I put in a gist. You may notice I needed to do one change to the original method so that the returning object would have a toString() to write to file. What was object type GildedRose in the first example, is no object type Items in the latter as it already had the necessary toString() defined.

With the 41616 tests, I have now a list of things I can verify with specification. Obviously I would sample heavily over checking them all. But with then visible, I can run into problems serendipitously. I can notice that for

[Sulfuras, Hand of Ragnaros, 27, 23] => Sulfuras, Hand of Ragnaros, 27, 23

the second value, sellin, isn’t in fact changing. I could visually and conceptually compare it to another item

[Foo, 1, 10] => Foo, 0, 9

Seeing that for all others but the legendary item Sulfuras, as speficied, “never has to be sold” in requirements means the sellin date won’t be changing.

Summing up to some fun pitfalls

For a little exercise like this, it has surprising dimensions. To end this article, I wanted to share lessons learned with one newbie tester who found out there was a lot to learn.

They did not get bit by missing the first test, because they were lucky enough to have it in a safe and good green starting point.

They started creating their tests from the end of the specification, where many claims depend on each other. They assumed they spotted an easy claim to verify which I find a fascinating judgement call of what is easy and what is difficult. Defining a test around that claim lead them to learn things that were not true, and double the time to finish the exercise to include unlearning things they had started to firmly believe in.

The lessons were:

  • First test was a poor choice leading to long-term confusion, we could either choose differently or pay attention to what we really know more actively.
  • Any coverage will do, some here some there and testing is done! That is true, but tracking how much done are you is a big part of testing. Even the code coverage can fool you because the number shows line coverage and the colors show branch coverage. Calling it done on line coverage left out lines leading to new branches.
  • Naming coverage dimensions isn’t a thing everyone does, they didn’t. Conceptualizing code, specification, environment and risk coverage is a necessary thinking model.
  • Code coverage can be achieved without proper asserts and gives a very false sense of security. You don’t even need to assert values of quality for the 16 tests above to have them still run to 100% branch coverage.
  • Problems against specification were all assumed distinct problems, no grouping around the fact that there was full areas of “I expect input validation” that were not implemented anywhere.
  • Specification having shorthand like saying “Sulfuras” instead of “Sulfuras, the Hand of Ragnarok” isn’t a major problem with the specification even if it bit them in the exercise. Best of specifications are helpful, yet incomplete. Best of testers using specifications don’t need them to be complete to do complete testing.
  • When results did not match the specification, quickly jumping to conclusion that we are testing a buggy software which wasn’t the case. The models to pinpoint whether problem could be in my tests that I have control over were not in place. We practiced many rounds of “is there another test we could do that would give us a second data point to verify we understood the requirement”.
  • One test one requirement isn’t a thing. They are a messy network of dependencies.

There is no easy recipe for testing any application. Stop to think. Approach from different angles. Check and double-check. And don’t hide in your corner, maximise your chances of getting to the right information by working with the others.