Sunday, August 4, 2024

Exploring Gilded Rose

This post has originally been published as Medium post on Aug 12th, 2018. Since Medium did not win on my choices of blogging platforms, I am including it here in efforts to consolidate my legacy. This post has 2.8k views on Medium.

There is a lot of talk around testing — who will do it, when it needs to happen, boxes it needs to fit in — yet not enough on the actual testing. This is the first article in the series of looking at software to test, and figuring out how to test it. This article is based on the experiences I’ve had watching people test as I coach and teach them using these examples.

Introducing Gilded Rose

Gilded Rose is a refactoring Kata (practice) created by Emily Bache. The idea with Gilded Rose is simple. There’s an inn somewhere that has an inventory system implemented. They would want it extended for new requirements but that won’t be all straightforward. Don’t break anything that used to work!

I’m a tester, so I don’t naturally come to the problem as it has been given, but I come with the idea that after someone changes it I may have to test it. How would I do that?

Priming With Information Sources and Tools

At work, things don’t come to you with the full range of sources and tools readily handed in. For doing the exercise, I no longer drop people in cold to “just figure it out” but I give them a few starting points.

  • Requirements. Gilded Rose comes with a requirements specification. Would that be of use?
  • Code. It “works in production” now and you can look at the implementation. If Java isn’t your cup of tea even if I use it while I teach this, Emily has been nice to provide it with tons of other languages.
  • Unit test. The one unit test gives a starting point of how to execute the code so that you don’t need to figure that out.
  • IDE with visual code coverage. Code and unit test in an environment where you can start working on it, with a code coverage tool. I use Eclipse with Code Coverage as I just like the visual nature of it showing green, yellow and red for the branches.
  • Ideas of the domain. You have past experiences of what makes sense for a shop and inventory management system. You have ideas of what inputs are meaningful, and what could be confusing. Everything you’ve learned about what makes software worthwhile is with you.

To make the exercise slightly more tester friendly and approachable in coaching people who never work with code, I extracted a method out of the original unit test.

If you want to try things out before spoilers, pause here and go do the exercise. Figure out what your test cases for it look like and why they are the way they are.

Getting to Work — How Would I Test this?

We’re approaching the exercise with exploratory testing, and all of our options are open. What makes this exercise particularly exploratory is that I will rule out the option of going away to your cubicle to write test cases based on the specification without running a single test. I expect you to design your tests as you go and allow you to learn rather sooner than later.

Unlike for me right now writing this article after having done the exercise, you are now at a time you know the least. You know there’s two information sources and either specification or code could be your starting point. If you are a tester by trade, you are likely to lean towards the specification and if you are a developer by trade, you are likely to lean towards the code and coverage tools.

As the exploratory tester, you are on the driver seat. You get to choose which way you go. And any mix is possible. There is no recipe. But there is options.

Just Try It

You could just forget about the specification for now, as well as not read the code that implements this, and start playing with values you can enter into the CheckItem-method. It takes three inputs:

  • a name (String item)
  • a number of days we sell the item (Integer selling)
  • a number indicating how valuable it is (Integer quality)

If we didn’t look at the specification, deducting this much info out of the interface is unlikely. We would just see it takes text and two numbers. But that is enough to play with it! This brings you to the problem with least structure, highest chance of getting confused and highest chance of learning something outside the spec and the code.

You could look at the problem with the simple ideas around the inputs. What if the input is really really long? What if the number is really really big? Oo, negative numbers? Special characters? All the things forbidden? Hey, there’s numbers that are special: what if your input is 00100, is that 100 or something different?

Read the Specification

Exploratory testing does not mean you have to jump in without considering any of the sources. It means you are intentional about what you choose, and you combine things in ways that keep you engaged as well as ensure you do a good job tracking coverage and meaningfullness of your work. Reading the specification gives you one way tracking coverage.

The specification is full of claims. Some of them are clear. Some will turn out not so clear when you’re testing and seeing what the application does. Some lines are single claims, some are have multiple claims in them. Some lines stand on their own, others depend on other lines of text. Not all lines are meaningful at all.

Which one do you choose to start from? How do you know? Actually, you don’t know. So you sample something and hope to have made a selection that leads you to understanding rather than confusion. And that if it leads you to confusion, you get out of there with later samples.

Read the Code

You could also choose to read the code. You could choose to introduce some tests that enable you to step the logic through in a debugger so that maybe you could see some patterns in how it is implemented. Maybe you just read it without executing so much. Read line by line, or read paying attention to some aspects like variable names or values the code checks against.

Code is the ultimate truth, what is not in the code that now “works in production” isn’t working in production. The spec and the code can be in sync or not, but if they disagree, the code wins until it gets changed.

Think about the environment

The program you’re supposed to test is probably intended for some use by some people somewhere. That somewhere most likely isn’t the test machine you’re using now, and the end user interface most likely isn’t going to be the method you have your hands on now.

What would change in the way you look at the application of the environment was different? Would someone try running it concurrently? Would it need to speak to an external system? Could it be limited to run with miniscule amount of memory, or in an environment that does not allow Java Virtual Machine to be running?

The First Test

There is no absolute choice for the first test, and having tested this with a good crowd of people both individually and in mobs, some people still make a different choice for the first test in the setup we test in.

For me, the first test is to see we can actually test. Running the test that has been given to us as an example. The test that reveals our ability to run any consequent tests.

There has been days of going into doing the exercise where this test fails, because I accidentally cleaned up more than I should after the previous run through the exercise.

Generally, this test results in a green bar.


As we are exploratory testing, we learn that the interface provided is good for us to go further. We learn that the tool reports us with green when a test is given that passes. We might even read the test to figure out that it says that when we start with item named “Any”, no days to sell it and quality of zero, the quality isn’t changing anywhere.

The Second Test

With the second test, we arrive at the significant divergence of choices.

  • the freeform value exploration
  • the specification
  • the code through measurable coverage
  • the context of use

The real choice is actually to realize they are all different dimensions and for properly testing this, some work on all might need to happen. Some tracking of coverage on all of them would need to happen. Some discipline in not abstracting results of one to fully cover the results of another would need to happen.

What I often see with the choice of the second test is that some people choose to just pay attention to the code, and end up missing out on all the problems the specification leads you to uncover. Some people choose to pay attention only to the specification and report problems that aren’t really problems and fail to cover the code. And some people just don’t feel like they are empowered to add anything beoynd the given artifacts, which is detrimental to their ability to uncover yet another category of problems.

Reading the Spec While Aiming for 100% Branch Coverage

Let’s assume we intentionally, not accidentally, chose to approach this code and code coverage first, with the help of the specification.

Here’s tests from one group to get to 100% branch coverage. 18 tests. Took two hours in a mob.

At first they tried giving good names, but knowing the least to begin with, the names were not really good. Halfway through they got tired of trying to think of names and just gave up thinking and trying to understand other things and focused on getting through the coverage of specification and code.

In every single test their assert was on quality, and they never see a requirement around how sellin-number would behave in case of legendary items.

Power of the group lead them to pay attention to the specification and they found the discrepancies there:

  • The lack of input validation implied by many requirements
  • The shorthand of naming items in the specification in comparison to full names used in the code
  • The fuzziness around limits the rules defined that behavior would change at

By the time they were done, they had worked significantly. Yet they called done before it was time. No cleaning up the names, no looking for rules they might have missed.

Tools Ease The Exploration

To contract to the 18 handpicked tests over 2 hour intensive work, I’ll show you a few minute example of just covering the code with ApprovalTests-library.

This tool includes a possibility to pass a group of values and automatically generate combinations of those values. The generated tests are pushed into a text file where we can visually verify and approve them. Within the minutes version, I would use the principle of them all being correct because this is legacy code that Works in Production. Within minutes, I generated 41616 tests to get to 100% branch coverage, and to run them again to make sure nothing breaks it takes 1.028 seconds to run. The code to do that is four lines and I go with the ultimate lazy of not even hand-picking relevant integer values but using all between -1 and 100.


If you miss the copy-pasteable example of this, I put in a gist. You may notice I needed to do one change to the original method so that the returning object would have a toString() to write to file. What was object type GildedRose in the first example, is no object type Items in the latter as it already had the necessary toString() defined.

With the 41616 tests, I have now a list of things I can verify with specification. Obviously I would sample heavily over checking them all. But with then visible, I can run into problems serendipitously. I can notice that for

[Sulfuras, Hand of Ragnaros, 27, 23] => Sulfuras, Hand of Ragnaros, 27, 23

the second value, sellin, isn’t in fact changing. I could visually and conceptually compare it to another item

[Foo, 1, 10] => Foo, 0, 9

Seeing that for all others but the legendary item Sulfuras, as speficied, “never has to be sold” in requirements means the sellin date won’t be changing.

Summing up to some fun pitfalls

For a little exercise like this, it has surprising dimensions. To end this article, I wanted to share lessons learned with one newbie tester who found out there was a lot to learn.

They did not get bit by missing the first test, because they were lucky enough to have it in a safe and good green starting point.

They started creating their tests from the end of the specification, where many claims depend on each other. They assumed they spotted an easy claim to verify which I find a fascinating judgement call of what is easy and what is difficult. Defining a test around that claim lead them to learn things that were not true, and double the time to finish the exercise to include unlearning things they had started to firmly believe in.

The lessons were:

  • First test was a poor choice leading to long-term confusion, we could either choose differently or pay attention to what we really know more actively.
  • Any coverage will do, some here some there and testing is done! That is true, but tracking how much done are you is a big part of testing. Even the code coverage can fool you because the number shows line coverage and the colors show branch coverage. Calling it done on line coverage left out lines leading to new branches.
  • Naming coverage dimensions isn’t a thing everyone does, they didn’t. Conceptualizing code, specification, environment and risk coverage is a necessary thinking model.
  • Code coverage can be achieved without proper asserts and gives a very false sense of security. You don’t even need to assert values of quality for the 16 tests above to have them still run to 100% branch coverage.
  • Problems against specification were all assumed distinct problems, no grouping around the fact that there was full areas of “I expect input validation” that were not implemented anywhere.
  • Specification having shorthand like saying “Sulfuras” instead of “Sulfuras, the Hand of Ragnarok” isn’t a major problem with the specification even if it bit them in the exercise. Best of specifications are helpful, yet incomplete. Best of testers using specifications don’t need them to be complete to do complete testing.
  • When results did not match the specification, quickly jumping to conclusion that we are testing a buggy software which wasn’t the case. The models to pinpoint whether problem could be in my tests that I have control over were not in place. We practiced many rounds of “is there another test we could do that would give us a second data point to verify we understood the requirement”.
  • One test one requirement isn’t a thing. They are a messy network of dependencies.

There is no easy recipe for testing any application. Stop to think. Approach from different angles. Check and double-check. And don’t hide in your corner, maximise your chances of getting to the right information by working with the others.

What is Exploratory Testing — the Programmer Edition

This post has originally been published as Medium post on Jul 30th, 2018. Since Medium did not win on my choices of blogging platforms, I am including it here in efforts to consolidate my legacy. This particular post was promoted in an in-medium publication called noteworthy. This post has 4.7k views on Medium.

It has been 34 years since Cem Kaner coined the term to describe a style of skilled multidisciplinary testing common in Silicon Valley. In the new software world regime where programmers find themselves taking a lot more responsibility of testing, we need to understand what exploratory testing is as it extends what most programmer’s find their tests covering, and causes us talk past each other in understanding what testing.

What Testing Gives You?

As a programmer, you know what you’re implementing. When you don’t know, you’ll learn. You explore the problem to figure out a solution.

Back in the days some bad organizations told you that they’ve hired a group of testers. They might even say that since they pay other people for testing, you shouldn’t be bothering yourself with that.

But you know you want to test. Because testing has direct value to you as a programmer. It gives you four things:

  • Specification
  • Feedback
  • Regression
  • Granularity

Specification means that tests you write can be concrete examples of what the program you’re about to write is supposed to do. No more fancy words around high level concepts — give me an example and what is supposed to happen with it. And when moving on, you have the specification of what was agreed. This is what we made the software do, change is fine but this was the specification you were aware of at time of implementation.

Feedback means that as the tests are around and we run the tests — they are automated, of course — the tests give us feedback of what is working and what not. The feedback, especially when we work with modern agile technical practices like test-driven development, gives us a concrete goal of what we need to make work and if it is working. They help us anchor our intent so that given the interruptions, we still can stay on the right path. And we can figure out if the path was wrong. This is the test that passes, yet you say it isn’t right. What are we not seeing?

Regression means that tests don’t only help us when we’re building something for the first time, but they also help us in changes. And software that does not change is dead, because users using and loving it will come back with loads of new ideas and requirements. We want to make sure that when we change something, we make only changes we intended. And regression is the perspective of doing more than we intended, without tests without us knowing.

Granularity comes to play when our test fail for a reason. Granularity is about knowing exactly what is wrong and not having to spend a lot of time figuring it out. We know that small tests pinpoint problems better. We know that automation pinpoints problems better than people. And we know that we can ask people to be very precise on their feedback when they complain something doesn’t work. Not having to waste time on figuring out what is wrong is valuable.

This is not exploratory testing. Exploratory testing often guides this type of testing, but this is testing as artifact creation. Exploratory testing focuses on testing as performance — like improvisational theatre — in a multidisciplinary way beyond computer science.

Exploratory Testing, eh?

I get all of this good stuff from testing as I know it, unit tests and test automation. What is this exploratory testing stuff then?

Exploratory testing is basically saying that spending time thinking with your application, your APIs, your environments as an external imagination, you will find things you did not realize you were missing. And time thinking with something you’ve implemented is real, empirical. Not just something you wish was true like most of our designs.

Exploratory testing is multidisciplinary, basically saying that it is not enough for you to take orders from whoever gives you requirements, but you have to think critically on why they are asking what they are asking and its practical implications. If someone asks you to do something that is illegal, you need to point it out. If you don’t know if it is illegal, you need to spend time figuring out what is and isn’t illegal, and how that would show up in the application you are creating. GDPR is a great example of this: not caring for privacy can cost your organization a lot. And it is not just law you need to care for. You need to care for business, users, environments your software runs in, ethics, psychology — and many more. You want to do no harm, and be conscious about what you are doing. Intentional, not accidental.

With the way you look at testing as programmer, what more is there that exploratory testing promises to give you? It’s an approach that focuses on learning while evaluating what you know and don’t know, giving yourself chances of learning the things you don’t know you don’t know but can see while using the application like your users would.

It gives you four things:

  • Guidance
  • Understanding
  • Models
  • Serendipity

Guidance is not just about the specification, but general directions of what is better and what is not. Some of it you don’t know to place in yes/no boxes, but have to clarify with your stakeholders to turn them into something that can be a specification.

Understanding means that we know more about the place of our application in the overall world of things, why people would find it valuable, how other’s design decisions can cause us trouble and how what should be true if things were right always are not. It helps us put the details we’re asked to code into a bigger picture — one that is sociotechnical and extending beyond our own organization and powers.

Models are ways of encoding knowledge, so that we can make informed decisions, understand things deeper and learn faster next time or with next people joining our teams.

Serendipity is the lucky accidents, running into information you did not expect to find. The lucky accidents of new information about how things could and do go wrong when using your application emerge given enough time and variety to use of your application. And knowing helps you not get the escalations waking you up to critical maintenance tasks because who else would fix all of this than the programmers?

An Approach To Testing

Exploratory testing is a approach to testing. It says whoever tests needs to be learning. Learning needs to change what you are doing. You can’t separate designing of tests and executing them without losing learning that influences your next tests. It is an approach that frames how we do testing in a skilled way.

We don’t do just what is asked, but we carefully consider perspectives we may miss.

We don’t look at just what we are building, but the dependencies too, intentional and accidental.

Unsurprisingly, great programmer teams are doing exploratory testing too. Creating great automation relies on exploratory testing to figure out all the things you want to go check. While with exploratory testing we believe that premature writing of instructions hinders intellectual processes, we also know that writing that stuff as code that can be changed as our understanding grows, this frees our mental capacity to think of other things. The executable test artifacts give us that peace of mind.

Programmer teams are also doing what I would consider low quality exploratory testing with limited ideas of what might make a difference in new information being revealed. And that is where testers often come in — mindspace free of some of the programmer burdens, they focus their energies elsewhere, raising the overall quality of work coming out of teams.

Finally, I want to leave you with this idea — bad testing is still testing. It just does not give much of any of the benefits you could get with any testing. Exploratory testing and learning actively transforms bad testing to better.

What is Exploratory Testing?

This post has originally been published as Medium post on Jul 30th, 2018. Since Medium did not win on my choices of blogging platforms, I am including it here in efforts to consolidate my legacy. This post has 1.3K views on Medium. 

It has been 34 years since Cem Kaner coined the term to describe a style of skilled multidisciplinary testing common in Silicon Valley. I’ve walked the path of exploratory testing for 25 years and it has been a foundational practice in becoming the testing professional I am today. Let’s look at what it is, to understand why it still matters — more than ever.

An Approach to Testing

Exploratory Testing is an approach to testing that centers the person doing testing by emphasizing intertwined test design and execution with continuous learning where next test is influenced by lessons on previous tests. As an approach, it gives a frame on how we do testing in a skilled way.

We use and grow multidisciplinary knowledge for fuller picture of empirical information testing can provide. With the product as our external imagination, we are grounded on what is there but inspired to seek beyond it.

We learn with every test about the application under test, ourselves as the tool doing the testing, other tools helpful in extending our capabilities and the helpful ways we can view the world the application lives in.

We keep track of testing that has been done, needs doing and how this all integrates with the rest of the people working on similar or interconnected themes.

What makes our activity exploratory testing over other exploratory approaches founded in curiosity is the intent to evaluate. We evaluate, seek for information we are missing, making sure what we know is real with empirical evidence.

Specifying It By Contrast

It’s not a surprise that some folks would like to call exploratory testing just testing. In many ways it is the only way of testing that makes sense — incorporating active learning is central to our success with software these days.

To contrast exploratory testing with what people often refer to as manualtesting, exploratory testing as a skilled approach encompasses use of programming for testing purposes. We use our brains, our hands as well as programs to dig in deep while testing. Sometimes the way of including test automation happens through means of collaboration, where ideas from exploratory testing drive implementation of automation that makes sense.

To contrast exploratory testing with people refer to as scripted testing, exploratory testing isn’t driven by scripts. If we create scripts from exploratory testing, we know to use them in exploratory fashion remembering that active thinking should always be present even when the script supports us in remembering a basic flow. We’ve talked a lot about scripted testing as an approach where we separate design — deciding what to test, and execution — making the test happen, and thus lowering our chances of active learning targeting the most recent understanding of the risk profile.

Another contrast to programming centric views to testing comes with embracing the multidisciplinary view of of testing where asking questions like “is my application breaking the law today after recent changes?” is something routinely encoded into exploratory testing, but often out of scope for a group of automators.

The Three Scopes of Exploratory Testing

As an approach, we can time and integrate this into our development efforts in many ways.

I find there are three ways of scoping:

  • exploratory testing as a way of extending existing test cases
  • exploratory testing as a limited timebox
  • exploratory testing as frame of all thinking

When exploratory testing is scoped as a way of extending existing test cases, the way of working does not look very exploratory. In places like this, you find yourself wondering why Tina is so successful with the same tests Toni is missing problems with. The secret is that one of them actively extends how they understand the test cases, refuses to follow exact instructions or only instructions. They learn and find new perspectives.

When exploratory testing is scoped as a limited time box, you find people feeling they can only be free on Friday afternoons. These are moments in project life that are structured separately, with different expectations of how things flow and where focus is. This is a great way of scoping exploratory testing into a process where the natural inclination is to believe we know where the tasks begin and end, and need explicit encouragement to see if what holds true.

When exploratory testing is the frame of all thinking, it encompasses all considerations about testing. We start with exploratory testing to identify something that needs documenting, and take time to document it — perhaps with automation that is the modern way of documenting as executable specifications.

Listen to Language

If you hear: “My boss asked me to test search so I searched something that was found and something that wasn’t and reported I was done” you may be witnessing very low quality exploratory testing. It relies on following high-level orders to an extent the tester can imagine based on their knowledge.

If you hear: “My colleague wrote me 50 cases of what to try out with search and I tried them and reported I was done” you may be witnessing testing that isn’t exploratory. There is no hint of learning, and owning responsibility of quality of the testing that happens.

If you hear: “My boss asked me to test search so I came back with 50 quick ideas of what could be relevant and we figured out I’d just do 10 of them before we decided if going further was worthwhile“, you are likely to be witnessing exploratory testing.

Similarly, in the automation first space, if you hear: “I got a Jira ticket saying I should automate this. I did, and found some problems while at it, and extended existing automation because of the stuff I learned.”, you may be seeing someone who is exploratory testing.

If you hear: “The Jira ticket said to automate A, B, and C, and I automated A and B, C could not be automated.”, you may be witnessing testing that isn’t exploratory.

Look at who is in the center: is the tester doing the work and learning actively applying a better way of doing the overall testing? If yes, that is exploratory testing.

As skilled approach, it is only as good as the skill of the person applying it. With focus on learning through, skill may be a problem of today, but improved upon every day as exploratory testing is being done. If you find yourself not learning, you most likely are not exploring. With the product as your external imagination, you should find yourself imagining new routes through the functionalities, new users with new perspectives, and relevant information your project teams would be happy to make justified decisions on their take for the risk. With and without automation, in a good balance.