End to End Testing as a Test Strategist

For the last 10 months, I've worked on a project that assigned me end to end testing responsibility. As I joined the project, I had little clue on what the work would end up being. I trusted that like all other work, things will sort out actively learning about it. 

Armed with the idea of job crafting ('make the job you have the job you want') I knew what I wanted: 

• principal engineer position with space for hands-on work
• making an impact, being responsible
• working with people, not alone

With a uniquely open mandate, I figured out my role would be a mix of three things:

1. tester within a team
2. test manager within a project delivering a system
3. test coach within the organization 

The learnings have been fascinating. 

My past experience of growing teams into being responsible for the system rather than their own components was helpful, and resulted in seeing both confusion and importance of confusion of one team member choosing scope of work bigger than the team. 

I had collected more answers to why than many people with longer history in the company, focusing on connections. The connections enabled me to prioritize the testing I did, and ask others for testing I wasn't going to do myself. 

The way I chose to do end to end testing differed from what my predecessor had chosen it to be for them, as I executed a vision of getting end-to-end tested, but in small pieces. 

I focused on enabling continuous delivery where I could, taking pride in the work where my team was able to release their part of the system 26 times in 2020. 

I focused on enabling layering of testing, having someone else test things that were already tested. It proved valuable often, and enabled an approach to testing where we are stronger together. The layered approach allowed me to experience true smoke testing (finding hardware problems in incremental system testing).

The layered approach also helped me understand where my presence was needed most for the success of the project, and move my tester presence from one team in the project to another half-way through. I believe this mobility made a difference in scale as I could help critical parts from within. 

I came to appreciate conversations as part of figuring out who would test things. I saw amazing exploratory testing emerge done by developers following hunches of critical areas of risks from those conversations. 

Instead of taking my end to end testing responsibility as a call to test, I took it as a call to facilitate. While I have used every single interface the system includes (and that is many!), every interface has someone who knows it deeper and has local executable documentation - test automation - that captures their concerns as well as many of mine. 

Looking back, it has been an effort to do end to end testing as a test strategist. You can't strategize about testing without testing. My whole approach to finding the layers is founded on the idea that I test alongside the others. Not just the whole team, but all of the teams, and in a system of size, that is some ground to cover. 

Three routes to fixing problems test cases hide

I'm a big fan of exploratory testing, which often means I have reservations about test cases or at least ideas of how to interpret test cases in a way that does not require such an intensive investment into writing things down, that help us write things down that are needed and how to not think of test cases - or any documentation for that matter - to represent all the testing we are doing. 

Today I wanted to share three experiences from my career from three different organizations on how we tweaked our test cases to fix a problem all three organizations shared: using a lot of time for testing, but leaking significant bugs we believed we should be able to find. 

Organization 1: Business Acceptance Testing

The first example is from an organization where I managed business acceptance testing. I was working with two different projects, moving the business acceptance testing phase from months after months endeavor to something that would fit 30-days. One of my projects had a history of writing detailed test cases, the other had a history of not really using test cases. In getting the timeframe condensed, understanding what we had in mind to do and being able to reprioritize was essential. 

For both projects, we used Quality Center (HPs ALM solution was called that back then). Both projects started with test data in mind, and that is what we used as a starting point for our tests. We selected our test data to a set of criteria, wrote the criteria down on the test case title summarizing the business need for that particular data type. And as test steps, we used Quality Center's concept of test templates - a reusable set of steps that described the processes the two teams were running on a high level, same for every single test case. 

Thus our test cases were titles, with template test checklists to help us analyze and reprioritize our work. Same looking tests on first week could take a day, and later in the cycle, we could spend 15 minutes. The test case looked same, but we used it different, to explore. 

On one of the two projects, they had a history of writing test cases where steps also described the detail, and were concerned that giving those up may mean they forget to cover something as the information of changes isn't easy to pass around for a whole group doing acceptance testing. So we split out weeks into two first where we used the "old style" detailed tests and two latter where we used the new style. We found all problems during the latter two weeks, but in general, the software contractor had done a really great job with regards to their testing and the numbers of bugs we had to deal with were record low. 

Organization 2: Product testing with Reluctant Developers

The second example is from an organization I joined as their first and only test specialist. With their project manager's leadership, they had figured out writing test cases into word documents, one for each major area of the product. Tracking that the test cases were completed was central to the way they tested amongst the group of developers. Automation, on unit or system level, was not a thing yet for them. 

As I joined, the project manager wanted me to start creating test case documents like they had, improving them, and had ideas of how many test cases they would expect me to complete every day. 

Sampling one of the existing test specifications, it had 39 pages, 46 test cases, and 3 pieces of relevant information I could not figure out without reading the text based on commonly available knowledge. 

I made a deal with the project manager to write down structured notes while I tested, and we got to a place where I was trusted with testing, reluctant developers were trusted to test with me, and the test cases went away. Instead we used checklists of features to remind us what could be checked to design tests in the moment with regards to what the changes to the system were. 

Organization 3: Product testing with certification requirements

The third example is from an organization with a history of writing test cases that are traced back to requirements. Test cases are stepwise instructions. 

The change I introduced was to have two kinds of test cases: [Scenario] and [Feature]. Scenarios are what we use to test with and leave a lot of room for what exactly needs to be verified. Same test could be a week or an hour. For Scenarios, the steps are features as checklist - what features are part of that user journey. When we feel we need a reminder of how-to see a basic, sunny day scenario of a feature to remember what testing starts from, that is where Feature tests come in. The guideline was to write down only what wasn't obvious and keep instructions concise. There can be a feature test, without any steps at all. Steps are optional. 

Clearly, the test cases don't describe all the testing that takes place. But they describe seeing that what we promised that would be there, is there, and help us remember and pass on the information of how to see a feature in action. 

The Problems Test Cases Hide

Test cases can lead people into thinking that when they've done what they designed to do - the test cases - they are done testing. But testing does not work that way. The ways software can fail are versatile and surprising. And we care about results - information and bugs - over the documentation. 

Too much openness does not suite most of us. But too much prescription suites us even worse. And if prescription is something we want to invest in, automation is a great way of documenting in a prescribed manner. 

Faking testing and getting caught

A lot of my work is about making sense into the testing we do, and figuring out the quality of testing. The management challenge with testing is that after they come to terms in investing in it, it isn't obvious if the investment is worth it. Surely, the teams do testing. But do they do it so that it provides the results we expect? Faking testing isn't particularly hard, and the worst part is that a lot of the existing processes encourage faking testing over testing that provides results. With an important launch, or an important group of customers we're building a business with, we don't want to be surprised in scale or type of issues we have never even discussed before. 

I find myself in hunt of two aspects of quality in testing: effectiveness and efficiency. 

Effectiveness is the idea that no matter what the testing we do is, does it give us the layers we need to not be caught red-handed with our customers? To be effective at testing, we have two main approaches:

• Build well - bad testing won't make quality worse
• Test well -  when issues emerge, finding them would be good

A good way to be effective at testing is to bring in more eyes at the problem. Don't ask only the developer-developer pair to test, add a third person you believe will take testing to heart and prioritize it. But why settle for three people if you could have ten, or 10 000. Grow the group. From unit testing to system testing, to end-to-end testing of systems, to pilot customers to beta customers, to real customers in production with continuous releases. That will tell you of effectiveness of earlier stages, and make you effective overall. 

This effectiveness consideration is the most important work I do, and sometimes - I would even say often, I catch people in this activity on faking testing. Most often people don't even realize they are faking testing, and they approach testing with wrong skills and focus, not providing the result that would be reasonable to expect for investing into that layer. 

Faking testing, no matter how unintentional, looks like we do a lot and we do, but we get nothing out. And we notice only when there is a need of testing well, because building well did not alone give us sufficient results. The later stages, hopefully designed into the overall testing in times close to the faked testing rather than discovering quality of testing at the delivery, reveal bugs in scales and types that we would reasonably expect to be found if we did the stages well. 

While effectiveness is the key, after effectiveness is in place I look at efficiency. There is only so much we should have to invest in testing on various layers, testing should increase our productivity and not slow us down, and we might be investing in a box of testing thinking it cover more boxes than it should for the return the investment gives us. 

These puzzles often fill my days, and I try to help people learn exploratory testing to get away from faking testing. They don't want to fake it, they just do what they think they should.

In Search of a Test Automation Strategy

The world of software as I know it has changed for me. I no longer join projects in preparation of a testing phase that happens in the end, but I am around from when I'm around until I am no longer around, building testing that survives when I am gone.

Back in the day of testing phase at the end of a project, test strategy used to be the ideas you prepared in order to work through the challenging phase. It gave the tests you would do a framing, guiding design. It usually ended up being written down in a test plan under a heading of approach, and it was one of the most difficult things to write in a way that was specific to what went down in that particular project.

With agile, iterations and testing turning continuous, figuring out test strategy did not get easier. But the ideas guiding test design turned into something that was around for longer, and in use longer. I talked about what ideas stuck with me at DEWT5 in 2015, and same ideas guide my testing to this day. 

Since then, I'm working even more on the strategy we share and visualizing it to nudge it forward. Seeing the strategy in action in a new team can be dug out of the team, asking the team to visualize their testing activities. 

The strategy I set does not matter, if it does not turn to action with the team. We now move versatile groups of people across different roles and interests. 

This week gave me a chance to revisit my ways on a theme of test automation strategy. I have never written one. I have read many, and I would not write any of those. But it stopped me to think of the ideas that guide my test automation design right now. These are the ideas that I brainstormed:

• Start with the end in mind
• Release time with minimal eyes on system. Rely on TA (test automation) on the release decision. 
• TA keeps track of what we know so that it remains known when we change things
• Incremental, incomplete, learning
• Work towards flow of TA value - small streams become a significant pool over time. Moving for better continuously matter, not starting well or perfect.
• Something imperfect but executable is better than great ideas and aspirations. Refactor to reveal patterns.
• Timing
• Feedback nightly, feedback on each change. 
• Maintain ability to run TA on every version supported for customers
• Early agreement
• Design automation visibility and control interfaces at epic kickoffs
• Scope
• For each epic (feature), add the positive case to TA. Target one. More is allowed but don't overstretch.
• Unit and software integration tests cover cruft of functionality. TA is for system level scenarios including hardware (as it is embedded for us). 
• Not only regression TA, also data, environments, reliability, security and performance in automation. 
• Acceptance tests for interfacing teams monitor expected dependencies.
• Save the data. Build on the data. But first learn to run it. 
• People
• Invest in skilled TA developers through learning and collaboration
• Require developers to maintain automation for breaking changes
• To facilitate GUI selectors, GUI devs create first test with keywords
• Allow for a "domain testing expert" who only contributes in pull request reviews on TA
• Practices
• Suites and tags give two dimensions to select tests, use tags for readiness
• Seek to identify reusable data sets and oracles
• Reuse of keywords supported through reviews and refactoring time

I guess this is as close to a test automation strategy I'm about to get. 

Requirements Traceability

I'm all for understanding the products we build on what they do, and why we think doing those things might be relevant. The language I choose on requirements is the language of discovery. Instead of us requiring something, we discover needs and constraints, and make agreements we visualize in structures and writings to create a common understanding between all of us in organizations and our customers. Instead of setting a requirement, I like to think of us setting a hypothesis of something being useful, and testing that. And since I like to approach it as uncertain, I can easily embrace the idea that we learn and it changes. I don't need the concept of managing change to requirements, it is already built in with incremental delivery.

Thus, there are very few things that annoy me to the brink of wanting to change jobs but this one is one of those: requirements traceability between the magical requirement and the test case. 

In organizations that belong to the school of requirements, this is the heartbeat that conflicts with my heartbeat set on releases. Releases include capabilities at that time, and understanding what capabilities we had, as well as idea of replacing those capabilities systematically with newer set of capabilities. Arrhythmia is irregularities in heart beat, and it serves as a description as how I feel in the conflict of these two worlds I need to learn to adapt together without giving up on what I have grown to see important. 

In search of congruency, I put significant effort in doing things I consider valuable. I have never considered it valuable to take a requirement written down in a tool, and create a test case written down in a tool and link those two together. I don't write test cases - I test directly against the requirements and creating this other document feels like wasted time. Also, I don't believe that the requirements we create are the best representation of the understanding that leads us in finding information I seek to find in testing, and often following someone else's structure takes away from my ability to contribute information that adjust that structure for the benefit of all stakeholders. 

So, requirements traceability does not only waste my time in creating material I consider useless but it  also makes creating the results I expect to create harder. Over my career, I have needed to set straight on a good path of testing that provides results many organizations that started off with a requirements-centric straightjacket creating testers I would recommend letting go. 

So I push through once more with what I will do:

• Given a requirement, I will confirm it in an exploratory testing session but only document that with closing of the epic, at a point of time we first introduce it into a release in the making
• I will work to include it in test automation, and keep a version of test automation around that matches that release while it is supported. I will not offer a link back to requirements from specific automation cases. 
• When using a feature is hard to figure out, I will write feature-specific instructions to document what I have learned while testing
• I will create whatever material supports continuous, growing testing, without linking it to requirements.
• I will care not only for my own work now, but for the work that comes after I am long gone

I recognize we are seeking these benefits from the mechanism the industry standards push:

• managing expectations on what the product does and does not do
• enabling support when product is in maintenance by
• recognizing defects (as being against specification) from adaptation requests
• retesting connected requirements when changes are needed
• ensuring we don't miss someone's perspective in delivery by making 'proof' of testing on all perspectives - connecting two ends of a process allows for being systematic
• making testing accountable for monitoring delivery scope on empirical level
• having an up-to-date description of configuration of each delivered version
• replacing old product with new could be easier
• tracking project against requirements to deliverables giving sense of control to project manager 

Having Testers Makes Quality Worse

I'm a professional tester. I have been one for quarter of a century. I know by my results and my feedback that I have been useful for my organizations. But the longer I've worked in these types of roles, the more I understand that while a tester may be a solution, a tester may also be a problem in a people system. And whether we, testers, become problems has very little to do with us, and a lot to do with the other people around us. 

Today, I want to dig in a little deeper on an idea that I have been on for most of last year. 

I'm reconsidering my position around need of testers in teams, and as a tester, I am not doing this lightly. I am starting to believe that with agile (and learning cycles), developers get smart enough not to need manual testers around.

— Maaret Pyhäjärvi (@maaretp) March 23, 2020

First of all, let's start with what my career of quarter of a century is really about. While I am a professional tester, a significant part of my professional skill is knowing when to refuse to do testing alone and only do it while pairing to grow people around me. Another significant part is to walk away from teams that grow better when they have no testers, and join teams that are in places where they can grow having a fully contributing tester. My career is on *testing* not on *being a tester* and that difference is a significant one. I believe, paraphrasing Elizabeth Hendrickson, that testing is too important to be left for just testers. It belongs to everyone in software development. Sometimes it is useful to allow space for people to concentrate on it, and sometimes people who get to concentrate on it are called testers. 

Growing the Understanding Over Multiple Jobs 

With this blog post, I want to look at back over the last three jobs I have held. They all have one thing in common: I work in product development, with lovely, brilliant but imperfect software developers who have been soaked into culture describing what the organization expects of them with regards to testing. Yet, the situations in the teams I have worked in could not be much more different.

The first position of the three was one where I was hired as the first tester the organization had. They had 20+ years of software development experience and a very successful software business, and they had managed to do this without a tester. They were seeking one as they had identified, based on internal product management feedback, that there could be enough work for someone so that product management did not have to do all of their testing. Also, they had a metric on big visible error screens per logins, which hinted they really needed someone to tell them how to reproduce problems customers were seeing but not complaining about. 

For the years I spent with the organization, I could see they needed different things from me at different times. I was a hands-on tester for them for the first year, reporting all those cases the customers didn't complain about but were experiencing and the developers fixed them. They needed me to catch up with the understanding of what feedback they had been missing. But as they learned what they had not known, they learned to know it, and through pairing (and stealth pairing), the developers built rapport for their testing discipline.  Finally, we got to place where I did my best to avoid reporting bugs and work with developers to improve their understanding of how problems emerged. In the end as I was leaving, they did not need me - they needed a voodoo doll of me they could look at and go "you'd want me to test this, you'd want me to try different values, you'd want me to try more times". And while being the person who holds space for good work to happen, it was evident it was my time to see other teams. Since it was always one of me and 20 of them lovely developers, we worked out the idea of not expecting me to do all the testing from the start. 

The second position was one that I rejoined after being away from a decade. While I had been away, the team had found themselves into a position where there was a good test automation discipline and hired people they vaguely called testers, into writing system level automation that was a full time job to keep maintained and going forward. I found myself building bridges of ideas of tests I had from exploring to test automation, and building a culture of fix-and-forget with zero bugs on backlogs. The developers were soaking in every piece of feedback, and fixing things so effortlessly that in hindsight I feel like there weren't problems to address. The culture of managers staying out of the way and senior colleagues wanting to make space for great quality combined with documenting with test automation, extending every test idea over different environments could have been run with developers. The challenge I addressed mostly was one of communication - with our code being part of the system, understanding where what testing would take place wasn't always easy and straightforward. 

The third position is one that I hold now. I joined one team first to learn that while they could use me for a while, their long term would be better if I walked away and left them to try out their lessons now without a tester. I joined another team starting a few weeks ago, and see a similar pattern of needing full-time people on tending the test automation system, even when whole team pitches in. My work, again, appears to be addressing communication, and understanding where testing of various pieces and perspectives reside. I test to learn, to understand, and to distribute my lessons. I have now multiple teams without testers, and some of those do brilliant. The pattern emerging seems to be one of ownership, and that isn't always an easy one with systems of scale in multi-team setting. 

Was I getting somewhere?

I've been hired with organizations as a tester, and by choosing the pattern of how I operate with the idea that testing belongs to us all, I've come to be appreciated for results. I've been a tester, test manager and test coach depending on what the day needs me to be, and promptly addressed the risks of assuming tester = testing. I've seen most developer do well with testing, and I've seen many developers do better with testing than professional testers who focus on testing. I've seen how having a tester, and making space for a tester to do testing, makes quality worse. But I have also seen testers who truly bring added value, in understanding what we are building, understanding the customer and the domain, and thinking in terms of systems over components. And increasingly, I've seen tester/developers who specialize in programming test systems be critical in sharing a good baseline practice with their teams, and find new labels like SDET, eventually dropping the T while still continuing on the test side of systems. 

There is no one clear recipe for testers and developers. As a recruiting manager, you need to understand where your team is now, and where a tester could take them. But it always comes with a risk: having testers makes quality worse if developers let go of testing they are doing now. 

Contemporary Exploratory Testing

We all know what testing looks like: it's when we hunt information, using the application, it's interfaces and all existing information as our external imagination to go even deeper in empirical understanding of what is true and what is an illusion. It involves a person or multiple people, using programming to get to places in time, but it is all framed in this quest of someone learning what more we could try knowing. 

Yet when organizations set up "testing", they ask for resources, pay limited attention to skills, focus on plans, covering requirements, writing and executing test cases, cutting down in testing when project schedules fall short. 

When we say "all testing is exploratory", we have the individual tester on their good day in mind. I call this exploratory testing, the verb. The act of testing is inherently exploratory when people are involved. No matter how strict a test case you gave, how much you told not to leave that path described, the human mind wonders and the human fingers make mistakes revealing more than the test intended. 

The organizational frame however makes a huge difference on how that inherently exploratory testing takes place, and how much of its wings are cut. I call this exploratory testing, the noun. Organizations often set up frames for testing that are far from exploratory, and get results that are far from what we would mean by results of exploratory testing. 

Exploratory testing (verb AND noun) is focus of my learning and teaching. I want to create excellent products with excellent testing, and I feel that the 36 year old coining of the term needs a revisit from its watered down interpretations. Thus I sometimes add still one more word to explain what I am aiming for: contemporary exploratory testing.

Contemporary is about today. I use that word to get away from the ideas of ISTQB and agile testing where exploratory testing was considered a technique, a thing you do for unknown unknowns on top of all the other testing recipes. For me, it has for the last 25 years of my career in it been an approach, a foundation of agency, learning and systems thinking that frames my choices of using all the other recipes. And I believe that is what it is at its roots, so calling my approach contemporary is saying I want to take a step away from some of the popular notions of it being the idea of just spending time with an application to find bugs. 

With this new year, I welcome you to join my exploration of how to understand and teach contemporary exploratory testing. I have many subproject on it:

• Exploratory Testing Podcast. I just published my first episode yesterday, and will continue to do so on a monthly cadence. 
• Exploratory Testing Academy. I work with  Parveen Khan (UK), Angela Riggs (US), Irja Straus (Croatia) and Mirja Pyhäjärvi (Finland) to create free testing video courses and a series of paid facilitated courses with focus on learning by doing testing under various constraints for various systems under test. 
• Exploratory Testing Book. I finish the book I started so that you don't have to read all the individual articles and get out a timeline of where I am now compared to what I wrote earlier. 
• Exploratory Testing Slack. I want to bring together people that are figuring out contemporary exploratory testing. I take it more as a forum of practitioners than a forum of consultants. 
• Exploratory Testing Twitter account. I use this for promotions, because the one advice I was given on marketing (collect people's emails) is the one advice I don't want to use. I want pull over push, even if it is against all things marketing. 

I have a full time job at a company as a tester. My job pays well, better than average developers, and I have my company specific goals there. I do this all because I feel the one thing I should do better is scale. I make my work available for free to support scale. I have things I could use extra money on, but I leave the payment part for the community based on value. You can pay (but don't have to) for my book in progress. You can pay by buying me coffee. Since I am Finnish, you can never donate me money - only pay for my services. But I want that to be optional, and work against paywalls in my own little way. 

If out of this I create more great testers, more testing trainers, more love of testing in both professional testers and programmers - my aspirations are fulfilled.