Lessons Learned on Working with Data-Intensive Applications

In my years of teaching test design techniques, I have come to teach people that there are (at least) two essentially different types of functionalities we design tests for:

• function-intensive applications are ones where you list the tricks the app can do, and a lot the work in designing tests are creating lists of functionalities and exploring when they work. 
• data-intensive applications  are ones where the same functionality is riddled with data-oriented rules, and you are collecting business rules captured in data. 

This difference became clear to me as I switched jobs long time ago from antivirus software development (function-intensive) to pension insurance (data-intensive), and spent the next few years trying to wrap my head around the new challenges I had not paid attention to before. 

When data became the center of my testing universe, I learned that one of the major challenges we would be spending a significant chunk of our time on would be picking the "test data". If we needed a person who was just about to turn 63 (age of early pension), and we wanted to test today the scenario that they were under the limit and tomorrow the scenario that they were over the limit, we needed to find a precise set of data with those conditions.

And the data was not simple and straightforward row in a database. It was a connected set of databases, some owned by our company, some by some other companies, and for getting a production like experience in test environment, we had tools to choose someone and pull all related information into our systems. Similarly, we know we had an agreement that with 14 days notice, the other pension insurance field players would set their data on request to match what their production had. When I knew who pulls one data set and scrambles the data in the process and who pulls entire copies of their production databases and scrambles the piece we use to match the way we scramble, I could help our testing by the business experts flow a lot nicer. 

10 years passes, and I forgot it was difficult because for me it was routine. Until today when I had to explain it in my current place of work why it is that something so easy and obvious to me is so difficult and complex to many others. 

This is what we had today. An application hooked into its own database. A separate production and test environment. But connected business application data between production and test environments. 

Some applications make it preventively hard to use artificial data in test environments. When your business systems run tens or hundreds of hourly synch batch jobs, have tens or hundreds of users executing their day-to-day manual processing tasks on the user interfaces and the system clock that inevitably changes things because you have time-based logic, you will need to replenish the data from production. 

What we had was two different very simple replenish cycles. Once a month on an agreed date, one of the systems would get a refreshed copy from production. Once a year, another of the systems would get a refreshed copy from production.

I had designed last year our test data, independent of production in the other parts of the end to end test environment to be ok when data moves like this. The two systems synchronizing had the necessary data in production, and would be reintroduced with replenishing the data. 

Except it did not work. 

The application had bugs around not expecting the data to be replenished, but only on the logic that changes once a year. 

Someone else had not understood the rule of how to set up the data and had requested data that vanished in the replenish. 

I spent significant time teaching how to follow the data across the systems, and how the logic works between connected data sources and different environments. If I did not know this, I could not have tested the functionalities (data-intensive ones) last year when I did. 

What I learned though is that: 

• documenting and knowledge sharing does not help if there is 10 months between being taught and needing the information
• what I consider clear may be unclear to others
• everything that can fail will fail, but at least it failed in test environment

Talks Turned Articles

At my blog, I write whatever I feel like writing down. With blog posts, I accept that the posts are windows to my thoughts, representing whatever I had as context in the moment. People read some of it, some of them connect with me on the topics I am discussing. The temporal nature of blogging means it that I don't expect to write here sources that would be useful as articles and references, but different perspectives that I am processing towards those articles and references - and talks. 

I am committed to writing proper articles, and I have different collaboration platforms I share those articles on. By an article, I am thinking of something that should be useful even if you don't came to it at the time it was written, that somehow collates work in a more concise package. Articles should stand on their own, and teach you something that new people need to learn, again and again. 

Talks delivered, however, are something in between. The spoken format allows for - and requires - contents in a style and format we rarely write in. Talks are created to stay valid for a longer time, but the way they are presented is very temporal. A lot of times when I choose talks to conferences, I find myself using a phrase "not a talk, should be an article". 

Thus I find it interesting to experiment with something in between. I have started writing some of my talks after delivering them once. Video is available but who watches videos as replay when there is continuous stream of new content. I call this experiment #TalksTurnedArticles and you can find those on my dev.to profile. 

The latest in the collection is the talk I delivered today at TestFlix: Better Ideas at Test Design. Before that I published my favorite experience of transformations Practice Makes Better - 5x to Continuous Releases. And first in this series was Exploring Pipelines. 

These are articles in the sense that they are content I believe will stand the test of time. They are in a different place so that this place remains as low bar to write on my experiences, which turn to summarized talks and articles usually on cycle of some years. 

In addition to #TalksTurnedArticles, I chose dev.to as the place to host my full courses as text. The first one is available already: Exploratory Testing Foundations. 

You follow what I write here, but my best writing - at least to my standards - resides elsewhere. 

Three Stories Leading Into Exploratory Testing

End of September, I volunteered on a small lunch-time panel on exploratory testing at the conference. I sat down for a conversation and had no idea it would be such a significant one for my understanding. The panel was titled "To Explore or Follow the Map" and I entered the session with concerns on the framing. After all, I explore with a map and follow the map while exploring. 

Dorota, our session facilitator, opened the session inviting stories like one she was about to share on first experiences to exploratory testing. 

Paraphrasing Dorota from memory, she shared a story of how her first testing experience in the industry was on a military project where the project practice included requirements analysis and writing and executing test cases to do the best possible testing she could. One day the test leader invited all the testers to a half-a-day workshop where they would do something different. The advice was to forget the test cases and explore to find new information. And they did. The experience was eye-opening to all the things the thorough test case writing was making them miss. 

I listened to Dorota's recount and recognized she was talking of exactly the expectations I am trying to untangle in my current organization. Designing test cases creates a lovely requirement to test linking but misses all too much of the issues we would expect to find before the software reaches our customers. 

Next up was Adam, who shared a story of his first job at testing. His manager / tutor introduced him to the work expected from him giving him an excel with test cases, an a column in which to mark the pass/fail results. Paraphrasing his experience from memory, he shared that after he finished the list, the next step was to start over from the beginning. The enlightenment came with a conference where he met an exploratory testing advocate and realized there were options to this. 

My story was quite different. When I first started as a tester, I was given test cases, but also a budget of time to do whatever I wanted with the application that I would consider taught me to understand the application and its problems better. The test cases gave some kind of structure of talking about progress in regards to them, and I could also log my hours on whatever I was doing outside the test cases without very rigid boundaries between the activities. The time budget and expectations was set for testing activity as a whole, and I could expect a regular assessment of my results by the customer organization's more seasoned testers. The mechanism worked so that for a new person, first "QA of testing" was feedback, and latter ones had financial penalty if I was missing information they expected me to reasonably find with the mix of freedom and test cases to start with. 

While I was given space for better, I did not do better. No one supported me the way I nowadays aspire to support new joiners. Either I knew what I was doing or a minor penalty on invoicing was ahead, I would still be paid for all of my hours. I never knew anything but exploratory testing, and the stories of injecting it into organizations as Friday afternoon sessions or rebellious use of test cases to stretch from have always been a little foreign to me. 

What the three stories have in common is that exploratory testing is part of these pivotal moments that make us love the testing work and do well with results. My pivotal moment came from my second job where I was handed a specification, not test cases and I had to turn my brain on, and I've been on the path of extraordinary agency and learning since. 

Also, these stories illustrate how important the managers / tutors are on setting people up on a good path. Given requirements to test cases, you simplify the work to miss the results. Given test cases, you do work better left for computers. Given time without support you do what you can, but support is what turns your usefulness around. 

The Six Things Facilitator Can Do To Improve Ensembling Session

Today as I shared on Ensemble Testing at EuroSTAR around lunch, one of the questions lead us to discuss specifically what the facilitator can do to make ensembling better. Watching people fail is the back row would make me squirm uncomfortably, so what can I do? 

These things have been very useful to me. 

1. Support Moving Forward with Questions

You see the driver on the keyboard not moving, and the navigator off keyboard unable to decide what to do. Ask questions! For example, you could ask the navigator "What would you like to do next?". A more general rule is to try to talk in questions as a facilitator. Think about what you want the ensemble to do so that they would work well and have a great experience, and frame your comment as a question that guides them towards that great experience. 

2. Call out a Thing They Do 

You see them doing something that they don't realize they do, but you realize because you've seen others do the same but struggle. Call out the pattern. For example, an ensemble making good notes about their testing with a feature - variable - data -structure modeling that you have learned to appreciate, name it in the moment. Giving something a label makes it something that is a little easier to retain. 

Don't overdo this during the session, you can also save examples to point out in the retro. Call it out if it helps in the moment to retain the thing and adds to vocabulary they use to successfully communicate. 

3. Step in to navigate

I use this one on teaching ensembles a lot, but also have found it useful on difficult work. For example, I might say: "Let's pause the rotation and I'll navigate a bit." This frees the current navigator who steps back to continue with timer as soon as I step out. 

I use the same pattern with a single expert - asking them to navigate when the struggle of the others in the ensemble is no longer benefit for their learning. That is, I only step in to show if I know, but I can also ask someone else, or even a volunteer to step in a while. 

4. Stop the Ensemble to Mini-retro

Make space for them to fix themselves. A lot of new ensembles need that. Well, a lot of older ensembles still need someone pointing out they should have a conversation. I once watched Woody Zuill do just that - point out a dynamic that the team needed to have a conversation on. 

Some of my best facilitation tricks are on calling out a retro after just a few rotations. People somehow need a moment where they agree on how to fix their work style to fix the work style. The facilitator can create those spaces. 

5. Set a Constraint

In one of the first ensembles I ever facilitated, I saw my co-facilitator use this to make me the expert of the group and stepping in to navigate. With a twist though - I had narrow rules on the type of work I was supposed to do. The work was exploratory testing, and the new group struggled with note taking. The constraint applied on me was to only improve notes - structure and content of what we had already learned. 

I have used this technique since, and it works great but different groups need different constraints. 

Helping the ensemble figure out what is the scope of the task they are on now is setting these constraints. Thinking in terms of what is included, how to add to what is included only with "yes, and..." rule and parking ideas for future help an ensemble work. 

6. Visual Parking Lot

Create a space, in the documentation or on a whiteboard - to make notes of things you leave for later. People generate great ideas while the work is ongoing and they may forget them by the time we seek for next work to do. Give them a space / mechanism to park those somewhere as they emerge, and call reflection on structuring the parking lot occasionally. 

Hiring manual and automation testers

 In a meeting about hiring a new tester, a manager asks me: 

Are we looking for a manual or automation tester? 

In my head, the time stops and I have all the time in the world to wonder about the question. I'm neither. Nor is none of my immediate team colleagues. Yet look into the next team and they have one manual one automation tester. No wonder the manager asks this. We've moved from this to the next level. We're neither. We're both. Preferably, we're something different: contemporary exploratory testers. 

In the actual conversation, I don't remember the exact words I use to explain this, but I remember the manager's reaction: "that makes sense". 

We are looking for someone who is *learning* and does not box testing into *manual* and *automation* but builds from a strong foundation of understanding testing but does not stop at the sight of code. 

We want a tester who, when changing the hardware setups and network configurations, also changes the setups in the test automation repos and verifies what ever test we have automated will still run, instead of handing the information and task to someone else. 

We want a tester who reviews other testers' test automation pull requests and proposes improvements both in what gets tested and how it gets tested, and understand what the automation now has. 

We want a tester who reviews application developers' pull requests for scope and risk of change, and target their activities using this information as one source of understanding what they might want the team to test. 

We want a tester who documents their lessons from spending days deeply analyzing features for problems with some tests they leave behind that run in automation. 

We want a tester who talks with documentation specialists, product management, project delivery organization and support, and turns the lessons into investigations that could leave something behind in either unit tests or system level test automation. 

We want a tester who pays attention to the state of the automated tests and volunteers to investigate and fix in the team. 

We want a tester who creates automation to do repeatable driving and monitoring of a feature the team is changing now, and analyze the insights from receptions, as well as consider throwing away the automation because it makes no sense to keep that around continuously. 

We want a tester who will spend four weeks building the most complex analog measurement test setup with resistors and power sources, and understands what of that they find relevant to include in our test automation setups. 

We want our testers to work without having to hand off everything within testing to someone else because it's the only way they imagine how both good testing and good test automation can co-exist. 

I have these testers and I grow these testers. The 14-yo intern that joined us this week has already been a tester like this while working in pairs and ensembles, and picking up tasks they can do. They've written tests in python for APIs and Robot Framework for GUIs, and found critical bugs in ongoing features. 

Hire for potential. Hire for growth. Hire for learning. Hire for attitude. 

If the attitude is missing the power of "yet" as in "I don't automate, yet", or "I don't design versatile tests, yet" it may be a better idea to invest time into someone who already harnesses the power of yet. I require working with code from a tester. But just as much, I require them to be ready to become excellent testers in its own right. 

There are Plenty of Ways to Talk about Exploratory Testing

Out of all things Ministry of Testing, I am part of a very small corner. I speak at the independent meetups even when they fly under that flag, I speak at their sessions if they invite me (which they don't), and I am a silent lurker on a single channel, exploratory-testing, on their slack. Lurking provided me a piece of inspiration today, an article from Jamaal Todd. 

Jamaal asked LinkedIn and Reddit about Exploratory Testing to learn that less than 10 % of his respondents don't do exploratory testing, 50-70% give it a resounding yes and there's a fair portion of people who find it at least sometimes worth doing out of the 400 people taking time to click on the polls. 

What Jamaal's article taught me is that a lot of people recognize it as something of value, and it surprised me a little bit. After all as we can see in the responses by one of the terrible twins of testing in the slack, they are doing a lot of communication around the idea that exploratory testing does not exist.

It exists, it is doing well, and we have plenty of ways to talk about it which can be really confusing. 

When I talk of exploratory testing, I frame my things as Contemporary Exploratory Testing. The main difference in how I talk about it is that it includes test automation. Your automated tests call you to explore when they fail, and they are a platform from which you can add power to your exploration. Some of them even do part of the exploration for you.

Not everyone thinks of exploratory testing this way. The testing communities tried labeling different ideas a few decades ago with "schools of testing", and we are still hurting from that effort. When the person assigning others labels does so to promote their own true way, the other labels appear dismissive. "Context-driven" sounds active, intentional. But "Factory" is offensive. 

One of the many things I have learned on programming that naming is one of the hard problems. And a lot of times we try to think in terms of nailing the name early on, whereas we can go about and rename things as we learn their true essence. Instead of stopping to think about the best name, start with some name, and refactor. 

So, we have plenty of ways to talk about exploratory testing, what are they? 

1. Contemporary Exploratory testing. It includes automation. Programming is non-negotiable but programming is created from smart people. The reason we don't automate (yet) is that we are ramping up skills. 
2. 3.0 Exploratory testing. It does not exist. There is only testing. And the non-exploratory part we want to call 'checking', mostly to manage the concern that people don't think well while they focus on automation. Also known as RST - Rapid Software Testing. It is all exploratory testing.  
3. Technique Exploratory Testing. We think of all things we have names for, and yet there is something in testing that we need to do to find the bugs everything else misses. That we call exploratory testing. Managing this technique by sessions of freedom is a convenient way to package the technique. 
4. Manual Exploratory Testing. It's what testers who don't code do. Essential part of defining it is that automation is not it, usually motivated by testers who have their hands full of valuable work already now without automation. 
5. Session-based Exploratory Testing. Without management with sessions exploratory testing isn't disciplined and structured. Focus on planning frequently what we use time on and ensure there is enough documentation to satisfy the organization's documentation needs we aren't allowed to renegotiate. 

Lets start with these. Every piece of writing there is on exploratory testing falls into one of these beliefs. The thing is, all that writing is useful and worth reading. It's not about one of these being better, but about you being able to make sense in the idea that NOTHING is one thing when people are involved. 

I invite you all to join the conversation on all kinds of exploratory testing at Exploratory Testing Slack. Link is available with main page of Exploratory Testing Academy. 

 

The Power of Three Plans

This week, I have taken the inspiration from discussions at FroGSConf last week, and worked on my test plans. And today, I am ready to share that instead of creating one plan, I created three - I call this the power of three. Very often different things will serve you best for what you are trying to plan for, and for the things I wanted, I couldn't do with one. 

The 1A4 Master Test Plan

The first plan I created was a master test plan. Where I work, we have this fairly elaborate template from the times when projects were large and not done in an agile fashion. That plan format overemphasized thinking of details prematurely, but has good ideas behind it, like understanding the relationship of different kinds of testing we will or will not do. 

Analyzing it, I could condense the relevant part of the plans into one A4 with many things that are specific to the fact that we are building embedded systems. 

While I don't think I wrote down anything so insightful into the plan that I could not share the actual plan I created for my project, I opt on the safer side. We did not tick some of these boxes, and with this one-glimpse plan we could see which we did not tick, and had conversation on one of them we should be ticking even if we didn't plan to. 

You can see my plan format has five segments:

• Quality Target describes the general ideas about the product that drive our idea of how we think around quality for this product. 
• Software Unit & Integration Testing describes the developer-oriented quality practices we choose from generally. 
• Hardware Testing addresses the fact that there is a significant overlap and dependency we should have conversations on between hardware and system testing. 
• System Testing looks at integrated features running on realistic hardware, becoming incrementally more complete in agile projects. 
• Production Testing addresses the perspectives of hardware being individuals with particular failure modes, and something in assembly of a system, we have customer-specific perspectives to have conversations on. 

For us, different people do these different tests but good testing is done through better relationships between the groups, and establishing trust across quality practices. The conversations leading up to a plan have taken me months, and the plan serves more as a document of what I facilitate than a guideline of how different people will end up dealing with their interdependent responsibilities. 

We could talk about how I came up with the boxes to tick - through workshops of concepts people have in the company and creating structure into the many opinions. A visual workshop wins over writing a plan but we could talk about those in another post later. 

The System Test Strategy

The second plan I created was inspired by the fact that we are struggling with focus. We have a lot of detail, and while I am seeing a structure within the detail, I did not like my attempts of how I wrote it down before. On the course I teach for Exploratory Testing Academy, I have created a super straightforward way of doing a strategy by answering three questions and I posted a sample strategy from the course on twitter. 

I realized I had not written one like this for the project I work in, and got to work and answered those questions. This particular one I even shared with my team, and while I only got comments from one, at their perception was that it was shining light on important risks and reactions in testing. 

In hindsight, my motivation for writing this was two fold. I was thinking of what I would say to the new tester on the ideas that guide our testing as they start in a week, and I was thinking what would help me in pruning out the actions that aren't supporting us in a tight schedule we have ahead of us. 

This plan is actually useful to me in targeting the testing I do, and it might help with some in-team conversations. I accept that no document or written text ever clears it all, but it can serve as an anchor for some group learning. 

The Test Environments Plan

The third plan I produced is a plan of hardware and connections into test environments. If there is one thing that does not move very agile fashion, it is that of getting hardware. And I am not worried of the hardware we build ourselves, but on the mere fact that we ordered 12 mini-PCs off the self type in May, and we expect currently to receive them in December. There's many things in this space that if you don't design in advance, you won't have when you need it. The hardware and systems are particularly tricky in my team with embedded software, since we each have our own environment, we have many shared environments and some environments we want to ensure have little to no need of rebooting to run particular kinds of tests on. 

So I planned the environments. I drew two conceptual schematics of end-to-end environments with necessary connections, separated purposes to different environments, addressed the fact that these environments are for a team of 16 people in my immediate vicinity, and for hundreds of us over the end to end scenario. 

It was not the first time I planned environments, and the main aspects I did this week on this plan is ensuring we have what we need for new hires and new features coming in Fall '21, and that we would have better capabilities in helping project discuss cost and schedule implications to not having some of what we would need. 

The Combination

So I have three plans: 

• The 1A4 master test plan
• The system test strategy
• The test environment plan

For now I think this is what I can work with. And it is sufficient to combine them into just links to each of the three. Smaller chunks are more digestible, and the audiences differ. The first is for everyone testing for a subsystem. The second is for system testing in the subsystem, in integration with other subsystems. The third is for the subsystem to be reused by the other teams this subsystem integrates with. 

I don't expect I need to update any of these plans every agile iteration we do, but the ideas will evolve while they might stand the test of time for next six months. We will see.