Friday, July 1, 2022

Testing on THEIR production

Many years ago, a candidate was seeking employment as software tester for a team I was interviewing for. The candidate had done prep work and tested the company's web site looking for functional, performance and security problems. They had caused relevant load (preventing other's from using the site), found functionalities that did not match their expectations and had ideas of possible vulnerabilities. They were, however, completely oblivious to the idea that other organisations production environments are available for *fair use* as per *intended purposes* and testing is not an intended purpose of production environments. They had caused multiple denial of service attacks to a site that was not built to resist those and considered it a success. We did not. We considered it unethical, borderlining illegal, and did not hire.

For years to come, I have been teaching on every single course that we as testers need to be aware of not only what we test, but where we test too. THEIR production isn't our test environment. 

When I discovered a security bug in Foodora that allowed me to get food without paying, I did my very best on not hitting that bug because I did not want to spend time on reporting it. THEIR production was not my test environment. Inability to avoid it lead to some folks in the security community speak poorly of me as I was unwilling to do the work but mentioned (without details) that such a problem existed, after I had done the work I did not want to do on helping them fix it. They considered that since I knew how to test (and was more aware of how the bug could be reproduced), my responsibilities were higher than a user's. I considered requiring free use of my professional skills unfair. 

What should be clear though: 

Other organisations' production is not your test environment. That is just not how we should roll in this industry.

When I teach testing, I teach on other people's software deployed to my own test environment. When I test in production, I do so because my own company asks and consents to it. When I test on other people's production, I do that to provide a service they have asked for and consented to. 

There are some parallels here to web scraping which isn't illegal. The legal system is still figuring out "good bots" and "bad bots", requiring us to adhere to fair use and explicitly agreed terms of use to protect data ownership. 

Building your scrapers and testing web sites are yet a different use case to running scrapers. When building and testing, we have unintentional side effects. When testing in particular, we look for things that are broken and can be made more broken by specific use patterns.

Testing on someone else's production isn't ethically what we should do even if legally it may be grey area. We can and should test on environments that are for that purpose. 

Regularly I still come across companies recruiting with a take-home assignment of automating against someone else's production. Asking a newer tester to show their skills by potentially causing denial of service impacts without consent of the company whose site is being tested is not recommended. Would these people have the standing to say no - most likely not. 

So today I sent two email. One to a testing contractor company using a big popular web shop as their test target letting them know that they should have permission to make their candidates test on other people's production. Another to the big popular web shop to let them know which company is risking their production for their test recruiting purposes. 

The more we know, the more we can avoid unintentional side effects but even then - THEIR production isn't your test environment. Stick to fair use and start your learning / teaching on sites with consent for such pattern. 

Wednesday, June 29, 2022

Fundamental Disagreements

There are things in testing that never get old for me. The intellectual puzzle of uncovering the answer key to bugs that matter, and the dynamics of removing friction from uncovering bugs to such a level that the bugs never really existed. Figuring out communication that conveys things the people in this particular conversation would otherwise miss, with realisation that writing a specification is the wrong mindset on a learning problem. The results in efficiency and effectiveness, and building something of relevance that works. 

As a tester, while I center information about products and organizations, I'm also an active agent in the system that creates the products.  We're builders, building together.

I felt the need of saying this once again seeing Michael Bolton proclaim: 

So, for the thousandth time: testing does not make things better. Weighing yourself does not improve your health. Weighing yourself informs decisions about what you might do with regard to your health.

Me and my tester colleagues in projects are more than a measurement device like a scale. If we were like a scale, we would be commodity. We choose information we produce based on risks, and make decisions when we call out risks we focus on. We find information and choose to present and act on it. We have agency and a place at the table to do something with that information. 

Testing is about making things better. It is about selecting the information that would be valuable enough to make things better. And it is about timely application of that information, sometimes so early that we avoid making certain mistakes. To make things better, we need to be founded in empiricism and experimentation. We learn about the state of things and we actively change the state of things every single day at work. We don't decide alone but in collaboration with others. 

It's not enough to know, we have long since moved to the combination of knowing AND doing something about it. 

We want to know some things continuously and we create test automation systems to replenish that information in a world that keeps moving, that we want to move. 

We want to know some things deeply, and spend time thinking about them with product as our external imagination, without caring to capture all of that to test automation systems. 

We build better people, better products and better societal impacts by actively participating in those conversations. We consume as much information as we produce. 

We can decide and we should. 

While this is a fundamental disagreement, acknowledging it as such is what could move us forward.

Monday, June 20, 2022

Untested testing book for children

Many moons ago when I had small children (I now have teenagers), I saw an invite to a playtesting session by Linda Liukas. In case you don't know who she is, she is a superstar and one of the loveliest most approachable people I have had the pleasure of meeting. She has authored multiple children's books on adventures of Ruby and learning programming. I have all her books, I have purchased her books for my kids schools and taught lower grades programming with her brilliant exercises while I still had kids of that age. For years I have been working from the observation that kids and girls in particular, are social learners and best way to transform age groups is to take them all along for the ride. 

The playtesting experience - watching that as parent specializing in testing - was professional. From seeing kids try out the new exercises to the lessons instilled, my kids still remember pieces of computer that were the topic of the session, in addition to the fact that I have dragged them to fangirl Linda's work for years. 

So when I heard that now there is a children's book by another Finnish author that teaches testing for children, I was intrigued but worried. Matching Linda's work is a hard task. Linda, being a software tester of a past while also being a programmer advocate and a renowned author and speaker in this space, sets a high bar. So I have avoided the book "Dragons Out" by Kari Kakkonen until EuroSTAR now publicised they have the book available on their hub.

However, this experience really did not start on a good foot. 

First, while promotional materials lead me to think the book was available, what actually was available was an "ebook" meaning one chapter of the book and some marketing text. Not quite what I had understood. 

Second, I was annoyed with the fact that the children's book where pictures play such a strong role is not promoted with the name of the illustrator. Actually, illustrator is well hidden and Adrienn Szell's work does not get the attribution it deserves by a mention on the pages that people don't read. And excusing misattributing a gifted artists work by not allocating her as second author works against my sense of justice. 

So I jumped into the sample, to see what I get. 

I get to abstract to start with annoyance. It announces "male and female knights" and I wonder why do we have to have children's books where they could be just knights, or at least boys/girls or men/women over getting identified by their reproductive systems. Knights of all genders please, and I continue.

Getting into the book beyond the front page keeping Adrienn invisible, I find her mentioned. 

Ragons. Cand. Typos hit me next. Perhaps I am looking at an early version and these are not in the printed copy? 

Just starting with the story gives me concerns. Why would someone start chapter 1 of testing to children with *memory leaks*? Reading the first description of a village and commentary of it representing software while sheep are memory, I am already tracking in my head where the story can be heading. 

For a village being software, that village is definitely not going to exist in the sheep? I feel the curse of fuzzy metaphors hitting me and continue.

Second chapter makes me convinced that the book could use a translator. The sentences feel Finglish - a translation from Finnish to English. Gorge cannot really be what they meant? Or at least it has to be too elaborate for describing cracks into which sheep can vanish? Sentences like "sheep had stumbled into rock land" sounds almost google translated. The language is getting in the way. "Laura began to suspect that something else than dangerous gorges was now." leaves me totally puzzled on what this is trying to say.

Realising the language is going to be a problem, I move to give less time to the language, and just try to make sense of the points. First chapter introduces first dragon, and dragons are defects. This particular dragon causes loss of sheep which is loss of sheep. And dragons are killed by developers who are also testers and live elsewhere. 

We could discuss how to choose metaphors but they all are bad in some ways, so I can live with this metaphor. There are other things that annoy me though.

When a developer makes and error, she is a woman. That is, when we are introduced in explanation text dragons as defects, we read that "developer has made an error in her coding". Yet, as soon as we seek a developer to fix it, we load on a different gender with "he is known to be able to remove problems, so he is good enough". Talk about loading subliminal gender roles here. 

What really leaves me unhappy is that this chapter said *nothing* about testing. The testing done here (noticing systematically by counting sheep every day) was not done by the knights representing developer/testers. The book starts with a story that tells that dragons just emerge without us leaving anything undone, and presents us *unleashing the dragons* as saviors of the sheep instead of responsible for loss of the sheep in the first place. The book takes the effort of making the point that knights are not villagers, developers/testers are not users and yet leave all of testing for the villagers and only take debugging (which is NOT testing) on the developers/testers. 

If it is a book about testing, it is a book about bad testing. Let's have one developer set up fires, wait for users to notice it and have another developer extinguish the fire! Sounds testing?!?!? Not really. 

On the nature of this red dragon (memory leak), the simplifications made me cringe and I had to wonder: has the author ever been part of doing more than the villagers do (noting sheep missing) with regards to memory leaks? 

This is a testing book for children, untested or at least unfixed. Not recommended. 

Unlearning is harder than learning right things in the first place, so this one gets a no from me. If I cared of testing the book, setting up some playtesting sessions to see engagement and attrition of concepts is recommended. However, this recommendation comes late to the project. 

Monday, June 13, 2022

Testing, A Day At a Time

A new day at work, a new morning. What is the first thing you do? Do you have a routine on how you go about doing the testing work you frame your job description around? How do you balance fast feedback and thoughtful development of executable documentation with improvement, the end game, the important beginning of a new thing and being around all the time? Especially when there is many of them developers, and just one of you testers. 

What I expect is not that complicated, yet it seems to be just that complicated. 

I start and end my days with looking at two things: 

  • pull requests - what is ready or about to be ready for change testing
  • pipeline - what of the executable documentation we have tells us of being able to move forward
If pipeline fails, it needs to be fixed, and we work with the team on the idea of learning to expect a green pipeline with each change - with success rates measured over last 10 2-week iterations being 35 - 85 % and a trend that isn't to the right direction with an excuse of architectural changes. 

Pull requests are giving me a different thing that they give developers, it seems. For the they tell about absolute change control and reality of what is in test environment, and reviewing contents is secondary to designing change-based exploratory testing that may grow the executable documentation - or not. Where Jira tickets are the theory, the pull requests are the practice. And many of the features show up with many changes over time where discussion-based guidance of the order of changes helps test for significant risks more early on. 

A lot of times my work is nodding to the new unit tests, functional tests in integration of particular services, end to end tests and then giving the application a possibility to reveal more than what the tests already revealed - addressing the exploratory testing gap in results based on artifacts and artifacts enhanced with imagination. 

That's the small loop routine, on change.

In addition, there's a feature loop routine. Before a feature starts, I usually work with a product owner to "plan testing of the feature", except I don't really plan the testing of the feature. I clarify scope to a level where I could succeed with testing, and a lot of times that brings out the "NOT list" of things that we are not about to do even though someone might think they too will be included. I use a significant focus on scoping features, scoping what is in a release, what changes on feature level for the release, and what that means for testing on each change, each feature, and the system at hand. 

In the end of a feature loop, I track things the daily change testing identifies, and ensure I review the work of a team not only on each task, but with the lenses of change, feature and system. 

I tend to opt in to pick up some tasks the team owns on adding executable documentation; setting up new environments; fixing bugs and the amount of work in this space is always too much for one person but there is always something I can pitch into. 

That's the feature loop routine, from starting together with me, to finishing together with me. 

The third loop is on improvement. My personal approach to doing this is a continuous retrospective of collecting metrics, collecting observations, identifying experiments, and choosing which one I personally believe should be THE ONE I could pitch in just now for the team. I frame this work as "I don't only test products, I also test organizations creating those products". 

It all seems so easy, simple and straightforward. Yet it isn't. It has uncertainty. It has need of making decisions. It has dependencies to everyone else in the team and need of communicating. And overall, it works against that invisible task list of find some of what others have missed for resultful testing. 

Bugs, by definition, are behaviours we did not expect. What sets Exploratory Testing apart from the non-exploratory is that our reference of expectation is not an artifact but human imagination, supported by external imagination of the application and any and all artifacts. 

Saturday, May 28, 2022

Sample More

Testing is a sampling problem. And in sampling, that's where we make our significant mistakes.

The mistake of sampling on the developers computer leads to the infamous phrases like "works on my computer" and "we're not shipping your computer". 

The mistake of sampling just once leads to the experience where we realise it was working when we looked at it, even if it it clear it does not work as someone else is looking at it. And we go back to our sampling notes of exactly what combination we had, to understand if the problem was in the batch we were sampling, or if it was just that one sample does not make a good test approach.

This week I was sampling. I had a new report intended for flight preparations, including weather conditions snapshot in time. If a computer can do it once, it should be able to repeat it. But I had other plans for testing it.

I wrote a small script that logged in, captured the report every 10 seconds, targeting 10 000 versions of it. Part of my motivation for doing this was that I did not feel like looking at the user interface. But a bigger part was that I did not have the focus time, I was otherwise engaged pairing with a trainee on the first test automation project she is assigned on. 

It is easy to say in hindsight that the activity of turning up the sample size was worthwhile act of exploratory testing. 

I learned that regular sampling on user interface acts as keep alive mechanism for tokens that then don't expire like I expect them to.

I learned that for expecting a new report every minute, the variation of how samples every 10 seconds I could fit in varies a lot, and could explore that timing issue some more.

I learned that given enough opportunities to show change, when change does not happen, something is broken and I could just be unlucky in not noticing it with smaller sample size. 

I learned that sampling allows me to point out times and patterns of our system dying while doing its job. 

I learned that dead systems produce incorrect reports while I expect them to produce no reports. 

A single test - sampling many times - provided me more value than I had anticipated. It allowed testing to happen, unattended until I had time to again attend. It was not automated, I reviewed the logs for the results, tweaked my scripts for the next day to see different patterns, and do now better choices on the values I would like to leave behind for regression concerns. 

This is exploratory testing. Not manual. Not automated. Both. Be smart about the information you are looking for, now and later. Learning matters. 

Friday, May 6, 2022

Salesforce Testing - Components and APIs to Solutions of CRM

In a project I was working on, we used Salesforce as source of our login data. So I got the hang of the basics and access to both test (we called it QUAT - Quality User Acceptance Testing environment) and production. I learned QUAT got data from yet another system that had a test environment too (we called that one UAT - User Acceptance Testing) with a batch job run every hour, and that the two environments had different data replenish policies. 

In addition to coming to realise that I became one of the very few people who understood how to get test data in place across the three systems so that you could experience what users really experience, I learned to proactively design test data that wouldn't vanish every six months, and talk to people across two parts of organization that could not be any more different.

Salesforce, and business support systems like that, are not systems product development (R&D) teams maintain. They are IT systems. And even within the same company, those are essentially different frames for how testing ends up being organised. 

Stereotypically, the product development teams want to just use the services and thus treat them as black box - yet our users have no idea which of the systems in the chain cause trouble. The difference and the reluctance to own experiences across two such different things is a risk in terms of clearing up problems that will eventually happen. 

On the salesforce component acceptance testing that my team ended up being responsible for, we had very few tests in both test and production environments and a rule that if those fail, we just know to discuss it with the other team. 

On the salesforce feature acceptance testing that the other team ended up being responsible for, they tested, with checklist, the basic flows they had promised to support with every release, and dreamed of automation. 

On a couple of occasions, I picked up the business acceptance testing person and paired with her on some automation. Within few hours, she learned to create basic UI test cases, but since she did not run and maintain those continuously, the newly acquired skills grew into awareness, rather than change in what to fit in her days. The core business acceptance testing person is probably the most overworked person I have gotten to know, and anything most people would ask of her would go through strict prioritisation with her manager. I got a direct route with our mutually beneficial working relationship. 

Later, I worked together with the manager and the business acceptance testing person to create a job for someone specialising in test automation there. And when the test automation person was hired, I helped her and her managers make choices on the tooling, while remembering that it was their work and their choices, and their possible mistakes to live with. 

This paints a picture of a loosely coupled "team" with sparse resources in the company, and change work being done by external contractors. Business acceptance testing isn't testing in the same teams as devs work, but it is work supported by domain specialists with deep business understanding, and now, a single test automation person. 

They chose a test automation tool that I don't agree with, but then again, I am not using that tool. So today, I was again thinking back to the choice of this tool, and how testing in that area could be organized. As response to a probing tweet, I was linked to an article on Salesforce Developers Blog on UI Test Automation on Salesforce. What that article basically says is that they intentionally hide identifiers and use shadow DOM, and you'll need a people and tools that deal with that. Their recommendation is not on the tools, but on options of who to pay: tool vendor / integrator / internal.

I started drafting the way I understand the world of options here. 

For any functionality that is integrating with APIs, the OSS Setup 1 (Open Source Setup 1) is possible. It's REST APIs and a team doing the integration (the integrator) is probably finding value to their own work also if we ask them to spend time on this. It is really tempting for the test automation person in the business acceptance testing side to do this too, but it risks delayed feedback and is anyway an approximation that does not help the business acceptance testing person make sense of the business flows in their busy schedule and work that focuses on whole business processes. 

The article mentions two GUI open source tools, and I personally used (and taught the business acceptance testing person to use) a third one, namely Playwright. I colour-coded a conceptual difference of getting one more box from the tool over giving to build it yourself, but probably the skills profile you need to work so that you create the helper utilities or that you use someone else's helper utilities isn't that different, provided the open source tool community has plenty of online open material and examples. Locators are where the pain resides, as the platform itself isn't really making it easy - maintenance can be expected and choosing ones that work can be hard, sometimes also preventively hard. Also, this is full-on test automation programming work and an added challenge is that Salesforce automation work inside your company may be lonely work, and it may not be considered technically interesting for capable people. You can expect the test automation people to spend limited time on the area before longing for next challenge, and building for sustainability needs attention. 

The Commercial tool setup comes to play by having the locator problem outsourced to a specialist team that serves many customers at the same time - adding to the interest and making it a team's job over an individual's job. If only the commercial tool vendors did a little less misleading marketing, some of them might have me on their side. The "no code anyone can do it" isn't really the core here. It's someone attending to the changes and providing a service. On the other side, what comes out with this is then a fully bespoke APIs for driving the UI, and a closed community helping to figure that API out. The weeks and weeks of courses on how to use a vendors "AI approach" create a specialty capability profile that I generally don't vouch for. For the tester, it may be great to specialise in "Salesforce Test Tool No 1" for a while, but it also creates a lock in. The longer you stay in that, the harder it may be to get to do other things too. 

Summing up, how would I be making my choices in this space: 

  1. Open Source Community high adoption rate drivers to drive the UI as capability we grow. Ensure people we hire learn skills that benefit their career growth, not just what needs testing now.
  2. Teach your integrator. Don't hire your own test automation person if one is enough. Or if you hire one of your own, make them work in teams with the integrator to move feedback down the chain.
  3. Pay attention to bugs you find, and let past bugs drive your automation focus. 


Thursday, May 5, 2022

The Artefact of Exploratory Testing

Sometimes people say that all testing is exploratory testing. This puzzles me, because for sure I have been through, again and again, frame of testing in organisations that is very far from exploratory testing. It's all about test cases, manual or automated, prepared in advance, maintained while at it and left for posterity with hopes of reuse. 

Our industry just loves thinking in terms of artefacts - something we produce and leave behind - over focusing on the performance, the right work now for purposes of now and the future. For that purpose, I find myself now discussing, more often, an artefact of answer key to all bugs. I would hope we all want one, but if we had one in advance, we would just tick them all of into fixes and no testing was needed. One does not exist, but we can build one, and we do that by exploratory testing. By the time we are done, our answer key to all bugs is as ready as it will get. 

Keep in mind though that done is not when we release. Exploratory testing tasks in particular come with a tail - following through to various timeframes on what the results ended up being, keeping an attentive ear directed towards the user base and doing deep dives in the production logs to note patterns changing in ways that should add to that answer key to all the bugs. 

We can't do this work manually. We do it as combination of attended and unattended testing work, where creating capabilities of unattended requires us to attend to those capabilities, in addition to the systems we are building. 

As I was writing about all this in a post on LinkedIn, someone commented in a thoughtful way I found a lot of value in for myself. He told of incredible results and relevant change in the last year. The very same results through relevant change I have been experiencing, I would like to think. 

With the assignment of go find (some of) what others have missed we go and provide the results that make up the answer key to bugs. Sounds like something I can't wait to do more of!