Assert and Approvals, and Why that Matters

As an exploratory tester, a core to my existence are unknown unknowns. I stumble upon problems - to an extent people like Marit van Dijk call out that "I don't find bugs, the bugs find me". But stumbling upon them is no accident. I intentionally find my way to places where bugs could be at. And when there is a bug, I try to finetune my ability to recognize it - a concept we testers call oracles.

As I'm automating, I codify oracles. In some ways of automating tests, the oracles are multipurpose (like property-based testing, describing and codifying rules that should hold true over generated samples), and sometimes they are very specific.

Both these multipurpose partial oracles and single purpose specific partial oracles are usually things we build as asserts. In the do-verify layers of creating a test automation script, asserts belong in the verify part. It's how we tell the computer what to verify - blocklisting behaviors that cannot be different. Much of our automation is founded on the idea of it alerting us when a rule we create does not  hold true. Some rules are fit to run unattended (which is why we focus on granularity) while others are for attended testing like exploratory unit testing.

Another approach to the same codifying oracles problem comes through approval testing. What if we approached the problem with the idea that a tester (doing whatever magic they do in their heads), would recognize right-enough when they see it, and approve it. That is where approvals come in. It is still in the verify-layer of creating a test automation script but the process and lifecycle is essentially different. It alerts us when things change, giving names to rules through naming tests without a pre-assumed rule of comparing to a golden master.

Approvals in automation increase the chance of serendipity, a lucky accident of recognizing unknown unknowns when programming, and they speak to the core of my exploratory tester being as such.

The Difference in the Process and Lifecycle


When we create the tests in the first place, creating an assert and approval is essentially different:

• An assert is added to codify the pieces we want to verify and thus we carefully design what we will tell us that this worked or didn't. Coming up with that design is part of creating the assert and running the assert (see it fail for simulated errors) is part of creating it.
• An approval is prepared by creating a way to turn an object or aspect of an object into file representation, usually text. The text file would be named with the name of the text, and thus whatever the textual representation of what we are creating is the focus of our design. We look at the textual representation and say "this looks good, I approve", saving it for future comparison. 
• Assert you write and run to see green. Approval you write and run to see red, then you approve to see green. 

When we run the tests and they pass, you see no difference: you see green.

When we run the tests and they fail for a bug we introduced, there is again an essential difference:

• An assert tells us exactly what comparison failed in a format we are used to seeing within our IDE. If run on headless mode, the logs tell what the failed assert was. 
• An approval tells us that it failed and shows the context of failure e.g. opening a diff tool automatically when running within our IDE. Especially on the unit level tests, you would want to run the tests in IDE and fix the cause of failure in IDE, having it all at your fingertips. 

When we run the tests and they fail for a change we introduced, we have one more essential difference:

• An assert needs to be rewritten to match the new expectation. 
• An approval needs to be reapproved to match the new expectation.

When looking for things we did not know to look for, we are again different:

• An assert alerts us to the specific thing we are codifying
• An approval forces us to view a representation of an object, opening us to chances of seeing things we did not know we were seeking.

Back to exploratory and why this distinction matters so much to me

Even as a programmer, I am first and foremost an exploratory tester. My belief system is built around the idea that I will not know the mistakes I will make but I might recognize them when I see them.

I will create automation that I use to explore, even unit tests. Sometimes these tests are throwaway tests that I never want to push into the codebase. Sometimes these tests belong to a category of me fishing for new problems e.g. around reliability and I want them running regularly, failing sometimes. I will keep my eye on the failures and improve the code they test based on it. Sometimes these tests are intended to run unattended and just help everyone get granular feedback when introducing problems accidentally.

With approvals, I see representations of objects (even if I may have to force objects into files creating appropriate toStrings). I see more than I specifically command to show. Looking at a REST API response with approvals gives me EVERYTHING from header and message and then I can EXCLUDE undeterministic change. Creating an assert makes me choose first and moves exploration to the time I am making my choices.

The difference these create matters to my thinking. It might matter to your thinking too.

A Man in Tech Doxed Me for a Copyright Dispute

"I know where you live" is not anyone's wish for a message to receive. Worse yet, "You've been sloppy with sharing your private information" was not true. The private information that lead to the contact was not shared by me but by someone I started writing a book with and never finished before we went our separate ways.

This is how I learned that Llewellyn Falco - a leading figure in the mob programming community - decided it was ok to publish my home address online without my knowledge or consent, leading people to my home by the mere choice of the place he published on. 

This is what the online abuse community knows as doxing, even if this was a mild version of what could still happen. The person contacting me was a privacy enthusiast with no malicious intent making a point that I had been not careful, all the while I had been made vulnerable by someone with awareness that this was not my choice and would never be my choice. 

As a result of this, I took the natural action:

• Requesting Llewellyn Falco to work to take down the information he had illegally posted. He has not responded. I would think both apology and action to correct would be in order. 
• Requesting Llewellyn Falco's lawyer who had posted the info on his behalf to take down the information as registering private information of a European Union resident would not be GDPR-appropriate. 
• Requesting the register holder Llewellyn Falco used to publish my home address to take down the information. 
• Protecting myself by isolating myself from people silently supporting Llewellyn Falco leaving communities we were both part of and blocking our mutual followers on Twitter. 

Out of communities we were both part of, I have taken steps back to Women in Agile who were quick to respond to the choice between the two of us and a small private community of programmers. I felt lucky to have Women in Testing and Makeupconf communities that would never have welcomed him in the first place and I could feel safe confiding in for advice and support. 

If you are no longer seeing me in twitter, you may be one of the 150 people that I blocked. I will unblock with a message that you have made a choice of not following him. I will regularly rerun the blocking script for people who are following him. 

Why Would He Do Such a Thing?

While I feel hurt and offended, I can bring myself to imagine that doxing me was not his intent but it was his impact. Just like with so many other cases we witness around bad actors, they don't mean to hurt when they do. The reason he does this is a copyright dispute. 

Normally people would resolve copyright issues in a court of law and I have welcomed him to invite me there. He chose to take a path of harassment. 

Back in the days, we started writing a book together under the idea of open source. We published early versions on LeanPub. The dispute, as I understand it, comes from our difference in understanding what open source means in the context of us creating a book together for a while and separating as authors before the book is finished. 

I take the perspective that we co-created text that is available openly for both of us to use forward. My name is not available for him to use forward when we part ways. 

He takes the perspective that last published copy on LeanPub ends the project and blocks both of us from continuing on the unfinished work. 

LeanPub, as a platform says nothing of the copyright. So I agree we don't agree and might need to resolve it. That is where court could be helpful should he feel he needs to change status quo. 

I deduct his intent from his actions:

• April 3rd he disputed contents on LeanPub using Github to claim text we had pair written (physically, I had written) on his computer was written by him and that I would think adding to text I had already written was removing his copyright. In fact I recognized his contribution inside the book but continued using my text to finish an unfinished book on LeanPub placeholder that was always mine but I had invited him as a contributor. He needed to ask to take it down because the project was mine and he was a second author while working with me. The book cover has my name first, the LeanPub page was held by me - it was my project and he was a contributor for a while.
  

• April 4th LeanPub made final call blocking the LeanPub book and all books I could write with Mob Programming on the title.  Deducting intent, I call out intent to harm me. Removing me from my marketing platform in attempts to alienate me from the mob programming community for his personal gain. 
• On April 6th I rebooted my mob programming guidebook completely, writing every single word of ideas I had gone through writing before again. This is why my book is available with https://mobprogrammingguidebook.xyz. It is not the same book, it is my fork of the book we never finished, with significant effort used to remove any of the stuff where I disagreed with what I wrote in the first place as compromise to two person co-creation effort. The book is still under construction. 
• On April 7th I learned that an unauthorized private copy of book we had previously distributed on LeanPub was on Github posted by Llewellyn. I hold copyright as the first author, my name is on the cover. I requested to take it down as copyright owners we had only agreed LeanPub as publication place and he had effectively removed that from both of us through a GitHub issue to the project he published it in. He never responded.
• On April 30th, his lawyer registered a US copyright for both of us for book that misrepresents the book he hosts on github presenting him as the first author. I was not made aware of the copyright registration at all. The book was written in Finland and I am the first author. He knew to protect his personal information while disclosing mine. 
• On October 23rd a privacy advocate uses the information from the copyright registration where he publishes my home address to contact me on my sloppiness on handling my personal data. 

The Personal

All of the above is very much business as usual, and even business. However, in my call for action, I want to add the personal. 

Yes, we used to date. 

It did not end pretty.

I had to ask Set Enterprises Inc under GDPR with a threat of fine to remove nudes and sexy stories I had given him while in relationship. He wanted to withhold those. I won and I'm eternally grateful to EU legislation and the mediators in the community a year ago failing to influence him then. 

There was no way our professional collaboration could continue. 

My Call for Action

If you read all of this, I have a request for you and you may choose to dismiss it. I had not asked to renounce him with any of the previous actions, but doxing my home address knowing what that means to a social justice oriented woman in tech, he crossed every single one of my boundaries. 

I would ask you to choose me to be part of your communities but that means that:

• If you follow him, I block you. We can't co-exist.
• If he is welcome as "ally" in your women in ... groups, I will not be welcome as a woman in your women in... group. 
• If he is welcome in an online slack/group, I will not be. I leave quietly, 
• If he is welcome in your conference as speaker, I will not be welcome in the same conference. I will leave as soon as I recognize he is there and facilitate your convenience through creating a speaker rider that states we cannot co-exist.

I wish you would kick him out as he encouraged kicking out "simpleprogrammer" a week ago, deplatforming him but I feel I am not in the place to call for that action. I wish the people would choose to include me by excluding him, but I recognize that since his bad behavior is more private of nature, he can keep his fake ally status and alienate me from the mob programming community I have also in my own right contributed. 

Communities are nothing more than connections of people, and it is my own choice to step to the side for safety. 

This is how women in tech leave. Thanking the people who are there for them, fading into invisible.

Edit 30.10.2019. Let me make this clear: I have done already my part of running blocking scripts and scheduling reruns of it. The only thing to do is change how things are by asking for unblock, conditioned on not following him. This is my equivalent of stepping out of the space he is in.

Our Three Ways to Beta

Working in a product company, the term "beta testing" we pass around in testing certifications actually has meaning and relevance to us. But while you can find a vocabulary definition to it, the real life with what it means is much more interesting and versatile. So let's look at three products and their three ways to do beta.  Each product version when fully taken to use means installation to computers counted in millions so set your eyes away from that one server and think about building something where distributing and installing is a major effort split between the user ("acceptance testing") and the product ("all testing") organizations.

Product A

Product A is targeted for big organizations and whether it is true or not, the day to day is filled with change avoidance, change control and unwillingness to take new versions into use. Product A version the customers have today is good enough as in not causing trouble, and there is no pull to move forward. Every move forward is work that includes acceptance testing of a new version of product A in whatever specifics each customer has around that ensuring standards around disallowing untested (by the customer organization separately) software to sneak into production.

Because releases are work and hard and to be avoided, the product A works with a very traditional release cycle. You can expect to see an annual cycle of planning to be set around the idea that there are two major releases and 2 minor releases. The customers recognize that a version with x.00 is always the major one and as it introduces new major features and changes, you should avoid it employing a conservative strategy. The minor releases, x.01 customers recognize for having what x had, but fixes from the first brave customers, with history showing that the conservative strategy is worth the wait.

With only a few releases a year, each release introduces thousands of changes even if the list of visible new things is only a few items per release. The risk accrued on not releasing is a realistic threat that the customers of the product experience because as hard and well as the product A team tests, the real operational environment is always more versatile and use cases more surprising that a test lab can simulate.

When product A does beta, it is a schedule milestone and warning of uncertain / low quality for customers, a release on top of the four releases. When you add two the two major versions a beta each, you have total of 6 releases a year! And if anyone takes a look at the beta version finding out it does not work in customer environments (again, product team tests hard in test lab before!) the features available for that release could already be somewhat improved for by time of the major release being "RTM" (release to manufacturing). Time between beta and RTM is not for bug fixing though, it is for the second batch of features that never see a beta. Sometimes when the stars align, testing and fixing work happens during this beta instead of running with the next batch of features.

The beta exists to enable a user organization to start their acceptance testing early but no one would expect it to end up in wide production use. That's what the major and minor versions are for.

Product B

Product B is targeted for big organizations too, but the big organizations serve a more individualistic user base. Major and minor releases work very similarly, with annual planning cycle seeing two of each but their size on effort allocated is different. They are not projects where one follows the other, but usually a new major release starts as previous goes out, and a minor release happens on the side. A significant difference comes with size of minor releases, that product B minimizes. Most of fixes are seen within major releases and going out only in the next major release, and doing a minor release is an effort to minimize the scope whereas product A sees it more as a maximizing the scope to get anyone to move to a new release.

The customer sentiment around what a major and minor release means is very much the same as with product A but there is slow yet significant pull to get the major releases out for real users to use. There is some avoiding of change as it is still a project, but it is considered a little less arduous.  And then there is some of exact symmetry of how product A customers would behave, but that is in the minority.  There's rules in place on how many versions are supported, which supports the slow pull.

When product B does beta, it's a pulse release assessing quality continuously, every two weeks. Whatever can be used internally, can be used externally for a selected group of thousands of users. Beta is more of a way of paying for product by telling if it fails, and it very rarely fails in ways users see. Meanwhile it is possible to see ways of failing users don't see through telemetry.

When a release is made available, every single feature it includes has been out in beta. People have been asked specifically about those changes. Some things have been in beta for 6 months, others two weeks. A RTM schedule depends on key features (major risks) having enough time in beta to assess feedback and RTM quality is solid enough so that major releases are regularly considered for use. 

Product C

Product C is targeted for medium sized organizations but lives in a service space where continuous changing of software is already more acceptable practice for customers. Since it is a service, moving from a version to another is in theory invisible to the user organizations, and they've been stripped from possibility to control it. New stuff comes with a cadence that does not show up even as a need to reboot. There are no major and minor releases, just releases every 2-4 weeks.

When product C does beta, it's a time to cancel wider distribution. It's not called beta really, but "early access" (external customers) and "pilot" (internal customers). The time for beta is one week before full availability, and as with product B, things run more on telemetry and seeing failure than hearing back from a real user seeing problems.

Do the differences matter? 

The differences cause some confusion as one product's beta is not another product's beta, and the rules around how to schedule are very essentially different. As a tester, I would never want to go back to product A style of working and from my experiences in continuous delivery as transformative practice, both B and C have a lot of similarities.

It's now been 12 years since I moved product B to the pulse style continuous beta. I've played a central role in defining product C taking the practices further, ensuring the best software we have created is running as a service on the customers machines.

I work from the idea that in the security business of defence, we need to move fast because the offense definitely is. The end users may not understand all the details of what we defend them against, but working with those details day in and day out, I know in my heart the speed matters. Even more in the future than today.

Stop Assigning Me Jira Tasks

People are social. I'm particularly social and need real human connection at work. Assigning me a Jira ticket describing a task that you think is doing fails my needs of connections on so many levels that I needed to stop working and start writing.

The Communication

I appreciate every colleague that drops in (even virtually) and talks to me like people talk to one another. They tell about a problem they have, an aspiration they have, a wish they have in hopes of influencing me to do something for them or rather with them to make the world a better place. I could not be happier.

Surely there are too many things. There is uncertainty. But as we connect, we establish what other things I may have on my mind. We may agree that while they were hoping that they could just dump the work on me, I may be busy elsewhere and this stuff is too important to wait for me, and the person trying to do the dumping could even do it themselves. Sometimes we end up doing it together enabling the idea that I don't need to be the only one doing things like this. All through the connection.

This human to human communication seeking mutual benefits instead of assigning tasks is a source of happiness. The opposite is a source of unhappiness. And Jira tickets where someone is hoping to dump the work on me will never do the same thing as that person talking to me, caring about my response.

"What's the status with ABC-123?" is not how people talk to people they like, value and appreciate.

The Decomposition

When we have a goal to achieve, different people can achieve the goal differently and it does not mean that one way is ultimately the best one. Usually the best ways to achieve goals are ones that teach us the most and help us stay honest about where we really are with the end result. At least personally I'm not happy that we built the perfect "smart inventory" as we imagined it, if it does not serve purposes we had in mind for creating it and I don't see a change in behaviors with use of products with such a thing. I recognize however that other people see success already with accomplishing a task, not assessing whether the outcome is that we are in a better place for it.


When I am given a task in Jira that someone else wrote, I find I treat it as decomposition of a goal. I first spend time recomposing the tasks to the goal, and then re-decompose to ensure I can also feel that the original decomposition is high enough quality that I find my sense of purpose in doing the work assigned to me. This is a lot of energy used.

But it is not just the energy drain that is a problem. It is also the fact that more often than not, I find there are tasks in the negative space and we would end up delivering with quality below standards I can find myself comfortable with. I could do what you asked, but I wouldn't do what you wished and intended.  So instead, why tell me the task, tell me what you wanted in the first place before you decomposed it for me as your decomposition isn't helping me.

If you decompose for me because you think I cannot, how will I learn when you always chew it for me? You may think I should be grateful for the work you do for me to help me but really: nobody's job should be to think for others but to grow others to think. Be there to support, give me feedback but let me do the work that makes me grow.

The Sense of Ownership

As a tester, I have grown to think that I look at things with an end in mind while people working with requirements often look at things with the beginning in mind. We both look all the way, but our emphasis in a different place, and because of it we see things differently.

A value item is not done for me until I have tested it in multiple ways, as per risk unique to each item. There is no recipe I follow for every single one, but there are patterns and heuristics that help me make those decisions. I look at the features we do in production, not only up to production and I learn of what I could improve on my work months, even years after first doing the work harvesting patterns that would prove me I am wrong now - like a true tester, seeking to falsify hypothesis is the way to get closer to being right about things.

I have grown a sense of ownership. I don't seek to avoid blame "I did what ticket ABC-123 said and it did not say to do X" but to learn from mistakes we made. FAIL is a First Attempt in Learning, and we are in it together. Someone else's plan that was wrong was my mistake on not decomposing the plan to find out how it was wrong and I may learn to make my choices differently or accept that the choices I made under conditions I was are choices I would do again, regardless of the result being less than perfect. Who seeks perfection anyway? 


For me a ticket in Jira, closed without doing anything after six months, is an indication I created waste not value in writing that ticket. I don't believe there is inherent value in remembering all the ideas we have had that we did not act on. And looking at the tickets closing, the evidence still suggests that I need to look more of rejecting the hypothesis.

People, as per Pink's book Drive, are creatures looking for Autonomy, Mastery and Purpose. Sense of ownership is how I frame those in my work.

Job Crafting and Why Specialty Titles are Still Relevant

I work with a cozy 11 person DevOps team. I say DevOps, because we are running the development and operations of a fairly reasonable sized (in the millions) user base for a particular windows application product. We do ok, and I really like working on this, with these specific people.

These specific people are all individuals, 6 developers and 5 testers. That is at least the high level categorization. Yesterday was a particularly interesting day and I tweeted about it. 

On this day when two of my team’s testers were implementing features and one of my team’s developers was manually testing for great results, I think back to ‘what’s the dev/tester ratio’.

— Maaret Pyhäjärvi (@maaretp) October 3, 2019

Watching the ratio, thinking it is telling about the work we do in the categories of "dev" and "test" makes little sense. But watching the ratio as how many people hold the space for quality ("what do we really mean by done") and how many people hold the space for smart solutions ("how do we change things for the better") makes a lot more sense.

The testers implement features. The developers implement tests and explore. And this all is very natural. Everyone pitches in within what they can do now, stretching their abilities a little.

I think of the two main roles we hire for as centers from which we operate. When named a tester, you naturally navigate towards caring for quality, feedback and systems to give that feedback. Your home perspective is one of skepticism, wanting to believe that things could be broken. When named a developer, you naturally navigate towards adding functionality and internal quality to enable future work, and delivering changes to users to change their experience with the product. Your home perspective is one of believing in possibilities, seeing how things could work. When there is space for both perspectives in collaboration, better software tends to emerge.

I have been the solo tester with 15 developers around me, and holding space for quality feels different there than it does here. Here I am not alone. And a lot of times I find the best quality advocates are those I would block as developers.

I still call them testers and developers, because they still are testers and developers. But they are not binary. The testers are a little bit more testers than developers. The developers are a little bit more developers than testers. 

Seeking for both helps in hiring. It helps in creating a sense of purpose these people fulfill within the team while allowing the stretch. In the end of the day we need both perspectives and having people who feel ‘home’ in different roles help keep the perspectives strong.

There is no good word to move both into in a way that doesn’t send the message we give up on testers. There are people who want to be testers. Being a tester is great. Yet when seeking this one word, our gravitation is towards making us all developers.

I'm a big believer in job crafting - the idea that no matter what job you hold, you do little (or bigger) things to make it the job you want. The job that looks and feels like you. If you were hired for a purpose, crafting into a place where you forget what you came to do isn't what we seek. Understanding your purpose and value you are hired to deliver is important. But letting that stop you from growing, doing things smart or different would not be right. 

So if a tester developers features, they can still be a tester. If they don't test anymore, they should probably not be testers. Promise of a service that isn't there is just dishonest.