Deep Exploratory Testing

There's a famous saying by Linus Torvalds:

Given enough eyeballs, all bugs are shallow. 

Crowdsourcing references often like to quote this, pointing out that out of the bugs we could find in testing, the users in production end up finding over masses all the relevant ones, even if they did not report. A crowd could do well in hitting a bunch of bugs.

For the purposes of me doing and guiding exploratory testing, I find it really beneficial to think in terms of shallow vs. deep testing. Shallow can be done with less skills, and with less time. Deep testing requires more skills, more insights, a foundation of learning that is built in layers and requires time.

Many people find that agile somehow guides them to only doing shallow testing. They feel their testing is always squeezed to the end of the sprints, and that it is so that development schedule is flexible, while testing schedule is fixed. However, they may fail to see the opportunity of testing continuing after the release, focusing on going deeper.

Shallow testing find shallow bugs. Shallow bugs are easy to find, they are obvious and would become a problem in production immediately. Deep testing finds deep bugs. It may lead us shallow bugs that just take a bit more of an effort to see, combinations and conditions that take time to set up. But it also may lead us to bugs some don't consider bugs: things that threaten the value of the product, things that should be different to be better.

Going deep happens in layers. You don't repeat the same, but you go further, deeper. You start before implementing. You continue while implementing. You don't stop for releasing. You don't have to, because you are not on a project. Agile made it a continuous process where there is no end.

Sum it up, and it totals to deep testing. Miss the skills, and all you get is shallow. The additive way of doing testing is not regression testing. It is finding new perspectives and exploratory testing is the core practice in doing that. 

Outing a Skeleton to Reclaim Control

When I told a friend I had allowed a boyfriend to take nude images of me, they were visibly worried. They recounted stories of how it had turned bad, and I brushed it off without paying attention. I was an adult, in a committed consensual relationship that involved him traveling a lot. I took the chance to make someone I really cared for happy. And he was happy.

He enforced more images and texts by being happy. By making me feel he appreciated me writing sex stories of our time together. But he also enforced this by messages of how hard being monogamous was for him, how important it was that he would be getting what he needed. And I did my share, writing over 50 stories.

As years passed, we grew apart, and I could no longer cope with the way he left me feeling: blackmailed, on the edge of receiving a call in the middle of the night of "following his heart". When I finally got that call I was afraid of all these years, I realized my fear had become a self-fulfilling prophecy. His heart had found someone new, and I was free of the nagging feeling this would happen one day. Like so many others, I realized we were never right for each other - our values were different. 

With the change of status, the materials I had created for in-relationship entertainment became an issue. I asked him to delete it. He refused. I asked again, over and over. He refused. 

For a few hours, I asked twitter until a friend contacted me to take the tweet down. She said it was hurting my professional image. It wasn't helping, my ex was not budging on the deletion. Like he said on his worst quality: "I'm right, a lot". He wasn't but he would remain stubborn. 

For a month and a half, I have been trying to convince him without success. Instead, I've successfully turned the feeling of powerlessness into nightmares. With long consideration, I'm outing this skeleton in attempt to reclaim control. 

In any sexual relationships between two adults, consent plays a role. I have expressed in so many words that he does not have my consent now or ever again to itemize me for his sexual pleasure. There's plenty of porn out there where consent is available, this material needs to be deleted. 

I'm not comfortable with the risk of revenge porn even if I believe he would not do it intentionally. I'm not comfortable with him ever using me as an item for his pleasure, and he should know that from the fights that lead to us not being a couple. 

My consent is mine to give, and mine to take away. He does not have it for use of this material.

I'm not comfortable with this material being a skeleton in my past, haunting my future. It exists. I've said it. But it only exists with him, and only because he refuses to delete it. 

I would hope people who know Llewellyn Falco  would manage to talk sense to him. I clearly don't manage to do so. His parents don't manage to do so at an adult age when he is right when he isn't.

UPDATE Oct-10th: Llewellyn confirmed he has deleted the materials. Thanks for support everyone. 

I've investigated my options legally, but unfortunately I'm not living in a country that has yet understood the idea of pre-emptive decisions on revenge porn. Existence of it threatens me professionally and privately much more than reclaiming my control to this information. 

When I talk to someone like me, like my friends talked to me years ago I have a personal experience to share on why you should *never* create material like this. Love fades. And that kind of material, intended for the relationship, may not be part of the contract to clear when the relationship ends. 

This material hurts women disproportionately. 

Experiment away: Daily Standup Meetings

In agile, many of of us speak of experiments. There's a lot of power with the idea of deferring judgement and trying things out, seeing what is true through experience rather than letting our mind fool us with past experiences we mirror into what could work.

Experimenting has been my go to way of approaching things. The best thing coming out of it for me has been mob programming. A talk four years ago by Woody Zuill introducing something I intellectually knew I would hate to do, that ended up transforming not just my future but the way I see my past. With cognitive dissonance - the discomfort in the brain when your beliefs and actions are not in sync - with mobbing my beliefs got rewritten. If I was ever asked for top three advice on what to do, experimentation would be on my list, as well as not asking for permission and stopping list-making to actually get stuff done.

Experiments with practices and people are not really very pure, they are more like interventions. The idea of trying before judging has power. But we keep thinking that for the same group, we could try different options without options expiring. The real thing is however that when we do one thing, it changes the state of our system of humans. They will never be as they were before that experience and it closes doors. And as a door closes, other one opens. Experimentation mindset moves us into states that enable even radical changes when the state is right for that transition to happen.

My internal state: Dislike of meetings

Four weeks ago, our team had just received three new members all of a sudden. Our summer trainee had changed into a part time appearance as school required their attention. My colleague appeared to feel responsible for helping people succeed so they turned to their default behavior: managing tasks on Jira, passing their information in writing and turning half of my colleagues into non-thinking zombies working in the comfort of "no one wrote that in the Jira ticket". We talked about our challenges, and they suggested we need to bring back daily meetings. I recognized my immediate strong negative response and caught it to say: "Yes, we should experiment with that".

I left the discussion feeling down. I felt like no one understood me. I felt like the work I loved was now doomed. I would have to show up in a standup meeting every day, after all I was the manager of the team. I would have to see myself stopping work half an hour before the meeting not to be late (being late is a big personal source of anxiety) and see again how a regular meeting destroys my prioritization schemes and productivity. It being in the middle of the day, I would again start working at a schedule that is inconvenient to me just to make space for uninterrupted time.

I knew I had worked hard to remove meetings from my job (unlike most managers around me, where meetings are their go-to mechanism for doing their work) and now the new joiners were forcing me to go back to a time I was more anxious and unhappy.

It's Just Two Weeks

Framing it as experiment helped me tell myself: "It is just two weeks, you can survive two weeks."

I sent the invites to the agreed time, showed up every day, tried hard to pay attention to sharing stuff of value and focused my energies in seeing how others were doing with the daily meetings.

I was seeking evidence against my strongly held stance.

I learned that there are many different negative reactions daily meetings can bring forth.

• Some people share every bathroom break they took, every trouble they run into and focus on explaining why it is so hard for them to not make progress. 
• Some people come with the idea of time boxing and mention always two things. You have to say something, so they choose something they believe they need to say.
• Some people report to others, some people invite others into learning what they've learned, others pass work forward. 
• Some people have low idea of their contributions and frame it into saying things like "I tested since yesterday, I will test more by tomorrow" - day after day. 
• Some people collect the best of the last day to share in the meeting, and hold on to information for that meeting instead of doing the right thing (sharing, working with others) when they come to the information. 
• Some people are happy with the meeting because they have NEVER tried any other options: pairing, mobbing, talking freely throughout the day, having a culture where pulling information is encouraged to a level where your questions always have the highest priority. 

Only one of us expressed they liked the meetings after two weeks. We did not make them work really well. We did not find a recipe that would bring out the best of our collaboration in those meetings. I could not shake the feeling that I was drowning into my "soul-sucking place", agile with rituals without the heart. So we agreed to stop. 

Things already changed

We could say the experiment failed. FAIL as in First Attempt in Learning. It did not stick. But it changed us. It made the problem some people were feeling visible that we did not talk about the right stuff and help our newbies. It changed the way people now walk to one another, giving us a way of talking of the options to a daily meeting. It changed how the team was sharing their insights on teams channel. 

Things were better, because the experiment opened us up for other possibilities. 

Experiments in agile teams are not really experiments. They are interventions that force us to change state.

So, Experiment Away! We need to be interrupted.

Managing Testing based on Threads

Session-based Test Management is a form of managing testing based on sessions - time boxes where ideally we could maintain focus, have a clear charter and generate some metrics by counting things around time boxes and results. I believe it is not one practice but many, but way too many ways we manage testing based on session get bundled with the original description of a technique that is very specific.

Later on emerged the idea that sometimes assuming you could maintain focus for a time box may not be possible. You might be interrupted, for various reasons. You may be jumping between this thing and that thing, and that way of managing testing was then dubbed Thread-Based Test Management.

Here, I wanted to describe one very recent experience from my work, that utilized managing testing based on threads, and some observations around how we ended up organizing and what unfolded in the activity.

Yesterday, we were wrapping up a release. Making a release in general is still a bit of an effort in my team, even if we have moved from it taking a week into it taking a few hours. But yesterday's release was special. It was a release we were shaping together to introduce two major product upgrade paths. each with their own logic, risk of dependencies in a complex environment and need of many pairs of eyes in verifying the functionality, each configuration and environment and problems we may be experiencing and resolving on the go.

We had strategized on a whiteboard the day before, in a session I considered magical. The three people around the whiteboard built on each other, added knowledge, clarified priorities and made an action plan we were ready to execute. The overall work would be done by eight people, which means a bit of coordination is required unless we somehow magically dance just right together.

In the morning, we got the build we through could be the one. We knew there were many things to do, and started a discussion thread - one single thread - in out teams chat. It did not take long for it to grow to hundreds of messages, and confusion to emerge on who was doing what, who was talking about what and what was really done and what was just a miscommunication. A colleague jumping into the discussion at some point exclaimed the awfulness of mixing it all there, and while I and the two others around the whiteboard could track with the model inside our heads, the other five people and everyone watching was in pain.

Seeing it did not work, we came to two immediate solutions.

First a Jira ticket got created by one of us just announcing they would now track the work there because the discussion was awful. They wrote down 13 steps to making a release, mentioning the two upgrade paths each as one step.

Almost same time, I pulled out the tasks intertwined and introduced them as threads: things we'd try to drive through, with some idea of priority and their status as they were named.

• Thread 1 was about "release as usual" - all the moves we had done and practiced for all the previous releases and it was just business as usual. 
• Thread 2 was about the most business critical of the two upgrade paths, and we had not set up the environment to be able to test that at all.
• Thread 3 was about the second upgrade path and we had just identified a blocking bug that needed addressing before it would make sense to finalize it
• Thread 4 was a surprise path of fast forwarding thread 3 in a specific way
• Thread 6 was doing thread 2 in production environments
• Thread 7 was doing thread 3 in production environments

Half a day later, we added thread 5 (because just for the fun of it, of course someone needed to make a joke of an off my one error) on yet adding test automation for thread 2&3 and not accepting to have to do this stuff without help of some tooling ever again. 

Teams wasn't making the thread management that easy, jumping focus where ever someone typed in something, and due to the abilities for us to use teams in anything but one long thread, we did not always trust a comment hit the right one. But they did, and we clarified and worked through each thread. The visibility per thread enabled people to call out for help in a more specific way, identify problems with each of the separate goals we were working towards and make priority calls of what we'd do first and later as this was heavily changing as we identified and investigated problems.

The Jira ticket looks clean, as if one person did it all. The threads in the discussion enabled us to start with a task that we discovered as it was happening, and coordinated many people pitching in information from test results, investigations, availability of fixes and plans for next steps and definition of being done with each of the threads. 

I wanted to share this experience as a reminder that threads may be a thing you want to visualize to share the load. Uninterrupted time is not something to use on all tasks. Sometimes threads are the best thing you can do. And they may enable discovering the work that really needs doing, whereas Jira ticket gets people to deliver what was asked and forget to discover many things that are implicit. 

Navigating in the wrong level of abstraction

It was a big agile conference in the US, and I participated a session on with a demo of mob programming. Demo meant the facilitator of the session called for volunteers, and of the 5 people that ended up in front, I was one.

The group of 5 was two women and three men. It wasn't my first time mobbing, it wasn't my first time mobbing on the particular exercise and I was looking forward to seeing how the dynamics with this team of random people would turn out.

The exercise we were doing followed a routine of example to English, English to test code, test code to implementation.

I remember that session as awful but I have not talked about how awful it was because I was told back then that I just did not understand it right. So today, I wanted to write about the idea of mild gaslighting when someone is seeking acknowledgement for their feelings you think they should not have felt.

The other woman in the group went before me as both navigator and then driver, and made it very clear to everyone it was her first time programming. Mob programming is great in the sense that this is possible, and in addition to remembering her emphasis on not knowing a thing, I remember the joy she felt and expressed on having contributed on creating something. As a total newbie, the group navigated her on the lowest possible level of abstraction. We'd tell her letter by letter what to write, not using conceptual language at all.

So when the next two in the rotation after her were men, the difference of how people navigated her and how they navigated the men was very obvious.

My discomfort came from being bundled together with the other woman. As far as I can see, I did none of the "help me, I know nothing" mannerisms. I sat with the crowd, navigated with the crowd, knew what I was doing. And yet the moment I sat on the driver seat, I was navigated on keystrokes, no concepts. I felt humiliated and paralyzed into a role I hated. The mob navigated me on keystrokes, and I was unable to correct them. I was holding myself together not to ruin things for everyone else.

As soon as the session was over, I expressed my emotions to the facilitator who told me it was all my fault. I would have needed to assert myself by saying "just tell me what the thing is you want me do", I was supposed to not feel like I was just treated as not a programmer but a woman trying programming with a mob, bundled together based on my gender with the other "representative". I missed out on the joy she was feeling, so we were clearly individuals regardless of the bundling.

My lesson of this is that to not offend people in ways they remember years later, you may want to always err on the side of navigating on an abstraction level that is too high rather than too low. Tell the intent and if you see no movement, then tell location and details. If you jump directly to the details, there's a chance you're explaining things your driver won't need, introducing inefficiency as well as making the other feel like they've been weighed on their looks and found not worthy.

It would have felt a lot better if the facilitator would have at least acknowledged my feelings. But as usual, it was a problem with me: me feelings were wrong, as a result of my (lack of) actions.

I rarely volunteered in mobs after this unless they would run a long time, unless I was the only woman or unless the other woman was even more of a programmer in her manners than I was.

Pairing and Mobbing are not the only ways to collaborate

A friend I enjoy connecting with on testing themes sent me a message today. They expressed they had enjoyed my pair testing article on medium, and had an idea they wanted to contribute on type of testing I had not addressed.

This was a form of collaboration where two people (a pair, you may say) worked on two computers, side-by-side on the same task. The two might be intertwined in the activity so that one looks one end, the other another end. And it is very much collaborative. They suggested this would be a model of pairing with two drivers, both with their hands on the keyboard - a separate one though.

In my article, I introduced the traditional style pair testing, where often the navigator would be making notes, deciphering what was going on with the driver who had control. Nothing says the second person couldn't be taking those notes on a computer, but what typically happens is that the notes, in comparison to strong-style pair testing, are of a private nature and don't always reflect what was going on in a manner that is shared between the pair.

Similarly, here with two computers and testing side by side, we can figure out many ways to still work in a very collaborative manner.

I find myself often testing in ways where there is two or more testers in the same space, at the same time, sharing a task while we still are not mobbing or pairing. It's more about sharing energy, and enthusiasm, giving hints of things others can pick up and run with, and continuously coming back together to understand where we are.

I tested that way today with a developer, over a teams chat. We would be calling out our progress and observations, pondering on things that did not work out quite as we thought. The chat version works when people are active in sharing. I absolutely adore this dev for the way he is transforming the team now from his position of seniority, and driving collaboration without pairing forward.

In the past, some of my most fun days have been when we work side by side with other people, testers and developers, growing ideas and solving problems. While formal pairing and mobbing are great and give a structure, the other ways of working together also beat the "alone in a corner" any day. Some tests require many computers, like the test at Finnish Parliament where they had to bring in 200 people serving with the Finnish military forces to control 200 devices, for one test to be completed.

No matter what, we should be learning and contributing. Work is so much fun, and it makes no sense to dumb or numb it down. 

Forgetting what Normal Looks Like

Today I reached 2 years at my current job that I still very much love. There's been mild changes to what I do by changing my title from "Lead Quality Engineer" to "Senior Manager", that shows up mostly in whole team volunteering more easily to do testing tasks and enable testability without me asking any differently than before. There are a few reasons why I love my job that I can easily point to:

• We have team ownership of building a better product. No product owner thinks for us, but some business people, sales people and customers think with us. From an idea to implementation and release, we've seen timeframes in days which is very unusual. 
• It's never too early or too late to test. Choosing a theme to give feedback on, we can work on that theme. With the power in us as a team, the tester in me can make a difference.
• We can do "crazy" stuff. Like kick out a product owner. Like not use Jira when everyone else worships it. Like pair and mob. Like take time for learning. Like have a manager who closes eyes to push the stupid approve button that is supposed to mean something other than "I acknowledge you as a responsible smart individual who can think for themselves". Like get the business managers to not book a meeting but walk in the room to say hi and do magic with us. Like pull work, instead of being pushed anything. 
• We are not alone, not stuck and generally people are just lovely even if there is always room for making us flow better together.

Living a life on the edge of "crazy" to some is fascinating when people don't all have the same past experiences. In particular, in last months we have had new people join who have brought in their past experiences from very different kinds of ways of working: with daily meetings, detailed Jira tickets, thinking for the developer, etc. 

I've been experimenting with finding a better, more fun way so long that I start forgetting what normal looks like. Today I found some appreciation for it. 

At almost two years in the job, I finally started a "Jira cleanup" about two weeks ago. I edited our team query to define what belonged to us: anything marked for the team, and anything marked for any of us that belonged in the team. All of a sudden for those few who had cared for the list based on their past experiences, realized that the list was much more significant that they had realized. About 120 items. We called for an old rule: all other work will wait until we are below 50. We didn't get to it though. Some people cleaned some of things up. Others were busy working on other things. 

Instead of one on one clearing of the personal lists, we called a workshop to share the pain of cleaning the lists. I had no idea what I might be learning with the workshop. 

I learned that half of the people had not used Jira beyond the "look at this individual ticket" use case. Seeing what was on the whole team list - a new experience. Seeing that you can drag-and-drop stuff like post-its - a new experience.  Marking a case into a state or assigning it to a person - a new experience. 

Even with the Jira avoidance I advocate for (fix and forget immediately, track themes on a physical whiteboard) I had not come to understand that I might have missed sharing a basic set of skills for a group of people coming to us from elsewhere, with other expectations of what it would mean.

A healthy lesson of remembering that what is obvious to me may not be such to others. And that building complex things on lessons that are not shared might make less sense to those with less experimentation experiences under their belt.