Spying on users - a new form of usability testing

There's all sorts of production monitoring tools we've been using, but I recently run into something different I've been looking into tonight. The tool is called Hotjar and a friend introduced it to me as a tool for usability testing. With the tool, you can see for each user in video format their mouse movements and clicks, and can build more fine-grained ways of analyzing when your users lose engagement on your pages.

How much of a spying tool this is became clear to me today as I went and checked for the first time the recorded uses of my personal landing page.

The red line traces the mouse movements. The red dots indicate clicks. I see what devices and browsers my visitors have used, and how long they've stayed.

Following what my users saw I can test different screen sizes and devices with eyes of my users, without setting the environments up myself. Doing this early on (and fixing), I could prune out problems through testing in production, annoying a limited number of users but coping with my limited ability to cover different combinations.

For now, I'm just blown away with this. And needed to share. I reserve my right to change my mind as always, but for now, I'm just excited. 

Roles and expected contributions

On a remote day of work, there was very little discussion. I was focused on the application over the team's Flowdock channel when a discussion started.

Four developers at the office had started to wonder about a user interface design element on something one of them was working on right now. The question on the channel was directed at the user interface specialist, with a picture: "what if it was this way instead?".

The discussion continued  on defining relationships of concepts to be selected: should you be able to select two at once out of the list or just one? Should there be a "no selection" option too?

The user interface specialist comes up with a conclusion on selection  between radio buttons and combo box, that the discussion boils down to: "There's no reason to hide the selections from the user".

A developer comes back with a consistency argument: there's another element conceptually just like this just right above this and it uses a combo box. Why shouldn't the two be the same?

I enter the discussion, repeating the consistency argument. But I also add a piece of data: in 90+ percent of the cases, the user does not want to make a selection of this. Selecting anything but "no selection" is a special case.

The conclusion changes with the added data. In this case it's natural to have a combo box over the radiobuttons. A design is agreed upon.

I share this story to emphasize that it does not matter whose role is what and what contribution you could expect based on the role. As a tester in the team, I have a lot of empirical data and a keen eye on the real use cases through listening to end users.

Seeing this through a discussion over addressing it through hands-on testing just made everyone's life a little easier. Hat tip to my developers on initiating a discussion that required them effort due to remoteness, and on persistency on caring how it would be. 

Cognitive Dissonance: How I learned to like programming

A remark in an online lean coffee caught my attention: "I don't write automation. I really don't like that work. And there's N developers and just one of me, so I leave the code for the developers".

It could have been me speaking. But it wasn't. It wasn't even someone who has been around me to learn to speak like me. But hearing something I recognized so strongly made me feel the urge I need to share my story of how things changed.

Rewind back a little over a year, and I would have sworn I will never be a programmer. "I really don't like that work" would have been exactly my words. Someone suggesting I could try that would be met with skepticism. You can probably find me saying that in public in this blog that is intended to keep me honest to myself - if you get something out of it that's a plus.

I started mobbing not because I wanted to learn programming, but because I wanted my team to work better together. I wanted to feel my social needs met at work. So in hindsight, little did I know about how the human mind works.

When thinking of changing jobs, I had an interview with a psychologist. Me mobbing with my team was one of the stories I shared with him, and he labelled what might have happened: Cognitive Dissonance.

Cognitive dissonance is a state in which our beliefs and actions are not in aligned. People have this habit of needing feel whole and consistent, and we do that to the extent of rewriting our history and perceptions.

If my foundational belief is that I don't really enjoy programming, I seek to not do it - after all, I believe I don't like it. Theories around cognitive dissonance seem to be saying that we change our beliefs from dissonant actions - not the other way around. When I do something that is against my belief system, it starts a process of rewriting parts of my belief system.

With mobbing, this happened slowly. After all, we were mobbing at most once a week in the office. The extra sessions to learn the teaching technique I organized with the local tech community added some. I always told myself I had other motives to join the mobs, I never really cared for the programming part of it. I wanted developers to learn testing. I wanted to see if I could think fast enough to spot problems without the product as my external imagination. I wanted to see if I could use mobbing to teach my exploratory testing skills to other testers. The coding was just something I had to endure.

But over time, my feelings changed. As I did more of the programming in the mob, it wasn't just that I became more confident with the things I already knew as technical non-programmer or that I picked up pieces while we were doing it. My attitudes started to change, impacting my belief system that I used to define my place and role in the system of creating software.

I went from "I don't write automation. I really don't like that work." to "Let's write some automation. But why focus only on test automation, we have a bigger problem to solve with thinking around code".

My claim is that convincing me on this in advance would have been close to impossible.

So think about it: how could you use cognitive dissonance in giving yourself a chance to change your fundamental beliefs by engaging in work you feel you don't do. Or even more: could you use cognitive dissonance in changing the mind of that difficult developer who never wants to create any unit tests?

Experiment. There's a lot of power there.

The stories you need to tell to product owners

"I could add this new reporting feature in less than a week", says the developer. "But that would not include the cleaning up and changing of components we should do in the same area", he continues. "We probably should just say the feature takes a month, because we have to be able to change the components too", he concludes.

Ever heard this monolog? The idea that you need to come up with a different story for your product owner, because they understand only features and not the technical maintenance work. The idea that all the technical maintenance work should be baked into the work, even if it transforms the scope of the work?

We ended up reframing this discussion. Let's be nice to the product owners, and enable them the fact that the feature is in the production sooner. It gets real feedback sooner, and it starts paying itself back sooner. And let's trust that it is ok to do the needed cleanup after, without baking it in.

Sometimes, the feature-orientation and control of the product owners causes development teams some peculiar behaviors. So this post is a start of my new thing to focus on: what could we do so that the relationship of the perspectives wouldn't be based on rules (who gets to decide what) but on trust and collaboration.

Approaching zero bugs

Zero bugs. Stop coding bugs. I can't help but smile a little whenever I hear this but often choose to step away from the argument. This was a topic, however, that Arlo Belshee addressed at Agile Alliance Technical Conference.

I don't really care much for zero. But I care for numbers becoming much smaller than what we're used to. And that is the perspective I listened into the discussion around the fancy marketing term.

The core idea I picked up was that when your code reads well, you make less mistakes. And that your ability to read code is more important than the executable tests keeping your code harnessed. Even more, it could be that the amount of code comments could be a negative indicator on the ability to code with less mistakes, as good code reads kind of as English.

I've been speaking about the fact that my team does continuous (daily) delivery without test automation around. It's nowadays so normal for me that I don't really put much effort anymore into explaining it to others. But when I still talked about it, I heard we're irresponsible. I heard it is not possible even if I've lived it for almost two years now. And it is possible with a team of 10 developers and one tester.

The zero bugs discussion gave me new tools to look at what we do and what might make us successful in what we do.

1. We focus on code readability.
   We're cleaning up and rewriting regularly. I encourage this behavior - even if it means I get to help test same things over and over again, as the amount of test automation is ridiculously narrow.
2. Developers do exploratory testing
   The developers test themselves quite extensively. This is actually the main reason I'm driving forward test automation in my team still, as I feel the manual testing must make us slower on this part. We're careful. But with care, we also succeed in introducing very little regression while delivering features with a steady pace.
3. Tester adds perspectives to exploratory testing
   When I test, I still find problems, so we're down to small numbers only after addressing the internal feedback. For the last year, the developer's testing skill has increased significantly, and a part of succeeding with this is my new discipline of avoiding writing bug reports (replacing them with discussions).
   
   There's a special approach on how I find my problems, and I still try to open it up better with my team. I read shape of code commits to know what I will cover. I will compare the shape I see to the shape I expect based on discussions. And shape of discussions is a form of both what is said a lot, a little and not at all. The shapes are models and I overlay a lot of different models to make up my mind of what to cover.

I see value in what I do to keep things great for production, but also love the fact that I'm increasingly less needed - I call that progress. But listening to the zero  bugs discussions, I get the idea that I might have been discounting the value of my other activity as a tester: making sure my developers are empowered and supported in their need of keeping the code readable, to their best current knowledge. I love how they pick up new things and ideas and hate their old stuff - that's how it is supposed to be.

The more we practice, the better we get. How often are we allowed to fix our past mistakes, made at times we knew the least? I would hope the answer is: every day.

Women in testing conferences

I've been around conferences a while. A little over a year ago in one of the conferences, I had a chat with one of the DEWTs (a peer conference in Netherlands) and expressed my interest in joining. I don't remember the exact response, but I remember my feeling: the remark on why I would be included was that they needed more women. Not more awesome people, but women. It turned into a laugh as these things usually do, but the feeling remained.

When I then got an email inviting me, I was happy and joined. I enjoyed the conference. I met some awesome people there, and the new people I added on my list of people to appreciate were mostly women. The men I already knew and the new men did not get through my limited bandwidth.

Late in the night, I remember hoping more people would talk testing. Instead, the discussions I ended up in were either about beer (I'm amazed how much there is to say about beers and I learned a lot even if I'm not particularly into the subject) or board games. My warmest memories are about talking on changing the world of testing with one of the DEWT fellows who I felt extended out of his usual circles to keep me company. Most of the time I felt alone in a crowd. And I was invited once only, for next ones I could see the same men there but know that I was excluded for lack of space. New women (and men) occupied that space, which is great. The spaces are limited.

I have social awkwardness around small talk (non-testing talk). In the last few years, I've found a perfect way for me to work through my social awkwardness. I go and talk to people about what they are into in testing, and find out if they would be interested in speaking in public. And a lot of people are, after I get to a point of sharing my honest excitement of something we've just discussed. I can do that because I have perspective of what is special and everyone has something special they could share.

Recently, I've been counting numbers. At AATC (Agile Alliance Technical Conference), I can easily remember meeting 10+ new women (including  the amazing Laurie Williams - highlight of my conference was to teach her about strong-style pairing she then mentioned in her talk on pair programming) and 3 new men. And quite an even distribution of old friends in both genders.

It seems this happens to me a lot. I go to a conference, and I come back with a list of names, primarily women. I'm biased, I know. It's easier to approach a lone stranger who is my gender. It's not active decision, it just happens. So when I organize conferences and need to name speakers, I have to work not to create 90 % female speakers rosters.

This experience makes me think that the same might happen the other way around. That the men naturally get to meet more men. And that when the bandwidth is limited, it would require a lot of effort to go talk to the women. And if the men tried this, who knows who would take that as an unwanted advance.

I also help women submit to conferences. I'm proud to say that there are 7 women on AgileTD list that are somehow connected to me. Some I encouraged to reuse their old talks (you'd be amazed it took me 14 years to learn I can and SHOULD reuse my talks instead of always writing a new one - a lot of women submit only new talks). Some I helped write their abstracts, listening to their stories of excitement they could not quite get on paper. Some I asked to share their stories, knowing they had one. Some I reminded on the fact that the conference pays for travel. I even suggested organizing a scholarship to compensate for time away from work (got declined, that was not an issue this time). Some I reminded that they can submit now and decide later if they will want to go, if they get accepted, knowing they are just afraid of the no and postponing your decision helps with that.

When the conference organizers say "We've tried everything" they mean "We haven't actively left women out and even reached out to a few". Let me show you a few things the recent people who did "everything" did not do:

• They did not invite me. I have no idea which other women that could have joined they excluded. Not only did they not invite me, they made me aware of the event only through "we had no women" messaging. Also, inviting once does not really count when others get to be regulars. 
• They did not ask me or Lisa Crispin for our extensive lists of amazing women around the world
• They did not ask Speak Easy for their contacts
• They did not spend 5 years in conferences meeting primarily women they could encourage, invite or promote

There's a few special aspects to women in testing conferences that are hard to express over twitter:

1. Speaking about lack of women makes this worse. It makes every woman a representative of their gender, over being a representative of personal awesomeness. 
2. Women group up so that they don't have to represent their gender. There is a reason why some IT work places have loads of women whereas others have none. It's safer in a group. 
3. Peer conferences with their debate culture (let's attack your ideas) is more often a turn-off for women. Why would I volunteer to be attacked just because that's a mechanism the men in the field found useful. I speak for dialog culture, the idea that we'd actively build a safe environment to talk about ideas over attacking them so that only strongest survive. The stories from past peer conferences where people felt attacked (!!) aren't helping. 
4. The "women and weekends can't be a problem, men have families too" feels ridiculous. The families where the work distribution (especially the meta work - organizing the work) are rare. I'm one of the lucky women who only feel extreme guilt leaving for weekends, I still get to go. Many don't get the chance. 

An average woman at conferences (or in IT in general) is better than an average man. They all feel they need to represent their gender. They get heavily filtered just from that. Helping the women get over all the obstacles they will face when they decide to speak in public is a support effort. More women helps. And allowing places for women isn't going to make your conferences worse.

An added note of someone who does well on this: James Lyndsay with LEWT. I was at LEWT#1. I've been invited to every single one since. I've only made it twice. But I feel I'm a LEWT. First come, first served. With enough women to never end up as a representative of my gender. 

Insights into training mob exploratory testing

I tried it again. Taking my style of mob exploratory testing to unknown, unchartered waters. I delivered a training day where we learn through testing of the client organization's own software. Software I've never seen before we start testing it.

I know I need more practice until the point where I no longer feel afraid of not being able to do it. Moving around with different software takes my quick learning ability out for a stretch, that often feels a little uncomfortable. But it brings such insights for my trainees that any of my discomfort is justified.

I asked the organization to  bring in a laptop with a test environment with their software on it, and select an area out of their documentation we could work on. The documentation is often a test case or a piece of requirements document. We start with identifying an area and exploring it. We extend to using the documentation only a little later, first impressions are built without constraint of a document.

There were a few insightful moments for me that I wanted to make note of:

Less than 15 minutes to first relevant bug

We were testing a feature I label here "meters". I quickly learned something that was obvious to everyone else that there's two kinds of meters, and that they are different enough to justify having their own create functionalities. Yet the form opening seemed very much the same, and I still seemed to have options of changing my mind not only between the two types we started with, but a dozen others. The concepts were drawing in front of me, as I could map those to physical entities taken to the software world. I did not agree with the design, but that was nothing to mention in the beginning since more info would be needed.

We created both kinds of meters, and making notes of functionalities we saw within those. As I asked the group to note a particular piece of functionality by giving it a name, the group also tried selecting the options. End result was serendipity: out of the two meter types, we could only be creating one as the other would have a remaining database error message. And it was not even illegal data, just a combination of something legal deeper than the first level.

Caring for code -discussion

Later in the day, a discussion took place that kind of surprised me. I shared how I test our daily releases looking at what developer says vs. what code / checkin says to assess risks and to build a per-change strategy. A developer in my training looked puzzled and voiced out his idea: "Is that what testers usually do?".

I explained that I look at the code to build a model of how well we understand the whole. Inconsistencies in what I hear and see lead me to insights on focus areas. Overemphasis of something does the same. I look for patterns that lead me to add things that are missing. And shape of code has proven very useful for me for that.

The puzzled look continues, and we end up asking the audience of 20 people - from management, testing, programming and support if anyone does anything of this sort. None came forward.

This left me wondering if it's something more special than I had given it credit for. Because still when doing that I don't "review code" or "read code". I look at size and shape rather than contents. And I find there often is an idea that the size and shape leads me to that makes the software speak to me to reveal more potential problems.

"This is not the testing we do here, but should"

My last note is on a comment I was given from behind the scenes by a friend that brought me in to do the training. As he checked on how we're doing, the comment was that I was teaching testing they don't really do, but should. I suspect they do, but don't talk about it enough.

Talking about how we do intelligent manual exploratory testing happens way too rarely. And sharing more about it with anyone who wants to hear (and some who don't) would make life as testers better.

My takeaway was that a lot of this stuff boils down to learning to talk about it. Communication. Helping people see what I do. And it left me with the idea that mobbing is indeed powerful. It allowed me to show testing in a way people think don't happen.