Thursday, July 19, 2018

Skipping Ahead a Few Steps

I work with an agile team. I should probably say post-agile, because we've been being agile long enough to have gone through the phases of hyper-excited to the forbidden-word and real-continuous-improvement-where-you-are-never-done. But I like to think of agile as a journey not a destination. And it is a journey we're on.

The place we're at on that journey includes no product owner and a new way of delivering with variable length time boxes, microreleases and no estimates. It includes doing things that are "impossible" fairly regularly, but also working really really hard to work smart.

Like so many people, I've come to live the agile life through Scrum. I remember well how it was described as the training wheels, and we have definitely graduated from using any of that into much more focus in the engineering practices within a very simple idea of delivering a flow of value. I know the whole team side of how we work and how the organization around us is trying to support us, but in particular I know how we test.

This is an article I started writing in my mind a year ago, when I had several people submit to European Testing Conference experience reports on how to move into Agile Testing. I was inspired by the stories of surviving in Scrum, learning to work in same teams with programmers, but always with the feel of taking a step forward without understanding that there were so many more steps - and that there could be alternative steps that would take you further.

The Pushbike Metaphor

For any of you with a long memory or kids that job your memory, you might be able to go back and retrieve one about the joy of learning to ride a bike. Back in the days I was little, we used training wheels on the kids learning biking, just to make sure they didn't crash and fall as they were learning. Looking around the streets in the summer, I see kids with wobbly training wheels where they no longer really need them, but they are around for just in case, soon to be removed as the kids are driving without them.

Not so many years back, my own kids were at an age where learning biking was a thing. But instead of doing it like we always used to do with the training wheels, they started off with a fun thing called pushbike. It's basically a small size bicycle, without pedals. With a pushbike, you learn to balance. And it turns out balancing is kind of the core of learning to bike.

My kids never went through the training wheels. They were scaring the hell out of me on going faster than I would on their pushbikes, and naturally graduating to just ones with added pedals.

This is a metaphor Joshua Kerievsky uses to describe how in Agile we should no longer be going through Scrum (the training wheels) but figure out what is the pushbike of agile taking us to continuous delivery sooner. Joshua's stuff is packaged in a really nice format with Modern Agile. It just rarely talks of testing.

People often think threes only one way to learn things but there are other, newer, and safer ways.

The Pushbike of Agile Testing

What would the faster way of learning to be awesome at Agile Testing then look like, in comparison to the current popular ways?

A popular approach to regression testing when moving to agile is heavy prioritization (doing less) and intensive focus on automation.

A faster way to regression testing is to stop doing it completely and focus on being able to fix problems in production within minutes of noticing them, as well as ways of noticing them through means of monitoring.

A popular approach to collaborating as testers is to move work "left" and have testers actively speak while we are in story discovery workshops, culminating into common agreement of what examples to automate.

A faster way to collaborating is mobbing, doing everything together. Believing that like in an orchestral piece, every instrument have their place in making the overall piece perfect. And finding a problem or encompassing testers perspectives in everyone's learnings is a part of that.

A popular approach to reporting testing is to find a more lightweight way, and center it around stories / automation completed.

A faster approach to reporting testing is to not report testing, but to deliver to production in a way that includes testing.

A popular approach to reporting bugs is to tag story bugs to story, and other bugs (found late on closed stories) separately to a backlog.

A faster approach is to never report a bug by an internal tester, but always pair to fix the problems as soon as they are found. Fix and forget includes having appropriate test automation in place.





Diversity of thought requires background differences

I care about diversity and inclusion. When I say that, I mean that I want to see tech and testing in particular to reflect the general population in its variety and conferences lead the forefront of the change in that equal opportunity. Looking from within my bubble, testing is already the area where there are a lot of women and seeing testing conferences that struggle with finding one woman worth giving stage (as some of them phrase it) is just lazy.

The diversity and inclusion means more than (white) women like me. I want to learn from people who are not like me or the men I've been strongly guided to learn most from through other people's choices. When people are different, their backgrounds and experiences are different, what is easy and hard is different and they end up learning new insightful ways of teaching from the platform they have created for themselves.

Some people seem to like saying they want to see diversity of thought in conferences as a way of emphasizing that they don't want to care of diversity and inclusion in the way I look at it, recognizing that the platform you are teaching from, the person you've become through your experiences is an essential part of being able to really have diverse perspectives.

Diversity of thought could mean:

  • I want conference talks where people tell me that the foundation of my beliefs is off even if it isn't so that I think about it.
  • I want talks of topics I have not yet heard, or really insightful ways of delivering a message I feel is important enough so that I could learn from how that message gets passed on
  • I want people who I feel can add something to what I know (even if usually it happens 1:1 discussing) 
I find real diversity and having a representative crowd of speakers worth focusing in conference design. Given any deeply technical or insightfully hard human topic where you can name a man, I can name a woman or a non-binary speaker who can deliver the talk with experiences the men can never speak of. I have a lot of work on recognizing the people of color due to my limited exposure, but I'm working on it. And I still work on being able to for any given international expert name a local expert.  

Representation matters. You need to see someone you identify with to believe you can go there. To get the general population of talent into the software development world, it is not enough to share the white men talking heads, but actively show a representation of what the world needs to look like.

It shouldn't be hard to have top-notch speakers for 10 slots in a conference when selecting from a pool of millions of candidates. Yes, there's a lot of people out there who feel they need to work twice as hard to get half the results. How about conference organizers working twice as hard identifying them and supporting places (like SpeakEasy for testing) that support those people starting off on a fast track to awesome speaking.

Finally, back to diversity of thought I wanted to add that I find that is a catchphrase not founded on reality. In the last three years of conference organizing, I have spoken through 200 proposals a year, totaling now at 500 since I'm half way through this year. There is no diversity of thought as such that I can see. There's diversity of topics and experiences, a lot of insistence of using the right words but overall we mostly share a vision of what good testing looks like from perspectives of testers and developers. Every one of those stories is worth a stage. But some of those stories are refused a stage because they require work before they are ready, others because there's someone who can deliver similar story with a different background, and others just because there is not enough space period.

In the same timeframe, I've spoken in tens of conferences a year and used the conferences in hearing other people talk. So I can probably add 15 a year, with 10 talks each - 450 more samples.

In addition, I volunteer for various other conferences that do traditional call for proposals as a reviewer. That adds more.

The datapoint that I have are the people who submit. I'm personally a datapoint that does not submit. I have given stage to many many people who did not submit - some that never did a talk before. I found them amongst participants. That's where the real stories I need to get out there are. 

Tuesday, July 17, 2018

Pay To Speak - why care and other collected thoughts

I hold a strong belief that the world of conferences needs to change so that it is not #PayToSpeak. This means that while you are not necessarily making money for speaking (that should happen too), you are not paying money out of pocket for speaking.

Recently I spoke at a big conference that has no call for proposals but invites all speakers and says to pay their travel, including the following weekend's hotel nights to enjoy the beautiful city the conference is at. They have a policy of checking in with them on bookings or them doing bookings for you. So when they did my bookings for flights, they had me leave home 3:30 AM and return home 2:00 AM. That meant no public transport available to the airport. I could drive and park there, or I could take taxi. I chose the latter as cheaper option. When I arrived after 8 hours of being stuck on travel for something that would be a 2 hour direct flight, I had a taxi pick me up. However as I needed to get out immediately after my scheduled talk to still almost miss my flight, I had to find (and pay) the taxi myself. The flight tickets did not include any luggage, so I ended up paying 80 euros just to bring my stuff with me. Packing so that I can take it all inside means compromises on cosmetics I would rather not do while having to feel presentable. That's one of the women's problems, I guess.

The extra costs totaled to 180 dollars, which was more than the cheap flight they found me. Their view was that they wouldn't pay and that was yet another nail in the coffin killing my speaking willingness. Now it looks like they might pay, but I believe when the money is on my account.

So being against #PayToSpeak means that I believe that while it is reasonable to ask speakers to be considerate of costs (no business travel), it is not reasonable to optimize in ways where you just transfer the costs to them.

To be clear, many conferences are #PayToSpeak. Most in fact in the field of testing. A little less in the field of programming. A little less in the field of testing now that we are getting to a point of being respectable industry (including automation).

Why should the conferences care?

We've seen examples like Selenium Conference moving away from #PayToSpeak. The results they report are amazing.

  • 238 % increase in number of submissions - there are a lot of people with great content that cannot afford to teach in conferences that are pay to speak
  • new subjects, new speakers, new nationalities and new perspectives - all valuable for us learning in the field
  • percentage of women submitting growing from 10% to 40% - showing that the pay to speak might disproportionately impact underrepresented groups ability to teach in conferences

Surely you don't mean that we should pay newbie speakers?

I find that #PayToSpeak conferences present their audiences three groups of speakers:

  • Those so privileged financially that paying isn't an issue
  • Those with something to sell so that their companies pay them to deliver a sales pitch in disguise. Some better disguised than others. 
  • Those dreaming of public speaking experience finding no other options but paying their woes, sometimes delivering great talks. Believing paying is temporary. 
I believe that people in first category can opt out of payments in a conference that pays the expenses. In my experience they rarely do opt out on the out of pocket money, but many have opted out on profit sharing to leave money behind to support new speakers. People in the second category might become sponsors and pay more than just expenses to attend, and have their sessions branded as "industry practice / tool talks" which is often a way of saying it's selling a service or a tool. 

The third category is what makes me sad. 
This should not be the case. We should pay these speakers expenses. Our audiences deserve to learn from them.

As conference organizer, the thing I'm selling is good lessons from a (insert your favorite positive adjective) group of speakers. There are other ways of managing the risk of bad speakers than making sure your audience only gets to listen to the financially-well-off-and-privileged segment.

You could ensure the speakers with less of reputation get support and mentoring. With things like Speak Easy and folks like myself and Mark Dalgarno always popping up to volunteer in twitter, this is a real opportunity for conferences as well as individuals.

For Profit and Not For Profit

Finally, these words don't mean what you think they mean around conferences. You can have a not for profit conference with really expensive tickets and they can pay the speakers (this is the vision I build European Testing Conference towards - making real money to pay real money to speakers within a not for profit organization using profits to pay other conferences speaker's travel). You can have a for profit conference with ridiculously small cost and they still pay the speakers (Code Europe was one like this - they made their money out of selling participants to sponsors in various ways in my perspective).

Speaker's choices of where they show up with great material matters. But most of all, participants using money on the conferences matter most. Pay for the good players. Be a part of a better world of conferences.



Sunday, July 15, 2018

Testing does not improve quality - but a tester often does!

Being a self-proclaimed authority in exploratory testing, I find it fun when I feel the need of appealing to another authority. But the out of the blue comment the awesome Kelsey Hightower made today succinctly puts together something I feel I'm still struggling to say:  Testers do their work to save time for stakeholders next in chain. 
Actually, nothing in this tweet says that you need a *tester* to do this. It just refers to highly intellectual and time consuming activity, which to me implies that doing something like that might take a bit of time to focus.

With the European Testing Collaboration Calls, I've again been privileged to chat with people who trigger my focus to important bits. Yesterday it was someone stating their observation very much in sync with what Kelsey here is saying: for many of the organizations we look at that go for full hybrid roles, it turns out that the *minority perspective* of the exploratory testers tends to lose in the battle, and everyone just turns into programmers not even realizing why they have problems while in production on scale beyond "this is what we intended to build".

Today in prep for one of the Collaboration Calls, I got triggered with the sentence "testing does not improve quality". I sort of believe it doesn't, especially when it is overly focused on what we intended to build and verifying that. The bar is somewhere and it is not going up.

But as a tester, I've lived a career of raising the bar - through what I call testing. It might have started off like in one organization that 20 % of users were seeing big visible error messages, and that is where the bar was until I pointed out how to reproduce those issues so that the fixing could start. But I never stop where we are now, but look for the next stretch. When the basics are in place, we can start adding more, and optimizing. I have yet to find an organization where my tester work would have stalled, but that is a question of *attitude*. And that attitude goes well with being a tester that is valuable in their organization.

How do you raise the bar through your tester (or developer) role?

Feeling Pressured to An Opinion

It was one of those 40 collaboration calls I've been on to figure out what is people's contents that was special because of the way it ended up being set up. The discussion was as usual. There were a few of us having a discussion on an intriguing topic. The original topic was way too big for a single talk, and we collaborated on what would the pieces of focus look like that we would consider. 30 minute talks about everything between life and death don't do so well and don't end up selected, so we always seek something in a call that really has a fighting chance.

As I ended the call, a friend in the room expressed they had been listening in saying "You can't seriously consider you'd take *that* into the conference program".

I was taken back but was stupid enough to budge under pressure to confirm that I wasn't seriously thinking of it. Even though I actually am. I always am.

Because of my failure to - again - stick up to what I believe and let a verbal bully run over me, I felt like I wasn't true to my values. But I also learned that while in the moment I may be run over, I always come back to fix what I did wrong. So a few hours later, I expressed my annoyance of the style of communication, and recognizing the added bias of this negative interaction, I'm more carefully taking the advice of the two other excited co-organizers I had on that call.

Looking at the interaction a little more was funny in the light of a discussion a few hours later on how  awesome visual thinking strategies (a lesson by Lisa Crispin) are in identifying when you're doing an observation and when you're doing an inference.

Observations are the facts. The exact words the person would use in a call. What you can visually verify. What you can sense with your senses without adding judgment.

Inferences are not facts. They are your observations mixed with your experiences. They are your biases at play. And the reason we teach the difference in testing (and try to practice it), is to recognize the difference and seek fairness.

What the friend didn't see fit in the conference program was only unfit through their biases and experiences of when people are collaborative and when they are pushy. There's room for pushy people like them pressuring me to opinions I need to come back defending, so I'm sure there's room for all sorts of people.

Software is easy but people are hard. And combination of the two is fascinating.

Saturday, July 14, 2018

All of us have a test environment

This post is inspired by a text I saw fly by as I was reading stuff in the last hours: All of us have a test environment, but some of us are lucky enough to have a production environment too.

Test environments have been somewhat of a specialty of mine for the 25 years I've spent in testing, yet I rarely talk about them. So to celebrate that, I wanted to take a trip down memory lane.

Lesson One: Make it Clean

I started as a localization tester for Windows applications. After a few decades, I still remember the routines I was taught early on. As I was starting to test a new build (we got them twice a week back then), I would first need to clean my environment. It meant disk imaging software and resetting the whole of Windows operating system to a state close to factory settings. Back then it wasn't a problem that after doing something like this, you'd spend the next hours in receiving updates. We just routinely took ourselves to what we considered a clean state.

As you'd find a problem, the first thing people always would ask you was if it was tested on a clean machine. I can still remember how I felt with those questions, and the need of making sure I would never fail that check.

Lesson Two: You Don't Have to Make it Clean

Eventually, I changed jobs and obviously took with the a lot of the unspoken attitudes and ideas my first job had trained me in. I believed I knew what a test case looked like (numbered steps, y'all!) to an extent that I taught university students what I knew ruining some of them for a while.

I worked on another Windows application, in a company with less of a routine on cleanliness of the test environments, and learned a lot about the fact that when environments are realistic as opposed to clean, there's a whole category of problems of relevance that we are finding. It might have made sense to leave those out as long as we were focusing on localization testing, but it definitely did not make any sense now that I was doing functional testing.

I realized that we were not just testing our code, but our code in an environment. And that environment has a gazillion variables I could play with. Clean meant less variables in play and was useful for a purpose. But it definitely was not all there was.

Lesson Three: The Environment is Not My Machine but It's Also My Machine

Time moved forward, and application types I was testing became more varied. I ended up working with something I'd label as client - server and later on web. The Client - Server application environments no longer were as much under my personal control as the Windows applications I started off with. There was my machine with the client on it, but a huge dependency to a Server somewhere, often out of my reach. What version was where mattered. What I configured the Client to talk to mattered. And I learned of the concept of different test environments that would be under fairly regular new deliveries.

We had integration test environment, meaning the Server environment where we'd deliver new versions fairly often and that was usually a mess. We had system test environment, where we'd deliver selected versions as they were deemed good enough from whatever was done with the Integration test environment. And we had an environment that was copy of Production, as most realistic but also not a place where we could bring in versions.

For most people, these different environments were a list of addresses handed to them, but that was never my approach. I often ended up introducing new environments, rationalizing existing ones with rules, and knowing exactly what purpose each of them could give me with regards to how it impacted the flow of my testing.

Lesson Four: Sometimes They Cost a Million 

Getting a new environment wasn't always straightforward, it was usually a few months of making a business case for it and then shopping some rack servers we could hide in our server lab. I remember standing in front of one of these racks, listening to the humming both from it and the air conditioning needed to run that much hardware and being fascinated. Even if it took a few months arguing and a few more months delivering, it was still something that could be done.

But then I started working with Mainframes and a cost of a new environment went from some thousands to a million. It took me two years to get in a new environment while in this environment.

Being aware of the cost (not just hardware but the work to configure), I learned the environments we were working on in even more detail. I would know what data (which day's production copy scrambled) would reside where. I would know which projects would do what testing that would cause the data to change. In a long chain of backend environments, I knew which environments belonged together.

In particular, I knew how the environments were different from the production environment to the extent that I still think of a proud moment in my career when we were taking a multi-million project into production as big bang, and I had scheduled a test to happen in production as the first thing to do, one that we couldn't do elsewhere as the same kind of duplication and network topology wasn't available. And the test succeeded, meaning the application failed. It was one of those big problems and my proudness was centered around the fact that we managed to pinpoint and fix it within the 4 hour maintenance windows because we were prepared for it.

Lesson Five: It Shouldn't be Such a Specialty

Knowing the environments the way I did, I ended up being a go to person for people to check of which of the addresses to use. I felt frustrated that other people - in same kinds of positions that I was holding - did not seem to care enough to figure it out themselves. I was being more successful than others in my testing for knowing exactly what I was testing, and what pieces it consisted of. My tests were more relevant. I had less of "oops, wrong environment, wasn't supposed to work there".

So think about your environments and your attitude towards the environments? Are they in your control or are you in their control?

Tuesday, July 10, 2018

What is A/B Testing About?

Imagine you're building a feature. Your UX folks are doing what they often do, drawing sketches and having them tested on real users. You've locked down three variations: a red button, a rainbow button and a blue button. The UX tests show everyone says they want the red button. They tell you it attracts then, and red generally is most people's favorite color. You ask more people and the message is affirmative: red it is.

If you lived in a company that relies heavily on A/B tests, you would create three variations and make releases available with each variation. A percentage of your users would get the red button, and similarly for the two other colors. You'd have a *reason* why the button is there is the first place. Maybe it is supposed to engage the users to click and a click is ordering. Maybe it is supposed to engage users to click and a click is just showing you're still active within the system. Whatever the purpose, there is one. And with A/B tests, you'd see if your users are actually clicking, and if that clicking is actually driving forward the behaviors you were hoping for.

So with your UX tests, everyone says red, and with your A/B tests, you learn that while they say red, what they indeed do is blue. People say one thing, and do another. And when asked on why it is the way it is, they rationalize. A/B tests exist to an extent because people being asked is an unreliable source.

What fascinates me around A/B tests is the idea that as we are introducing variation, and combinations of variation, we are exploding the space that we have to test before delivering a product for a particular user to use. Sometimes I see people trusting that the features aren't intertwined and being ok with learning otherwise in production, thus messing the A/B tests when one of the variation combinations has significant functional bugs. But more often I see people not wanting to invest in variations unless variations are very simple like the example of color scheme of buttons.

A/B testing could give us so much more info on what of our "theories" of what matter to user really matter. But it needs to be preceded with A/B building of feature variations. I'm still on the fence with understanding how much effort and for what specific purposes organizations should be willing to invest to really hear what the users want.