Finding bugs that have been around a while

As part of my usual routines, there was yet another release. To keep myself alert and awake, I picked up yet another set of data over using the one safe one automation relies on.

The major change in the release was a new luminaire printout, so I had sort of an idea on testing through some luminaire-functionality still in the staging environment. I brought in a common set of data (lazy selection), and a big visible error message pops up.

Looking at the error message, I could see the application did not agree with my data. The data that was there already - specifically a string "100+" - was failing.

A discussion with the developer later, I stop and I'm feeling puzzled. I had managed to not run into that bug for months and now, just when we were about to release again (daily), it just came my way by a lucky chance (from the change of data).

Users with respect to this bug are like me. At some point they will end up trying to do just this in one of the projects (sets of data) that just so happens to have this condition. They haven't yet, or we'd see that from a log and fix. But since I spend much more time on the software, actively varying, I get there sometimes sooner. And whenever I get there, unlike the users, I take the time to discuss about it with my developers.

Half an hour later the problem is gone. Like it never existed. But I still am stopping to wonder: how could I have found that bug earlier and I formulate a heuristic to take with me:

For string fields with imaginable integer comparisons, it's different to try 100a and 100+. Arithmetic is closer to the realm of numbers.

 

Shame in not becoming a speaker

This post is somewhat personal. It talks about my feelings of shame and ways I've recognized I used to deal with shame: self-delusion. It touches on my experience of being more responsible for home and kids, and how hard it can be to be away, always balancing. And it talks about a complaint I've never made even in private for conference organizers. If you're fine with this, read on. 

I need to tell you a bit of personal history on how I got started on speaking. I share this to help others make more informed decisions. And I offer a view that while I overcame all my obstacles so far, these obstacles and that they exist may be preventing the emergence of many other great, diverse speakers. Diverse as in women: having to balance the perception of financial responsibility for home vs. running the conference pipeline. And diverse as in people who are not consultants or from organizations selling to testers or recruiting testers. 

In hindsight, how I became an international speaker

In 2001 I was a researcher at university - a job I took as a safe environment to learn about testing in the world and to learn to overcome my speaking phobia. Poor students had a shaking nervous teacher, with a lot of commitment on sharing stuff on something she was super excited about: software testing. I was a bad speaker, but I started practicing - a lot. I used every chance I could find to deliver talks. I found my style of what I was comfortable with. And teaching turned into learning, a habit. 

In hindsight I recognize that 2005 was a major milestone in what set my future in speaking. I quit a consulting job that had, just like the research work earlier, paid for me to speak. I had more practice  of speaking under my belt, and back then thought I knew more of testing than I know I know now. With a long list of companies I couldn't work for as they were customers of mine in the consultancy, I landed a job on one of those companies. The contract was a 6-month period, just to match the time my hands were tied. It turned into 3 joyful years. But since it was short and my intention was becoming independent, my contract was created so that I could start my business on the side. I've been training through my side business for 10 years now. It was a major milestone because the side business would later turn out to be the main vehicle to enable me to speak at conferences. 

Let's talk about money 

Local conferences are easy to go to: just go, speak. They rarely pay even travel, but then again, travel within your own city is a non-issue. International conferences are a different deal.

My first international conferences was EuroSTAR. I prepared for my talk for weeks. I got a free conference pass to a super-expensive conference, and was ecstatic. My organization back then paid my flights and hotel, telling me it would be just this one I could do for quite some time. I never realized it was wrong that I had to pay to speak - there was nothing in it for my company, other than making me happy. 

Doing this a few times and paying with my own money (from my side business) as not all companies will pay for me, I started realizing there were better conferences, ones that paid for travel + stay. They were rare though. I started prioritizing showing up in these conferences with my best possible content. 

The year 2015 was really a turning point in how I would view this. I did a total of 33 talks, and traveled to a lot of places on two continents. I was working with a company that was generous to allow me to use 5 days of invoiced time into trainings per year. That is 37.5 hours in which I would not lose money to be out of office. The personal investment grew, and awareness with the experience. So if I would be out of office for conference time, it would cut away my base salary. But at least I had the side business and a deal from my lessons in 2005 that allowed me to continue to do trainings / conferences on the side. Without it, there's no way I could afford this!

None of the conferences I spoke at paid for my time on being away or preparing for the track sessions. Some of the conferences paid travel + hotel. Some paid just hotel. Some paid just part of travel + hotel. But many had me pay my own travel too. One of the conferences organized an on-the-side paid training, and I can't say enough positive words about Rosie Sherry and TestBashes on enabling things financially. 

So it's expensive. How much did you want to invest in this if your company does not do it for you? This is not just becoming a speaker, but becoming a great speaker. It means practice. Reflection. Trial and error. 

And the shame comes in... 

Conferences that pay for travel and expenses have all operated on the idea that you get your money back after the conference. It means that when you speak, you become a personal loan office for the conference. Your money can be tied in typically from 3 - 9 months. And when you have many of these, the amounts accrue. Multiply that by the number of conferences, and you'll figure out that there must a bit of financial flexibility.

When I realized the money I need, I was lucky to build the flexibility through delivering a paid testing training. I understand that is not an option for everyone. Introspectively, I see I could have easily stayed at home to avoid mom-guilt and just tell myself that I did not really want to do it anyway. It's not a part of my job. My job does not benefit from it, or need it, directly at least. Time is limited and I don't want the other kind of job, perhaps I should take what comes with this (I LOVE product development!). When you miss out on an opportunity, self-delusion helps keep you sane. I recognize I have a lot of that. 

I've only recently started voicing out my concern on this and the reason for not doing this is shame. It's hugely embarrassing to have to tell an unknown conference organizer that I'm out of cash because of this approach. Even worse, to tell them that since I have not had gigs on my side job, the money I'm tying into this is what I'm supposed to run my family with. 

I feel shame because I should save up and have the money available. But I haven't.  Just before Christmas, I was waiting on several conferences to come back with payments of thousands of euros for conferences some months before the time. I had gone so low on my flexibility that all I could do is to tell myself that it's better that kids get their presents after Christmas on the sales, and deliver just great promises as presents.

Sometimes I wonder who would want to take that extra load? Why do I take that extra load? And on those occasions, I need recognition that what I do is worth something to someone - that I've helped.  I need someone to remind me that I speak to change the world. World of testing. World of conferences. Together with my communities. 

Money extends to organizing homelife

If you have kids, taking them with you or leaving them behind is a lot of organizing - money makes that easier. Do I really always need to choose between family at home and a conference away, or could conferences make it financially easier to deal with this? 

I rarely talk about my kids in the professional circles. Even less I mention for years I was running this all in a single-parent setting. There's a lot to orchestrate. Kids make a great excuse, should I want one. I could just do less. 

And the diverse speakers then? 

The world of conferences needs to change. In addition to support we're giving through great mentoring initiatives like Speak Easy, the conferences need to start paying for the people who deliver the talks. I did not mention this much before. Perhaps others too, like me, are ashamed of having to reveal that the finances stop them, to a point where they resolve in self-delusion telling they don't want it, they can't do it and no one cares what they could share anyway. 

I want diversity. I want to enable women to travel. I want to enable people who work in companies without a will to pay for them to travel. I want to hear more from people like me (in product development) and less from those who have become independent consults. 

Some professionals travel easier because keeping themselves visible keeps their companies visible. A majority of people - with great stories and experiences - are not in that position. Thus I believe the world of conferences must change and we need to: 

1. Pay the speakers, not just travel
2. Pay costs as soon as they incur

These ideas and experiences are just one of the reasons I'm organizing European Testing Conference as a platform where I can change the world of conferences one conference at a time. 

Know your environment features

There's one thing in particular that I find always a little challenging to explain to non-testers, and it is that of a dependency to things we did not implement in our projects. This is part of what I think of testing of the system, and groups often under "environment". For various reasons, the same software tends to work differently when placed in a different environment.

The environment here is many different things. It's the network, it's the operating system, it's the other programs our program could interact with. And with what I test at work right now, the most obvious bit of the environment is the browser the software runs on and versions of excel it uses to generate reports. 

Testing on a browser is not just about covering different browsers and versions. It's also tweaking any relevant browser features, security settings and zooming being the most prominent ones to my experience. The browser (or any bit of the client environment for that matter) tends to be out of control for us, so we can only see that whatever it turns out to be, we know the implications to how our app works in that environment. 

I've explained this in the theme of "why does it take so long to test" so many times, that I realize I was getting comfortable in thinking that I know which pieces to look at. Thus being fooled by one feature of the operating system leaves me re-considering what other features I might be dismissing. 

Last week, I was testing a report and paging, and getting buggy reports out. The pages would look awful to me. When I discussed this with my team, a comparison started: it looked just fine for everyone other than me. It turned out that in Win7 settings there's a feature called Display under Appearance and Personalization that allows me to change the size of the items on my screen. I defaulted to 125 %, whereas all developers and the project manager defaulted to 100 %. Bigger screens, less on the road I guess. I was not aware this thing could change our reports completely, but in hindsight I can see it does. 

 

The discussion in fixing was interesting. This is a "feature we're not supporting". I really don't care what we call it, but a few days later, it was addressed for end users. Because we knew about it now. 

Yet another reminder on how much of the technology around the application the tester should know to test in a meaningful, relevant way.

Users reporting bugs

I've had the pleasure of seeing how a friend deals with his open source project ApprovalTests. There's a lot for a tester to learn on the attitude you have towards free labour on your project: encourage it!

Even in closed source project, we have people who are not paid but will contribute to our project. I tend to think of the main group of them as users.

A recent tale of a user

He was playing a popular game, MineCraft, and run into a bug. He isolated the bug and was sure it was one, and decided to go over the lowest level of commitment someone unpaid can have on a project: reporting the problem. The bug wasn't critical for him, but it was not insignificant either. So he decided to contribute some of his time, to help a little.

He found the MineCraft Jira and run into quite a quandary. There were many possible projects the bug could belong to, and instructions disencouraged duplicates. Finding out if yours was in there was already quite a puzzle, but with the work done, he ended up realizing it most likely was not. Thinking he found the right Jira project, he logged an issue with all necessary details of steps to reproduce this.

In two hours, he got a response. Sounds great, right? To his disappointment though, the response was to point him to another project as he apparently had chosen the wrong project to report to.

Being an outspoken fellow, he responded with a kind email suggesting that the company / employee of the company might want to, in future, to consider the answers they give to their end users doing free work for them. Not to say "do more", but to show gratitude on work already done and drive things forward as paid people. That way the user might volunteer time again next time they run into something relevant. The response back was unappreciative, outlining that the employee already did the right thing guiding him to the next project.

As for how the tale ends, the bug got lost. The user never continued to the next project. And most likely won't bother again on spending time.

The contrast to ApprovalTests approach

I've been stunned to learn about how a friend deals with similar cases in his ApprovalTests project. When contacted on a bug, he asks the user for a pairing session. If it's nothing, he gets to briefly meet someone new somewhere. And if it is something, he pairs up with the user, fixing the problem during the session.

Unsurprisingly, he has made friends for life with this approach. He has users who seem to be coming back to report more.  His approach surprises people as it is so rare. But perhaps it should not be.

To get out software to improve, we really need to harness the opportunity and encourage free people to do more work for you. Paid people are in the position to treat the unpaid in a way that encourages their contributions.

I recognize that as professional tester, I hardly ever anymore volunteer reporting bugs without someone paying me for it. I've learned to report bugs as user with "reclamation" added to them, to get proper reaction and compensation for my time in cases of serious issues (2 times tried, 2 times got a discount of the service price I was paying).

Couldn't we treat the users as if we cared since we do? I care enough to take tasks from my internal users and to report back to them. Pairing for fixing with them would be awesome. Perhaps even something to aspire for next.

The risk of things not working

Yesterday morning, a developer suprised me: "We just did a major refactoring, removing the model from our code and moving all responsibilities to core. I was thinking of checking this into the integration." After a few clarifying details the change could be summed up as we changed everything.

As usual, when my teams devs change things, they test themselves. They have little of unit tests they could just run, so they explore the application. But with changes like this, it's hard. There's no specific place to check - it could be anything. It means the developers test less.

The change comes my direction, and I test a basic positive scenario of sampled features from my checklist. I find nothing on this change but spend a few hours on this. From the idea (Monday evening) to production (Wed morning), things go pretty smootly. A day in production, no problems.

When I decide to be done with this, I'm painfully aware of how little I tested for a change of everything. I could easily spend weeks on testing it. But I assess the risk, knowing from my testing that the basic flows work and we publish it. There's a significant risk of things not working, but I remind myself: not tested does not mean not working.

The experience leads me to think again about many of the discussions we have on automation. How so many times people assume that anyone (or me) would actually run the tests that automation could run, if it was implemented. I don't: not testing does not mean it does not work. How people don't see that an option to investing in automation that helps repeat some tests is that you work in a context that allows you to publish with the risk of things not working. Risk is not a fact, it's a chance. Playing on the risk is a gamble.

We've done many things to make the gamble worthwhile from production monitoring to abilities to fix things quickly to deeper understanding of our business and users. Testing is just one piece in the puzzle. But as a tester caring for the system and user experience, it's a piece that glues perspectives together.

The end of my crowdsourcing experience

I tried testing on a crowdsourced project with Testlio from a direct request. I did not care to ask the pay in advance, I just thought it would be a fun experience.

The testing part of it was a fun experience. If you love testing even a little, you might recognize the rush of getting to know a new application and finding problems there. You might also know the frustration when you need to find simple and basic bugs in functionality, knowing that those anyone could spot: including the developer who created the app in the first place. New project is always a rush of emotions, and new projects tend to end up with positive. This one was no different.

While testing, there was already more of frustrations. The idea of shallow testing due to a short timeframe, in which you won't have time to investigate or dig deeper. The idea of paying only for sessions, when everyone doing exploratory testing knows that only some of your working time is in sessions. The idea of crowdsourcing downplaying what skilled testers can accomplish, for people who can't distinguish "testing" from "testing" and buy just any testing.

I did two projects on the same application: testing on web for 2 hours 10 minutes, and testing on iOS devices for 3 hours 30 minutes. I logged 12 issues on web and 19 issues on iOS, and seems that I'm doing fine with accuracy/relevance. That's 11 minutes to a bug. And reporting properly takes time of course, since I would check reproducability, just like any good tester would.

I got paid for the iOS project today: 20 dollars for an hour. The Testlio site says they pay up to 35 euros an hour, but even with a lot of experience elsewhere, clearly my first project was not valued that high. I don't know what the criteria of deciding on payment is, but with the payment I end up with "thanks for the experience, but there's better things for me to do with my time". At first I thought the pay was 10 euros an hour, but since the payout was only for the iOS project, so 18 euros (20 dollars).

As financial comparison, let's do a bit of a calculation. Paying only for the effective testing hours means that the usual in-between-tasks time is included in the price. If you look at a full day of focused exploratory testing in session-based style, let's say that for 2 hours of testing, you would have 1 hour of off-charter time. You're still working, but not really on the charter. Digesting. Filling in gaps. Letting your brain bring things together. In this testing, for me it meant that I went back, unpaid, to close a bug I realized was duplicate. I investigated one bug further because I wasn't happy with what I new of it. I discussed with the project leads. Read instructions. Reviewed what other people had reported. And went back to check my feedback and issue resolutions.

That 18 euros for an hour is now only 12 euros for an actual hour. I would hardly volunteer to take a job that pays me 12 euros an hour, including "holiday pay". That's the minimal pay in Finland, not the pay for a test professional with 20 years of experience.

Feedback-wise, my dislike of templates got me a lower score eventually. I get it: as a test lead on a project where you can expect low-quality investigation for bug reports, you believe less of people. If I don't mention my user account or my network for a typo that is available also without logging in, I must not have checked. Or I must not have the skills to know what types of bugs might depend on what types of things that I will report whenever they are relevant. And I definitely wouldn't actively learn from my mistakes, since I'm only paid for the hours when the bugs are first logged. Except I did, since I'm a professional tester.

12 euros an hour might be much more lucrative for someone who does not live in Finland. Crowdsourcing customers who want testers specifically in the market / geographical area they target will have less choice around here. 

With the pay settled and me valued to a lot less than I make in my day job, I choose to vote that my free time is more valuable. I rather change the world by enjoying new applications in an open source project that values my contribution.

Crowdsourcing and what to think of it

About three years ago as I had just joined my current projects, we needed more testing to happen than I could do. The developers test, they had tested before but with the level of skills on testing (and interest) the end result was that the main outcome of their testing back then was increased coffee consumption in avoidance of doing the work.

Back then, I identified two options we had:

1. We could use Crowdsourcing - pay a fee for service that allows us to use random testers to use the product to find problems.
2. We could use Contracting - pay a fee for having one tester dedicated to grow with us through testing for us. 

Comparing back then the prices of uTest and the Romanian contractor we considered, the differences on the monthly fee were not relevant. So the question really went down to contents. We decided at that point to be better off with contracting, and from that decision, coaching to have another deep tester in the world commenced.

I come back to thinking of this, as last week I decided to try testing with Testlio. As far as I can tell, they are another option to uTest back then (Applause nowadays) on crowdsourcing, with the specific aspect of not paying for results but for assigned hours.

I'm just about to start my third round of "let's see what this is about" with Testlio. At this point I don't know what the payment by hour is - from their pages I can tell that they pay less than my day job even in the highest category, but I guess that comes with seniority. But from Testlio perspective, I'm a newbie, so I doubt they will place me in the higher rates without specific Testlio-experience - one that I'm unlikely to build up.

From a professional tester's viewpoint, here's what happens with Testlio:

• They invite me to projects and assign me work packages 
• Timeframes for work package availability are short and I can say if I volunteer 2 hours of my time today or not before starting
• As I start a work package, I start a timer. You're paid on the time. Forgetting to start a timer (newbie mistakes) can be corrected by discussing with the test leads. 
• I get a high level checklist (find this feature and mark pass/fail) or a charter (spend an hour doing X).
• I test what I can on the time given, and report. Reporting includes adding notes on a very high level plan and bug reports. 

Since I volunteer for this as tester, I probably try to do my best in the time allocated. Even as user, I am likely to do a decent job if nothing special is expected of me. Buggy software (like work package 2) makes skilled tester like me even less necessary. For a tester like me, it ended up being hard to stop, so free work gets done. I even dream of things I could have still checked.

Testlio approach is better with the hourly payment than the other options back in the days I looked at them for a tester perspective. If you get paid only when you find valid bugs first, it adds another level of shallowness and leaves out things that might get reported in the hourly model.

I like looking at how I feel while doing this.

• Refreshed. Testing something different is refreshing. 
• Unchallenged. Short timeframes invite shallowness. Even a few consecutive work packages on the same test target with different versions still leaves the feeling of shallowness. Within 2-3 hours, you won't investigate anything complicated.
• Unresponsible. My job is done with the clock. I do the best I can within that timeframe. I have no connection with the product. I have no social relations to others - at least not paid social relations.

I currently have three main concerns:

1. The pay & the progress. I expect that as soon as I learn how little I was paid for these, I rather contribute time on open source projects. 
2. The bureaucracy. I already got a request to follow the issue template. I hated the issue template and felt constrained. It had repetitive information to be filled in. It had unnecessary information for most bugs I logged. Seriously, even copypasting my service provider info when there's simple UI bugs that have NOTHING to do with the network makes me feel like a rubber stamp. I would choose to use my time better while I understand that requiring same info comes from lessons of missing that info on some bugs sometimes. 
3. The shallowness in the model. It's great that some companies use this to find bugs. But I find the idea that they use this as replacement of inhouse testers, thinking they are getting the same thing inherently wrong. The timeframes here are too short for proper testing, the testing in 2 hours remains shallow.  I feel the whole crowdsourcing model drives intelligent deep testing down by implying that a short time is enough. It's not about just having skill in testing, but learning to dig deeper takes time. 

The two first concerns will soon drive me to prioritize other things to do with my time. The third concern is bigger: I don't need to be a part of this change but the change has been here for a while already. What should be my stance on it?

Apprenticeship vs. Mobbing

Last week, I had a weird experience at a medical clinic in USA. For a full week, I hang out to see various kinds of medical tests performed to isolate symptoms into causes, with a bunch of medical specialties involved in a process. Very unlike what I'd expect, these different specialties collaborated during the week, having discussions and reading a lot of the texts that the other specialties left behind in the systems.

Towards the end of the week, the experience got more weird as the patient I was accompanying decided to invite the general doctor to a mob programming conference in May 2016. What if the doctors would, instead of co-creating a diagnosis, actually collaborate more deeply and even address the patient all at once? Would that change things?

The doctor's response was interesting and reminded me of many of the responses mob programming gets in the software development world. First of all, the doctor pointed out that the medical industry used to do this in the past. They used to have rounds, where all doctors would together meet the patient. The rounds that I've experienced personally from a patient perspective were always within specialty as a means of transferring knowledge, like apprenticeship. Let the juniors see the seniors do the work to mimic behaviors. And let the juniors try doing the work so that seniors are there to correct. The doctor here pointed out that they also have used to have cross-specialty rounds but that was considered inefficient leading to the current ways of working.

During the week, most of the specialty doctors (we saw 6 different specialties) asked exactly the same questions when working on the same problem. We would have loved to give that information once. We would have loved seeing the doctors comments influencing the others perception. And especially on details where they did not agree in the end, it would have been lovely to see how the argument would have developed to the "specialty doctor will have a final say on this" we saw in the end with co-creating the diagnosis.

All of this left me thinking that apprenticeship seems relevantly different than mobbing, even if apprenticeship happens in a group setting. What the doctors used to have was apprenticeship. It was based on a clear senior-junior hierarchy with switch on task -rotation of roles. There would be traditional pairing over strong-style pairing: they would review and monitor what happened over making it happen together.

Also, specialties can easily get in the way. The cool biofeedback techniques the psychologists teaches for managing fears could easily be taught by other specialties. The summaries of overall status could be done by a specialist. It seems that inefficient might mean optimizing for people's time, not the patient throughput to a diagnosis and treatment. It was a great reminder again on the potential harm siloed specialist roles can have, if we are not willing to be flexible with our role boundaries.

All groups programming are not doing mob programming. And while finding the common ground with different individual in groups takes more work for the team to form, a mixed group will produce different results.

Renaming in retrospect

In the testing world, a blog post reclaiming the word "testing" to its original use is fairly popular. I keep hearing about it in conferences, when people feel I shouldn't talk about exploratory testing when all testing is exploratory.

Here's the thing however: when people around me talk about testing, they hardly ever mean the thing I do. I do exploratory testing. They do testing that focuses mainly on automation-related strategies.

I feel strongly on not going into the 3.0 bandwagon reclaiming the word testing, but going forward with the word exploratory testing.

Back in the days when people started creating musical instruments, someone came up with a guitar. In these days we would normally call that guitar acoustic guitar, to distinguish from a later invention of an electrical guitar.

Things are named in retrospect. And exploratory testing to me seems to be one of those things that, in contexts I work in at least, benefits from the renaming in explaining why what I do as an exploratory tester is so different from the testing that I too feel should vanish completely: the manual detailed test cases run by human drones without active thinking involved, or commodity testing.