Wednesday, January 20, 2016

The end of my crowdsourcing experience

I tried testing on a crowdsourced project with Testlio from a direct request. I did not care to ask the pay in advance, I just thought it would be a fun experience.

The testing part of it was a fun experience. If you love testing even a little, you might recognize the rush of getting to know a new application and finding problems there. You might also know the frustration when you need to find simple and basic bugs in functionality, knowing that those anyone could spot: including the developer who created the app in the first place. New project is always a rush of emotions, and new projects tend to end up with positive. This one was no different.

While testing, there was already more of frustrations. The idea of shallow testing due to a short timeframe, in which you won't have time to investigate or dig deeper. The idea of paying only for sessions, when everyone doing exploratory testing knows that only some of your working time is in sessions. The idea of crowdsourcing downplaying what skilled testers can accomplish, for people who can't distinguish "testing" from "testing" and buy just any testing.

I did two projects on the same application: testing on web for 2 hours 10 minutes, and testing on iOS devices for 3 hours 30 minutes. I logged 12 issues on web and 19 issues on iOS, and seems that I'm doing fine with accuracy/relevance. That's 11 minutes to a bug. And reporting properly takes time of course, since I would check reproducability, just like any good tester would.

I got paid for the iOS project today: 20 dollars for an hour. The Testlio site says they pay up to 35 euros an hour, but even with a lot of experience elsewhere, clearly my first project was not valued that high. I don't know what the criteria of deciding on payment is, but with the payment I end up with "thanks for the experience, but there's better things for me to do with my time". At first I thought the pay was 10 euros an hour, but since the payout was only for the iOS project, so 18 euros (20 dollars).

As financial comparison, let's do a bit of a calculation. Paying only for the effective testing hours means that the usual in-between-tasks time is included in the price. If you look at a full day of focused exploratory testing in session-based style, let's say that for 2 hours of testing, you would have 1 hour of off-charter time. You're still working, but not really on the charter. Digesting. Filling in gaps. Letting your brain bring things together. In this testing, for me it meant that I went back, unpaid, to close a bug I realized was duplicate. I investigated one bug further because I wasn't happy with what I new of it. I discussed with the project leads. Read instructions. Reviewed what other people had reported. And went back to check my feedback and issue resolutions.

That 18 euros for an hour is now only 12 euros for an actual hour. I would hardly volunteer to take a job that pays me 12 euros an hour, including "holiday pay". That's the minimal pay in Finland, not the pay for a test professional with 20 years of experience.

Feedback-wise, my dislike of templates got me a lower score eventually. I get it: as a test lead on a project where you can expect low-quality investigation for bug reports, you believe less of people. If I don't mention my user account or my network for a typo that is available also without logging in, I must not have checked. Or I must not have the skills to know what types of bugs might depend on what types of things that I will report whenever they are relevant. And I definitely wouldn't actively learn from my mistakes, since I'm only paid for the hours when the bugs are first logged. Except I did, since I'm a professional tester.

12 euros an hour might be much more lucrative for someone who does not live in Finland. Crowdsourcing customers who want testers specifically in the market / geographical area they target will have less choice around here. 

With the pay settled and me valued to a lot less than I make in my day job, I choose to vote that my free time is more valuable. I rather change the world by enjoying new applications in an open source project that values my contribution.


  1. I wonder - what drove you to give crowd-testing a try?

    When I look at the things that motivates me in testing, one of the major drives is the ability to influence the product I'm working on in a meaningful way. Besides that, I also take pride in being a professional tester (I have quite a lot to learn, but that's my profession, and I like to think I'm also good at it).
    There's none of that in crowd-testing. First of all, at least in the crowd test projects I've heard of, the testing begins once the application is relatively mature. Also, since most crowd-testers are not professional testers, there has to be some sort of mitigation mechanism - be it strict guidelines, off-limit areas ("don't file usability bugs") and probably a template to fill. If you join a crowd, why would you expect to be treated differently than any other crowd-member? In order to get your day job you had to be personally selected (by whichever mechanism) and deemed more suitable than any other applicant. In order to join the crowd one need only to click a button and pass a short and shallow review of some sort of profile by the person responsible for that project.

  2. I think I had two drivers:

    Curiosity for one. I wanted to know if Testlio had found a better way of doing this than the old uTest way I run into years ago (could have changed since). There's differences for the better, but not enough.

    Personal request was probably the main driver. Someone I know from the testing circles who works for Testlio personally asked me.

    I enjoy testing also mature applications, like ones that are in production and showing problems in them. But I too enjoy the long-term product development always-there type more. Variation is nice though. But if I had to choose one, I would go for the latter.

  3. I recently joined crowd-testing platforms too, and I had several reasons to do that:
    1. I had to step in for a colleague for 2 days in a new project. I knew nothing about it, regression testing was planned and I was told by the PM to ignore some of the tickets as they had been raised by a crowd-testing platform and would be given to them for the re-test. I immediately wondered how that worked, especially because the PM described them as the "knowledge of the masses". I grew curious and I think we should know what happens in the world of "testing" without being put-off by the perceived lack of professionalism of people engaging in that kind of testing.
    2. Ever since I stepped into testing, which was already 3 years ago, I constantly feel like an amateur. I've never ever taken a CS course before, so all I've learned in testing was on the job, by reading blogs etc. I come from the social sciences, I am not technical. I need to learn more and it is difficult to find where from. I am not a well-rounded professional and I think I need to train more, read more, do more. So when I have the time, I engage in this kind of exercising, because I am given access to a real app/site that will be out there for real users to use. Money in my case is no incentive, but potential learning in exploratory sessions is.
    3. I notice that I have a real pleasure in exploring the issues people submit. I think of it like this: these are I don't know, 20 people using the app. Sometimes they go into territories I never thought of testing before. Sometimes they look at functionalities that I as a tester perceived to be less important than others, but they engage in real bug hunting there. They are potential real users. When I test "professionally" I test for real users also. Why not learn from the small "patterns" I observe? Why not derive some use cases from the ticktes I see them raise? Issues that concern them, things that stand out to them but I would've ignored? And to be also completely honest, seeing some of the tickets has reassured me that testing is also about some mislaid images, missing page titles and icons…

    1. It's great to hear that it works as a training platform for you. I too look at everyone else's reports with curiosity. I do that also in my own project at work. I do that sometimes for open source projects, just to understand. And I adore Cem Kaner's list of issue taxonomy in his book Testing Computer Software, as it has over the years expanded my ideas of what a bug is.

      There's many ways of investing time to learning. It's great that you find that value in crowdsourcing. I find I get more when I actually pair with people over just reading the reports. I'd love to see a crowdsourcing platform that allows, even encourages that.