Showing posts with label Crowdsourcing. Show all posts
Showing posts with label Crowdsourcing. Show all posts

Wednesday, August 1, 2018

Seeing what is Right when Everyone Seems Happy with Wrong

I'm uncomfortable. I've been postponing taking this topic up in a blog post, convinced that there must be something I'm missing. But it may be that there is something I'm seeing that many others are not seeing.

A few months ago, I volunteered at work to try out crowdtesting on my product. It wasn't the first try of crowdsourcing, quite the contrary as I learned we were a happy user of one of those services for some other products.

The concern I had surprised everyone, providing a perspective no one else seemed to think about. I was concerned if using a service like this would fit my ideas of what is ethical. I was not focused on the technicalities and value and usefulness of the service but whether using the service was right. It isn't obviously wrong because it is legal. Let's look a little more where I come from.

I live in Finland, where we have this very strongly held idea that work needs to be paid for. Simultaneously, we are a promised land of non-profit organizations that run on volunteer work having  huge numbers of organizations like that in relation to number of people. But the overall structure of how things are set up is that you don't have free labor in companies.

The unemployed and the trainees get paid for the work they are doing. And this forces companies to be on good behavior in the social structures and not leech of the less fortunate.

So if my company in general wants someone to do testing, they have three options.

1) Customers Test
They mask it as "no need to test" and force their end users to test, and pay for structures that enable them to figure out what a normal user is actually complaining on, and structures for fixing things fast when the problems hit someone who not only raises their voice but walks out with a relevant amount of money.

2) Someone in their employee list tests
They pay someone for some of the testing, and some companies pay testers to get more of the whole of testing done. If you tested half of what you needed, you still tested. And this is the narrative that makes the current programmers test it all trend discussions so difficult in practice. There's testing and there's testing. It takes a bit of understanding to tell those two apart.

3) Someone in a service provider organization tests
They pay for a vendor for doing some testing. The vendor is an abstraction, a bubble where you conveniently allocate some of your responsibilities, and in many cases you can choose to close your eyes at the interface level.

Crowdtesting belongs to the third option, and relies in my view on closing your eyes at the interface level.  Crowdtesting is the idea of paying a company for a service of them finding you a crowd. They find the crowd by paying those testers, but I'm not convinced that the models of paying represent what I would consider fair, right and ethical.

So, I pay a legitimate company 5000 euros for them to do some testing we agree on under the label "crowdtesting". Yay! Is that really enough of thinking from my part? You get 30 testers with that, so cheap! (numbers are in scale, but not actuals). They even promise most of them will be in Finland and other Nordic countries. If your alarm bells aren't ringing, you are not thinking.

The more traditional companies producing things like coffee or clothes know painfully well it isn't. You really don't want your fashion brand be associated with child labor, inhumane working conditions or anything other nasty like that. Surely you can save loads of money using companies where their delivery chain hides the unethical nature of how the service is created, but you risk a reputation hits. Because someone hopefully cares.

Crowdtesting companies are not in the business of child labor, at least to my knowledge. But they are in the business of forcing my people (testers) away from being paid for their work, to be paid for their results. And with results being bugs and checkmarks in test cases, it's actively generating a place where folks in these programs are not necessarily paid for the work they do.

The way the monthly fee is set up for the customer makes this worse. There's a limit on how much of these paid checkmarks you can allocate a month, but you're told you're allowed to ask for unlimited exploratory testing  on top of that. The financial downside for the people doing testing in the low end of this power dynamic is that they are only paid for bugs customer accepts.

Many people seem really happy with doing testing in these schemes.

The ones promoted to the middle management layer get paid for their hours, leaving even less of the financial cake for the low end of the power dynamic. But why would they care, they get paid.

The ones privileged enough to not really need a job that pays you a salary get paid whatever, but since the money never mattered, why should they care? This is a hobby on the side anyway.

The ones living in cheaper countries getting paid the same amount as the testers from the Nordics may actually make a decent amount of money in finding problems, and might not even have a choice.

The really good ones who always find problems can also get paid, as long as the effort of finding something relevant isn't getting too high.

But the ethical aspect that I look at is local. The low end of the power dynamic are the testers kicked out of their projects who no longer need them, but actually do. They just no longer want to pay for all of their work and hide this fact in the supply chain.

Maybe I should just think more global. I don't know. But I am uncomfortable enough to step away. I don't want to do this.

With the same investment, I can get someone who works close to the team and provides more value. Without being exploited. With possibilities of learning and growing. Being appreciated.

And I say this looking from a higher end of that dynamic. I'm not losing my job in this shift.

I don't like crowdsourcing. Do you?


Wednesday, January 20, 2016

The end of my crowdsourcing experience

I tried testing on a crowdsourced project with Testlio from a direct request. I did not care to ask the pay in advance, I just thought it would be a fun experience.

The testing part of it was a fun experience. If you love testing even a little, you might recognize the rush of getting to know a new application and finding problems there. You might also know the frustration when you need to find simple and basic bugs in functionality, knowing that those anyone could spot: including the developer who created the app in the first place. New project is always a rush of emotions, and new projects tend to end up with positive. This one was no different.

While testing, there was already more of frustrations. The idea of shallow testing due to a short timeframe, in which you won't have time to investigate or dig deeper. The idea of paying only for sessions, when everyone doing exploratory testing knows that only some of your working time is in sessions. The idea of crowdsourcing downplaying what skilled testers can accomplish, for people who can't distinguish "testing" from "testing" and buy just any testing.

I did two projects on the same application: testing on web for 2 hours 10 minutes, and testing on iOS devices for 3 hours 30 minutes. I logged 12 issues on web and 19 issues on iOS, and seems that I'm doing fine with accuracy/relevance. That's 11 minutes to a bug. And reporting properly takes time of course, since I would check reproducability, just like any good tester would.

I got paid for the iOS project today: 20 dollars for an hour. The Testlio site says they pay up to 35 euros an hour, but even with a lot of experience elsewhere, clearly my first project was not valued that high. I don't know what the criteria of deciding on payment is, but with the payment I end up with "thanks for the experience, but there's better things for me to do with my time". At first I thought the pay was 10 euros an hour, but since the payout was only for the iOS project, so 18 euros (20 dollars).

As financial comparison, let's do a bit of a calculation. Paying only for the effective testing hours means that the usual in-between-tasks time is included in the price. If you look at a full day of focused exploratory testing in session-based style, let's say that for 2 hours of testing, you would have 1 hour of off-charter time. You're still working, but not really on the charter. Digesting. Filling in gaps. Letting your brain bring things together. In this testing, for me it meant that I went back, unpaid, to close a bug I realized was duplicate. I investigated one bug further because I wasn't happy with what I new of it. I discussed with the project leads. Read instructions. Reviewed what other people had reported. And went back to check my feedback and issue resolutions.

That 18 euros for an hour is now only 12 euros for an actual hour. I would hardly volunteer to take a job that pays me 12 euros an hour, including "holiday pay". That's the minimal pay in Finland, not the pay for a test professional with 20 years of experience.


Feedback-wise, my dislike of templates got me a lower score eventually. I get it: as a test lead on a project where you can expect low-quality investigation for bug reports, you believe less of people. If I don't mention my user account or my network for a typo that is available also without logging in, I must not have checked. Or I must not have the skills to know what types of bugs might depend on what types of things that I will report whenever they are relevant. And I definitely wouldn't actively learn from my mistakes, since I'm only paid for the hours when the bugs are first logged. Except I did, since I'm a professional tester.

12 euros an hour might be much more lucrative for someone who does not live in Finland. Crowdsourcing customers who want testers specifically in the market / geographical area they target will have less choice around here. 

With the pay settled and me valued to a lot less than I make in my day job, I choose to vote that my free time is more valuable. I rather change the world by enjoying new applications in an open source project that values my contribution.



Tuesday, January 19, 2016

Crowdsourcing and what to think of it

About three years ago as I had just joined my current projects, we needed more testing to happen than I could do. The developers test, they had tested before but with the level of skills on testing (and interest) the end result was that the main outcome of their testing back then was increased coffee consumption in avoidance of doing the work.

Back then, I identified two options we had:
  1. We could use Crowdsourcing - pay a fee for service that allows us to use random testers to use the product to find problems.
  2. We could use Contracting - pay a fee for having one tester dedicated to grow with us through testing for us. 
Comparing back then the prices of uTest and the Romanian contractor we considered, the differences on the monthly fee were not relevant. So the question really went down to contents. We decided at that point to be better off with contracting, and from that decision, coaching to have another deep tester in the world commenced.

I come back to thinking of this, as last week I decided to try testing with Testlio. As far as I can tell, they are another option to uTest back then (Applause nowadays) on crowdsourcing, with the specific aspect of not paying for results but for assigned hours.

I'm just about to start my third round of "let's see what this is about" with Testlio. At this point I don't know what the payment by hour is - from their pages I can tell that they pay less than my day job even in the highest category, but I guess that comes with seniority. But from Testlio perspective, I'm a newbie, so I doubt they will place me in the higher rates without specific Testlio-experience - one that I'm unlikely to build up.

From a professional tester's viewpoint, here's what happens with Testlio:
  • They invite me to projects and assign me work packages 
  • Timeframes for work package availability are short and I can say if I volunteer 2 hours of my time today or not before starting
  • As I start a work package, I start a timer. You're paid on the time. Forgetting to start a timer (newbie mistakes) can be corrected by discussing with the test leads. 
  • I get a high level checklist (find this feature and mark pass/fail) or a charter (spend an hour doing X).
  • I test what I can on the time given, and report. Reporting includes adding notes on a very high level plan and bug reports. 
Since I volunteer for this as tester, I probably try to do my best in the time allocated. Even as user, I am likely to do a decent job if nothing special is expected of me. Buggy software (like work package 2) makes skilled tester like me even less necessary. For a tester like me, it ended up being hard to stop, so free work gets done. I even dream of things I could have still checked.

Testlio approach is better with the hourly payment than the other options back in the days I looked at them for a tester perspective. If you get paid only when you find valid bugs first, it adds another level of shallowness and leaves out things that might get reported in the hourly model.

I like looking at how I feel while doing this.
  • Refreshed. Testing something different is refreshing. 
  • Unchallenged. Short timeframes invite shallowness. Even a few consecutive work packages on the same test target with different versions still leaves the feeling of shallowness. Within 2-3 hours, you won't investigate anything complicated.
  • Unresponsible. My job is done with the clock. I do the best I can within that timeframe. I have no connection with the product. I have no social relations to others - at least not paid social relations.
I currently have three main concerns:
  1. The pay & the progress. I expect that as soon as I learn how little I was paid for these, I rather contribute time on open source projects. 
  2. The bureaucracy. I already got a request to follow the issue template. I hated the issue template and felt constrained. It had repetitive information to be filled in. It had unnecessary information for most bugs I logged. Seriously, even copypasting my service provider info when there's simple UI bugs that have NOTHING to do with the network makes me feel like a rubber stamp. I would choose to use my time better while I understand that requiring same info comes from lessons of missing that info on some bugs sometimes. 
  3. The shallowness in the model. It's great that some companies use this to find bugs. But I find the idea that they use this as replacement of inhouse testers, thinking they are getting the same thing inherently wrong. The timeframes here are too short for proper testing, the testing in 2 hours remains shallow.  I feel the whole crowdsourcing model drives intelligent deep testing down by implying that a short time is enough. It's not about just having skill in testing, but learning to dig deeper takes time. 
The two first concerns will soon drive me to prioritize other things to do with my time. The third concern is bigger: I don't need to be a part of this change but the change has been here for a while already. What should be my stance on it?