Back then, I identified two options we had:
- We could use Crowdsourcing - pay a fee for service that allows us to use random testers to use the product to find problems.
- We could use Contracting - pay a fee for having one tester dedicated to grow with us through testing for us.
I come back to thinking of this, as last week I decided to try testing with Testlio. As far as I can tell, they are another option to uTest back then (Applause nowadays) on crowdsourcing, with the specific aspect of not paying for results but for assigned hours.
I'm just about to start my third round of "let's see what this is about" with Testlio. At this point I don't know what the payment by hour is - from their pages I can tell that they pay less than my day job even in the highest category, but I guess that comes with seniority. But from Testlio perspective, I'm a newbie, so I doubt they will place me in the higher rates without specific Testlio-experience - one that I'm unlikely to build up.
From a professional tester's viewpoint, here's what happens with Testlio:
- They invite me to projects and assign me work packages
- Timeframes for work package availability are short and I can say if I volunteer 2 hours of my time today or not before starting
- As I start a work package, I start a timer. You're paid on the time. Forgetting to start a timer (newbie mistakes) can be corrected by discussing with the test leads.
- I get a high level checklist (find this feature and mark pass/fail) or a charter (spend an hour doing X).
- I test what I can on the time given, and report. Reporting includes adding notes on a very high level plan and bug reports.
Testlio approach is better with the hourly payment than the other options back in the days I looked at them for a tester perspective. If you get paid only when you find valid bugs first, it adds another level of shallowness and leaves out things that might get reported in the hourly model.
- Refreshed. Testing something different is refreshing.
- Unchallenged. Short timeframes invite shallowness. Even a few consecutive work packages on the same test target with different versions still leaves the feeling of shallowness. Within 2-3 hours, you won't investigate anything complicated.
- Unresponsible. My job is done with the clock. I do the best I can within that timeframe. I have no connection with the product. I have no social relations to others - at least not paid social relations.
- The pay & the progress. I expect that as soon as I learn how little I was paid for these, I rather contribute time on open source projects.
- The bureaucracy. I already got a request to follow the issue template. I hated the issue template and felt constrained. It had repetitive information to be filled in. It had unnecessary information for most bugs I logged. Seriously, even copypasting my service provider info when there's simple UI bugs that have NOTHING to do with the network makes me feel like a rubber stamp. I would choose to use my time better while I understand that requiring same info comes from lessons of missing that info on some bugs sometimes.
- The shallowness in the model. It's great that some companies use this to find bugs. But I find the idea that they use this as replacement of inhouse testers, thinking they are getting the same thing inherently wrong. The timeframes here are too short for proper testing, the testing in 2 hours remains shallow. I feel the whole crowdsourcing model drives intelligent deep testing down by implying that a short time is enough. It's not about just having skill in testing, but learning to dig deeper takes time.