Continuous Delivery using Crowdsourced Testing

Often testing can be viewed as a barrier to implementing Continuous Delivery and DevOps, but it shouldn’t be. I'd like to share with you my experiences of how we crowd sourced manual testing, learned more about our customers and removed the “testing bottleneck” in our delivery process.

What is crowdsourced testing

In our myCloud project we have been successfully using crowdsourced testing services, provided by one of the world leaders in that field, since June 2016.

Crowdsourcing is “Taking a service traditionally performed by a person or team and sourcing it to a large online group of people as an open project”. (Definition by Jeff Howe, Wired journalist & author of the book Crowdsourcing).

Crowdsourced testing is therefore a unique way of involving crowd i.e. the real users/testers, into software testing under real world conditions.

Why crowdsourced testing

Even if the continuous delivery pipeline is almost all about the automation, there are still some  manual validations needed before your product is ready to be deployed to production. Our software is very user-centric and its success is determined by its user feedback. Our end users do not live in testing labs, but (potentially) in the whole world.

Also, testing our applications requires not only testing it from the end-user perspective but as well covering the tests on different mobile iOS and Android devices, operating systems and internet browsers.

Finally, this approach gives us the possibility to scale the testing resources in a very short time period (e.g. one to two days) to carry out the manual testing tasks.

What are we achieving with crowdsourced testing

  • From the business side, cost-benefit ratio is very good, since we get a bunch of testers who can be employed on a short-notice and flexible work schedule
  • Extensive operation system (Mac, Windows, iOS, Android) and browser test coverage is achieved.
  • Besides executing the regression test cases, the testers are also testing our products exploratory. Exploratory testing seeks to find out how the software works, and to ask questions about how it will handle difficult and easy cases. The quality of the testing is dependent on the tester’s skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be.
  • By using the crowdsourcing platform and adjusting our agile testing strategy over time, we were able to greatly improve the quality of our products and increase the faster quality time to market.

How we organize our crowd test cycle

Every week Crowd Test Leads and our Test Lead (myself) have a short Skype conference to discuss different topics like our test and release plan (shortly: what will be tested when), lessons learned from the finished test cycles and planning the next ones.

Towards the end of the Sprint (which lasts two weeks) the Crowd Test Leads will be informed about the testing assignments including:

  • Testing scope aka what products should be tested
  • Test plan with a list of known and out of scope issues
  • Internal release notes and sprint review presentation / demo so testers can understand the product(s) and its new features even better

After the test cycle is closed, our Test Lead will be provided with the test report showing the most important found issues as well as the device, browser and OS coverage. Every issue found by the crowd tester will be first approved or rejected by Crowd Test Lead before it’s handed over for a review to our Test Lead, Development Team and Product Owner.

Learn more about crowdsourced testing

If you want to hear in much more detail about this topic and especially what else it takes to be able to continuously deliver to production, come to my talk at Swiss Testing Day conference on 14. March at Samsung Hall Zurich Stettbach.


Photo by Jonathan Singer on Unsplash