Crowdsourcing has shown to be a powerful technique for overcoming many challenges in data and information processing where current state-of-the-art algorithms are still struggling. This is especially true for workflows that transparently combine algorithmic heuristics and dynamically crowdsourced tasks that are performed by human workers, and which promise to solve even more complex tasks effectively and efficiently. But still, such hybrid crowdsourcing workflows can be difficult to approach, and they are often designed in an ad-hoc fashion. Therefore, in this paper, we extensively investigate such crowdsourcing workflows as described in the literature, and abstract generic design patterns, which codify commonly recurring challenges and their best-practice solutions. Each design pattern is described and discussed with a special focus on its requirements, constraints, and effects on the overall workflow. We illustrate the practicality of these patterns by providing real-world application examples where such patterns can or have been applied. Furthermore, we showcase how the individual design patterns can be extended and combined to support more complex workflows. Our design patterns provide an extensive overview of the hybrid crowdsourcing workflows’ design space, and allow for a more efficient modeling, analysis, and documentation of such workflows.
|