Crowdsourcing is a newly emerging service platform and business model in the Internet. In contrast to outsourcing, where a job is performed by a designated worker or employee, crowdsourcing means to outsource a job to a large, anonymous crowd of workers, the so-called human cloud, in the form of an open call. Current research in crowdsourcing addresses the following issues: crowdsourcing as a novel methodology for user-centered research; development of new services and applications based on human sensing, computation, and problem solving; engineering of improved crowdsourcing platforms including quality control mechanisms; incentive design and gamification of work; usage of crowdsourcing for professional business; theoretical frameworks for evaluation. The topic on crowdsourcing may have a huge impact on the Internet and its technical infrastructure, on society, and the future of work. Therefore, this seminar helps coordinating research efforts in the different communities, especially in US currently leading the crowdsourcing market and in Europe. In summary, crowdsourcing will be a guiding paradigm and form the evolution of work in the next years.

This Dagstuhl Seminar brings together experts from the different research fields as well as experts from industry with a practical background on the deployment, operation or usage of crowdsourcing platforms. From industry, real-world problem statements, requirements and challenges, position statements, innovative use cases, and practical experiences are desired. The collection and analysis of practical experiences of the different crowdsourcing stakeholders are one of the important outcomes of the Dagstuhl Seminar. Since the aim is to bring together researchers from academia and industry, the identification of problems and challenges and how to tackle them will be very inspiring. In particular, platform providers may report about their experiences and lessons learned when building and operating the platform. But also the experiences from experiment designers and employers are crucial, what are pitfalls or harming user engagement, how to foster the quality of work.

As the field of crowdsourcing is novel, a common terminology, classification and taxonomy of crowdsourcing systems, as well as evaluation frameworks are required. This will be one of the goals of the Dagstuhl Seminar to derive such a research methodology. The impact of crowdsourcing from different perspectives shall be discussed. In this context, we invite experts from different research disciplines to discuss the impact on society, on business and economics, on law, but also on the Internet infrastructure. It has to be clearly noted that the scope of the seminar is on technical challenges, but the potential impact and long-term perspectives have to be discussed from an interdisciplinary point of view, too. Theoretical results and research methodologies from different disciplines may improve current platforms and applications.

Within the Dagstuhl Seminar, the following topics will be addressed.

• Quo vadis Crowdsourcing: From sensing to problem solving: Crowdsourcing may be used for simple, micro-tasks like sensing of information (e.g. with smartphones), tagging of data (e.g. labeling of pictures). The challenge is on automatic sensing, where relevant data (e.g. for environment pollution sensing) is collected and analyzed automatically and in a collaborative way among users. Furthermore, the energy consumption of mobile devices with mobile human sensors has to be addressed. As a next step of crowdsourcing, the crowd is used as problem solver. In this context, human-based computation is gaining momentum where the machine cloud "asks" the human cloud to solve a problem. The solutions proposed then have to be collected, aggregated, interpreted, evaluated and integrated.
• Mechanisms for improving crowdsourcing: In order to reach the long-term perspectives of crowd¬sourcing, mechanisms are required which allow to improve the current platforms. Therefore, various requirements to the platforms exists which have to be addressed. Those mechanisms address recommendation systems, anonymous user profiles and specialized crowds, automated task design, incentive design, quality assurance and reliability.
• Applications and Use Cases: The development of improved crowdsourcing mechanisms will be based on relevant and innovative use cases. These use cases cover orthogonal dimensions that are type of work and operational conditions: cowdsensing considers in particular mobile or participatory sensing part of ubiquitous crowdsourcing; crowdsolving lets users perform tasks in categories such as field research, photography and media, etc; crowdtesting utilizes the human cloud for conducting scientific studies, e.g. in the context of user perceived quality; another relevant category considers data extraction, e.g. to sort data according to certain criteria like quality of music; further applications consider crowdvoting for gathering opinions and trends, as well as crowdwisdom to gather knowledge e.g. Wikipedia.
• Operational conditions: In this context, we differentiate between enterprise crowdsourcing for professional usage within an enterprise. In that context, the separation of efforts that engage employees vs. public crowds is to be considered. Real-time crowdsourcing addresses the completion of work in real-time and invokes additional challenges, also on the technical infrastructure to cope with massive requests at the same time, i.e. flash crowd effects as known from P2P networks. As a result of ubiquitous connectivity and advances in mobile technologies, ubiquitous crowdsourcing emerges with mobile users seamlessly forming interactive networks and participation in a variety of tasks involving gathering, analyzing and sharing data, such as reporting security threats, natural disasters, or information for location-based services.

The following research questions will be addressed in the seminar towards a common research methodology and long-term perspectives of crowdsourcing.

• What is an appropriate taxonomy for classifying and analyzing crowdsourcing systems? How can crowdsourcing tasks be grouped based on their task complexity and along key challenges in successfully harvesting expertise of large human networks?
• Which use cases and applications will exploit the potential of crowdsourcing?
• How does the research community approach improved crowdsourcing mechanisms e.g. for quality and cost control or reliability of users and devices? Which requirements and challenges occur for particular operational conditions, like ubiquitous crowdsourcing due to the user mobility in time and space?
• How to design incentive schemes for coordinated problem solving of the crowd among individual humans with their own goals and interests? How to realize gamification of work for improved user engagement? How to identify expertise of users? How to implement such incentive schemes technically?
• How can the experiment and task design be standardized? Which kinds of APIs or templates are promising and useful in practice?
• What are the objectives to be fulfilled and the necessary capabilities of platforms towards the provision of Future Internet services built on top of crowdsourcing facilities?
• How can crowdsourcing systems be evaluated? Which common research methodologies are applicable? Which theories and models from a number various fields are applicable, including artificial intelligence, multi-agent systems, game theory, operations research, or human-computer interaction? How to include human-centric measures such as costs, availability, dependability and usability, including device-specific properties in evaluation frameworks?
• How does the research agenda for crowdsourcing look like in the next years?

Please note that a topically related seminar on "Cloud-based Software Crowdsouring", organized by Michael N. Huhns, Wei Li, Martin Schader and Wei-Tek Tsal,( Dagstuhl Seminar 13362) takes place in parallel to this seminar. We plan common sessions on topics of mutual interest.

• Abraham Bernstein (Universität Zürich, CH) [dblp]
• Kathrin Borchert (Universität Würzburg, DE) [dblp]
• Alessandro Bozzon (TU Delft, NL) [dblp]
• Cristina Cabanillas (Wirtschaftsuniversität Wien, AT) [dblp]
• Joseph Davis (The University of Sydney, AU) [dblp]
• Gianluca Demartini (University of Fribourg, CH) [dblp]
• Klaus Diepold (TU München, DE) [dblp]
• Matthias Hirth (Universität Würzburg, DE) [dblp]
• Tobias Hoßfeld (Universität Würzburg, DE) [dblp]
• Andreas Hotho (Universität Würzburg, DE) [dblp]
• Deniz Iren (Middle East Technical University - Ankara, TR) [dblp]
• Christian Keimel (TU München, DE) [dblp]
• Shinichi Konomi (University of Tokyo, JP) [dblp]
• Vassilis Kostakos (University of Oulu, FI) [dblp]
• Markus Krause (Universität Hannover, DE) [dblp]
• Martha A. Larson (TU Delft, NL) [dblp]
• Babak Naderi (TU Berlin, DE) [dblp]
• Nhatvi Nguyen (Weblabcenter, Inc. - Texas, US) [dblp]
• Munindar P. Singh (North Carolina State University - Raleigh, US) [dblp]
• Phuoc Tran-Gia (Universität Würzburg, DE) [dblp]
• Maja Vukovic (IBM TJ Watson Research Center - Yorktown Heights, US) [dblp]
• Florian Zeiger (AGT International - Darmstadt, DE) [dblp]