Crowdsourcing is a kind of participative online activity in which a requester submits a human intelligence task to a platform on the Internet; the task is then performed by data contributors as volunteers or for payment. Crowdsourcing has proved itself as a scalable and flexible approach that can handle the rapid demand for data collection, annotation, and management methodology for the variety of applications, including surveys, disease spread prevention, ML/AI applications, etc.

Due to the compexity of the crowdsourcing phenomenon, ethical and trusted crowdsourcing at scale is still a challenge that requires effort from all the stakeholders of the crowdsourcing industry. Researchers, requesters, platforms, and data contributors need to work together to help the ecosystem be successful. Our workshop allows the community to focus on identifying and sharing best practices for delivering individual tasks, while involving efficient data management methods to mediate and facilitate the interaction for obtaining reliable results.

Topics

In order to achieve the goal of our workshop, we identify four essential topics of interest that we expect to raise, discuss, and advance this year: trust, ethics, large-scale data excellence, and crowd-AI interplay.

Instructions and Format

There are two types of submissions:

• "regular" papers that provide a validated scientific contribution to crowd science;
• work-in-progress or vision papers that propose ideas or present preliminary results.

In both cases, a paper can provide research contributions (theoretical and applied) or can describe designs and implementations of solutions/systems for practical tasks in some industry. In the latter case, papers solve or advance the understanding of ​issues related to deploying technologies based on human data supply for automation in the real world.

Papers should be submitted via EasyChair. All submissions must be in PDF format. Short papers for work-in-progress/vision have 5–9 pages, long papers have 10–15 pages. These limits are for main content pages, including all figures and tables. Additional pages containing appendices, acknowledgements, funding disclosures, and references are allowed. You must format your submission using the CEUR-WS.org style, CEURART. The maximum file size for submissions is 50MB. Submissions that violate the paper style (e.g., by decreasing margins or font sizes) or page limits may be rejected without further review. 

In all cases, a submitted paper must describe new, unpublished research, and must not have been published or under review elsewhere.

Review Process

Each paper will be reviewed by at least three members of the program committee. Reviewers will be instructed to evaluate paper submissions according to specific review criteria. We encourage authors to review them before submission. The reviewing process will be double blind at the level of reviewers (i.e., reviewers cannot see author identities) but not at the level of workshop organizers. As an author, you are responsible for anonymizing your submission. In particular, you should not include author names, author affiliations, or acknowledgements in your submission and you should avoid providing any other identifying information (even in the supplementary material).

Paper Presentation

At least one author of each accepted paper must register for the main conference, VLDB 2021, to present the work or acceptance will be withdrawn.

Invited Speakers

• Michael Bernstein, Stanford University
• Wuraola Fisayo Oyewusi, Data Science Nigeria
• Ujwal Gadiraju, Delft University of Technology

Contact

If you are interested in helping out with the review process or you have other questions, please get in touch with us: crowd_workshop@yandex-team.com