Crowdsourcing has become a standard tool to obtain large amounts of carefully annotated data. Nowadays, its applicability has exceeded the classical purpose of collecting data to train AI algorithms. Indeed, crowdsourcing is often used to conduct research studies and to gain valuable insights from public. 

The focus of this workshop is on the best practices of efficient and trustworthy crowdsourcing. We identify three essential topics of interest that we expect to raise, discuss, and advance this year: Large-Scale Data Excellence, Trust and Ethics, and Crowd-AI Interplay.

Instructions and Format

Papers should be submitted via EasyChair. All submissions must be in PDF format and must be anonymized (names and affiliations of the authors removed).

You can use any common template to format your submission. We do not impose constraints on the length of submissions, but reviewers will not be required to read beyond the first 8 pages.

In addition to presenting in the workshop, autors of accepted papers will have an optional opportunity to publish their work in the proceedings of the workshop. If you are willing to have your paper published, you should adhere to additional requirements:

• A paper should be 5–15 pages. These limits are for main content pages, including all figures and tables. Additional pages containing appendices, acknowledgements, funding disclosures, and references are allowed.
• You must format your submission using the CEUR-WS.org style, CEURART. The maximum file size for submissions is 50MB. Submissions that violate the paper style (e.g., by decreasing margins or font sizes) or page limits may be rejected publication in the proceedings.

Review Process

Each paper will be reviewed by at least three members of the program committee. Reviewers will be instructed to evaluate paper submissions according to specific review criteria. We encourage authors to review them before submission.

The reviewing process will be double blind at the level of reviewers (i.e., reviewers cannot see author identities) but not at the level of workshop organizers. As an author, you are responsible for anonymizing your submission. In particular, you should not include author names, author affiliations, or acknowledgements in your submission and you should avoid providing any other identifying information (even in the supplementary material).

Paper Presentation

At least one author of each accepted paper must register for the main conference, VLDB 2021, to present the work or acceptance will be withdrawn.

Invited Speakers

• Michael Bernstein, Stanford University
• Wuraola Fisayo Oyewusi, Data Science Nigeria
• Ujwal Gadiraju, Delft University of Technology

Contact

If you are interested in helping out with the review process or you have other questions, please get in touch with us: crowd_workshop@crowdscience.ai