Crowdsourcing has been used to produce impactful and large-scale datasets for Machine Learning and Artificial Intelligence (AI), such as ImageNET, SuperGLUE, etc. Since the rise of crowdsourcing in the early 2000s, the AI community has been studying its computational, system design, and data-centric aspects at various angles at such workshops as CSS, CrowdML, DCAI, and HILL. 

This workshop focuses on improving the welfare of the invisible labeling crowd force behind the AI, i.e., the crowdworkers.

We welcome studies that focus on the following (non-exhaustive) list of topics:

• developing and enhancing crowdworker-centric tools that offer task matching, requester assessment, and instruction validation, among other topics;
• active learning techniques, methods for joint learning from noisy data and from crowds, novel approaches for crowd-computer interaction, repetitive task automation, and role separation between humans and machines;
• designing and applying such techniques in various domains, including e-commerce and medicine.

Instructions and format

Papers should be submitted through EasyChair. All submissions must be in PDF format and must be anonymized (names and affiliations of the authors removed).

You can use any common template to format your submission. We do not impose constraints on the length of submissions, but reviewers will not be required to read beyond the first 8 pages.

In addition to presenting in the workshop, authors of accepted papers will have an optional opportunity to publish their work in the proceedings of the workshop. If you are willing to have your paper published, you should adhere to additional requirements:

• A paper should be 5–15 pages. These limits are for main content pages, including all figures and tables. Additional pages containing appendices, acknowledgements, funding disclosures, and references are allowed.
• You must format your submission using the CEUR-WS.org style, CEURART. The maximum file size for submissions is 50MB. Submissions that violate the paper style (e.g., by decreasing margins or font sizes) or page limits may be rejected publication in the proceedings.

Review Process

Each paper will be reviewed by at least three members of the program committee. Reviewers will be instructed to evaluate paper submissions according to specific review criteria. We encourage authors to review them before submission.

The reviewing process will be double blind (i.e., reviewers cannot see author identities). As an author, you are responsible for anonymizing your submission. In particular, you should not include author names, author affiliations, or acknowledgements in your submission and you should avoid providing any other identifying information (even in the supplementary material).

Paper Presentation

At least one author of each accepted paper must register for the main conference, WSDM 2023, to present their work.

Invited Speakers

• Ujwal Gadiraju, Delft University of Technology
• Djellel Difallah, NYU Abu Dhabi

Organizers

• Dmitry Ustalov, Toloka, Belgrade, Serbia
• Saiph Savage, Northeastern University Boston, MA, USA
• Niels van Berkel, University of Aalborg Aalborg, Denmark
• Yang Liu, University of California Santa Cruz, Santa Cruz, USA
• Alisa Smirnova, Toloka, Lucerne, Switzerland

Contact information

Don’t hesitate to contact us if you have any questions or suggestions regarding this workshop: research@toloka.ai