CiML 2020: NeurIPS 2020 Workshop on ML Competitions at the Grassroots virtual December 11-12, 2020 |
Conference website | https://sites.google.com/chalearn.org/ciml/ciml2020 |
Submission link | https://easychair.org/conferences/?conf=ciml2020 |
Abstract registration deadline | September 21, 2020 |
Submission deadline | October 12, 2020 |
For the eighth edition of the CiML (Challenges in Machine Learning) workshop at NeurIPS, our goals are to: 1) Increase diversity in the participant community in order to increase quality of model predictions; 2) Identify and share best practices in building AI capability in vulnerable communities; 3) Celebrate pioneers from these communities who are modeling lifelong learning, curiosity and courage in learning how to use ML to address critical problems in their communities.
The workshop will provide concrete recommendations to the ML community on designing and implementing competitions that are more accessible to a broader public, and more effective in building long-term AI/ML capability.
The workshop will feature keynote speakers from ML, behavioral science and gender and development, interspersed with small group discussions around best practices in implementing ML competitions. We will invite submissions of 2-page extended abstracts on topics relating to machine learning competitions, with a special focus on methods of creating diverse datasets, strategies for addressing behavioral barriers to participation in ML competitions from underrepresented communities, and strategies for measuring the long-term impact of participation in an ML competition.
Submission Guidelines
We welcome 2-page extended abstracts on topics relating to challenges in machine learning. Selected papers will be presented primarily as posters, but exceptional contributions will be given oral presentations. Abstracts should be submitted by October 2nd, 2020 (by following instructions on the website: https://sites.google.com/a/chalearn.org/workshop/ciml2020/callforabstracts). [You can use the NeurIPS template for your submissions; submission need NOT be anonymized; and extra page can be used for references and acknowledgements].
List of Topics
Topics of interest for 2020 are methods of creating diverse datasets, strategies for addressing behavioral barriers to participation in ML competitions from underrepresented communities and strategies for measuring the long-term impact of participation in an ML competition. Other topics of interest include, but are not limited to:
Methods:
- Novel or atypical challenge/competition protocols, particularly relating to research.
- Novel or atypical challenge/competition protocols to tackle complex tasks with very large datasets, multimodal data, and data streams.
- Methods and metrics of entry evaluation, quantitative and qualitative challenges.
- Methods of data collection, "ground-truthing", and preparation including bifurcation/anonymization, data generating models.
- Teaching how to organize competitions and challenges
- Hackathons, datathons and on-site challenges.
- Challenge indexing and retrieval, challenge recommenders.
Theory:
- Experimental design, size data set, data split, error bounds, statistical significance, violation of typical assumptions.
- Game theory applied to the analysis of challenge participation, competition and collaboration among participants.
- Diagnosis of data sanity, artifacts in data, data leakage.
- Understanding of human behavior to design challenges that result in long-term positive, change in the community
Implementation:
- Reusable challenge platforms, innovative software environments.
- Linking data and software repositories to challenges.
- Security/privacy, intellectual property, licenses.
- Cheating prevention and remedies (i.e. leaderboard climbing).
- Issues raised by requiring code submission.
- Challenges requiring user interaction with the platform (active learning, reinforcement learning).
- Dissemination, fact sheets, proceedings, crowdsourced papers, indexing post-challenge publications.
- Long term impact, on-going benchmarks, metrics of impact.
- Participant rewards, stimulation of participation, advertising, sponsors.
- Profiling participants, improving participant professional and social benefits.
Applications:
- Challenges for the benefit of society, as a scientific research tool, for up-skilling, or to solve industry problems.
- Where to venture next: opportunities for challenge organizers to organize challenges in new domains with high societal impact.
- Successful challenges leading to significant breakthrough or improvement over the state-of-the-art or unexpected interesting results.
- Rigorous study of the impact of challenges, analyzing topics and tasks lending themselves to high impact machine learning challenges.
- Challenges organized or supported by Government agencies, funding opportunities.
Committees
Program Committee
- Isabelle Guyon (UPSud/INRIA, U. Paris-Saclay and ChaLearn)
- Evelyne Viegas (Microsoft Research, USA)
- Wei-Wei Tu (4Paradigm Inc. and ChaLearn)
- Sergio Escalera (U. Barcelona, Spain)
- Andreas Holzinger (U. Graz, Austria)
- Hugo Jair Escalante (IANOE, Mexico)
- Vincent Lemaire (Orange Labs, France)
- Fabrice Popineau (Centrale Supelec, France)
Organizing committee
Invited Speakers
Publication
CiML 2020 proceedings will be published on the CiML website.
Venue
The conference will be held virtually.
Contact
All questions about submissions should be emailed to kathryn@technovation.org.