MuMe 2018: 1st International Workshop on Multi-Method Evaluation of Personalized Systems Nanyang Technological University Singapore, Singapore, July 8, 2018 |
Conference website | https://multimethods.info |
Submission link | https://easychair.org/conferences/?conf=mume2018-workshop |
Submission deadline | April 17, 2018 |
The MuMe 2018 workshop is based on the objective to raise awareness in the user modeling community for the significance of using multiple methods in the evaluation of recommender systems and other personalized systems.
Employing a multi-method evaluation integrating a number of single methods (e.g., a combination of think-aloud and survey with open-ended questions or a combination of offline prediction simulation with an open dataset and survey with closed- and open-ended questions) allows for getting a more integrated and richer picture of user experience and quality drivers of personalized systems.
The primary goal of this workshop is to build a community around the multi-method evaluation topic and to develop a long-term research agenda for the topic.
The primary goals of the MuMe workshop are:
- to build a community around the topic of multi-method evaluation;
- to discuss challenges for identifying, employing, and integrating multi-method evaluation;
- to develop a long-term research agenda to solve and/or overcome the unsolved challenges.
Submission Guidelines
We solicit position and research papers (4 pages excluding references, UMAP 2018 Extended Abstracts Format) that address challenges in the multi-method evaluation of recommender systems and other personalized systems.
This includes
- “lessons learned” from the successful application of multi-method evaluations,
- “post mortem” analyses describing specific evaluation strategies that failed to uncover decisive elements,
- “overview papers” analyzing patterns of challenges or obstacles to multi-method evaluation, and
- “solution papers” presenting solutions towards identified challenges.
All papers must be original and not simultaneously submitted to another journal or conference.
We support and encourage submissions addressing diverse evaluation scenarios from various RecSys application domains (e.g., real-time music-recommendation for jogging, emotion-aware microblog discovery, trust in social recommendations and cultural differences, healthy-food recommendation for children education, lifestyle recommendation based on natural language interaction).
Submissions will undergo single-blind peer-review by two reviewers and a meta-review by a program committee member and will be selected based on quality, novelty, clarity, and relevance; aiming to foster discussions. Accepted papers will be invited to present their work during the workshop and will be included in the UMAP ‘18 adjunct proceedings by ACM. For each of the accepted submissions, at least one author must attend the workshop and register for at least one day of the UMAP 2018 conference.
List of Topics
Possible questions addressed may include (but are not limited to):
- How can we select evaluation methods that allow to identify blind spots in user experience? What may be criteria to compare and evaluate the suitability of methods for given evaluation objectives and how can we develop those?
- How can we integrate and combine the results of multiple methods to get a comprehensive picture of user experience?
- What are the challenges and limitations of single- or multi-method evaluation of RecSys? How can we overcome such hurdles?
- What are viable user-centric multi-method study designs (guidelines) for evaluating RecSys? What are the lessons learned from successful or unsuccessful user-centric multi-method study designs?
Committees
Program Committee
- Shlomo Berkovsky, CSIRO, Australia
- Peter Brusilovsky, University of Pittsburgh, USA
- Toon De Pessemier, Ghent University/IMEC INTEC-WAVES, Belgium
- Alexander Felfernig, Graz University of Technology, Austria
- Bruce Ferwerda, Jönköping University, Sweden
- Francesco Ricci, Free University of Bozen-Bolzano, Italy
- Mark Graus, Eindhoven University of Technology, The Netherlands
- Lijie Guo, Clemson University, USA
- Dietmar Jannach, Alpen-Adria-Universität, Austria
- Denis Parra, Pontificia Universidad Católica de Chile, Chile
- Martin Pichl, University of Innsbruck, Austria
- Alan Said, The University of Skövde, Sweden
- Olga C. Santos, Universidad Nacional de Educación a Distancia, Spain
- Hanna Schäfer, Technical University of Munich, Germany
- Giovanni Semeraro, University of Bari Aldo Moro, Italy
- Guy Shani, Ben-Gurion University, Israel
- Christoph Trattner, University of Bergen, Norway
- Daricia Wilkinson, Clemson University, USA
- Martijn Willemsen, Eindhoven University of Technology, The Netherlands
- Carrie Wu, Hong Kong Baptist University, Hong Kong
- Markus Zanker, Free University of Bozen-Bolzano, Italy
Organizing committee
- Christine Bauer, Johannes Kepler University Linz, Austria, christine.bauer@jku.at
- Eva Zangerle, University of Innsbruck, Austria, eva.zangerle@uibk.ac.at
- Bart P. Knijnenburg, Clemson University, USA, bartk@clemson.edu
Publication
Accepted papers will be invited to present their work during the workshop and will be included in the UMAP ‘18 adjunct proceedings by ACM. For each of the accepted submissions, at least one author must attend the workshop and register for at least one day of the UMAP 2018 conference.
Venue
The conference will be held in conjunction with UMAP 2018 (User Modelling, Adaptation and Personalization) in Singapore.
Contact
All questions about submissions should be emailed to mume2018-workshop (at) easychair (dot) org.