Sim4IR 2021: Workshop on Simulation for Information Retrieval Evaluation Online, Canada, July 15, 2021 |
Conference website | https://sim4ir.org/ |
Submission link | https://easychair.org/conferences/?conf=sim4ir |
Abstract registration deadline | May 4, 2021 |
Submission deadline | May 4, 2021 |
Simulation techniques are not foreign to information retrieval. Simulation has been employed, for example, for constructing test collections and for model performance prediction and analysis in a broad array of information access scenarios. The need for simulation has become ever more apparent recently with the emergence of areas where other types of evaluation are infeasible. One such area is conversational information access, where human evaluation is both time and resource intensive at scale. Another example is provided by settings that do not allow sharing of data, e.g., because of privacy constraints, and therefore necessitate the creation of synthetic test collections.
Despite the apparent need, a standardized methodology for performance evaluation via simulation has not yet been developed. The goal of the Sim4IR workshop is to create a forum for researchers and practitioners to promote methodology development and more widespread use of simulation for evaluation by:
- identifying problem settings and application scenarios;
- sharing tools, techniques, and experiences;
- characterizing potentials and limitations; and
- developing a research agenda.
We invite submissions of regular papers, position papers, demonstrators as well as encore talks (featuring already published work).
Topics
Topics of interest include, but are not limited to:
- Problem settings and application scenarios that lend themselves to evaluation via simulation, for example
- Simulation of users and user interactions
- Synthetic test collections
- Characterizing the capabilities and limitations of simulation approaches for various IR problems
- Simulation methods, tools, techniques, and toolkits
- Evaluation of simulation
Submission Guidelines
We welcome submissions of regular papers (4-6 pages) that present original technical, theoretical, or experimental contributions; position papers (2-4 pages) that explore controversial, risk-taking or nascent ideas that have the potential to spark debate and discussion at the workshop; demonstrator papers (max 4 pages) that present first-hand experience with research prototypes or operational systems. Submissions are single-blind and should use a standard double-column ACM SIG proceedings format.
In addition to the paper contributions, we also invite submissions of encore talks, to present work that has already been published in a leading conference or journal, but is relevant to the topics of this workshop. We require a copy of the paper along with a brief (max 200 words) statement of relevance.
The submission deadline for papers is 23:59 (AOE) on May 4, 2021. Encore talk submissions are due at 23:59 (AOE) on May 18, 2021.
Committee
- Krisztian Balog (University of Stavanger, Norway)
- David Maxwell (Delft University of Technology, The Netherlands)
- Paul Thomas (Microsoft, Australia)
- Shuo Zhang (Bloomberg, England)