ABSTRACT. In explainable artificial intelligence (XAI), different types of explanations have been proposed -- feature highlighting, concept-based explanations, as well as explanations by prototypes and by contrastive (near miss) examples. In my talk, I will focus on near-miss explanations which are especially helpful to understand decision boundaries of neighbouring classes. I will show relations of near miss explanations to cognitive science research where it has been shown that structural similarity between a given concept and a to be explained concept has a strong impact on understanding and knowledge acquistion. Likewise, in machine learning, negative examples which are near-misses have been shown to be more efficient than random samples to support convergence of a model to the intended concept. I will present an XAI approach to construct contrastive explanations based on near-miss examples and illustrate it in abstract as well as perceptual relational domains.

ABSTRACT. Integrating human knowledge into neural networks has the potential to improve their robustness and interpretability. We have developed a novel approach to integrate knowledge from ontologies into the structure of a Transformer network which we call ontology pre-training: we train the network to predict membership in ontology classes as a way to embed the structure of the ontology into the network, and subsequently fine-tune the network for the particular prediction task. We apply this approach to a case study in predicting the potential toxicity of a small molecule based on its molecular structure, a challenging task for machine learning in life sciences chemistry. Our approach improves on the state of the art, and moreover has several additional benefits. First, we are able to show that the model learns to focus attention on more meaningful chemical groups when making predictions with ontology pre-training than without, paving a path towards greater robustness and interpretability. Second, the training time is reduced after ontology pre-training, indicating that the model is better placed to learn what matters for toxicity prediction with the ontology pre-training than without. This strategy has general applicability as a neuro-symbolic approach to embed meaningful semantics into neural networks.

ABSTRACT. For single-plant specific weed regulation, robotic systems and agricultural machinery in general have to collect a large amount of temporal and spatial high-resolution sensor data. SEEREP, the Spatio-Temporal-Semantic Environment Representation, can be used to structure and manage such data more efficiently. SEEREP deals with the spatial, temporal and semantic modalities of data simultaneously and provides an efficient query interface for all three modalities that can be combined for high-level analyses. It supports popular robotic sensor data such as images and point clouds, as well as sensor and robot coordinate frames changing over time. This query interface enables high-level planning and reasoning systems as well as other data analysis methods to handle partially unstructured environments that change over time, as for example agricultural environments. But the current methodology of SEEREP cannot store the result of the analysis methods regarding specific objects instances in the world. Especially the results of the anchoring problem which searches for a connection between symbolic and sub-symbolic data cannot be represented nor queried. Thus, we propose a further development of the SEEREP methodology in this paper: For a given object, we link the existing semantic labels in different datasets to a unique common instance, thereby enabling queries for datasets showing this object instance and with this enabling the efficient provision of datasets for object-centric analysis algorithms. Additionally, the results of those algorithms can be stored linked to the instance either by adding facts in a triple-store like manner or by adding further data linked to the instance, like a point, representing the position of the instance. We show the benefits of our anchoring approach in an agricultural setting with the use-case of single-plant specific weed regulation.

ABSTRACT. Recently, the research into language models fine-tuned to follow prompts has made notable advances. These are commonly used in the form of chatbots. One special case of chatbots is that of Task-Oriented Dialogue (TOD) systems that aim to help the user achieve specific tasks using external services. High quality training data for these systems is costly to come by. We thus evaluate if the new prompt-following models can generate annotated synthetic dialogues and if these can be used to train a TOD system. To this end we generate data based on descriptions of a dialogues goal. We train a state-of-the-art TOD system to compare it in a low resource setting with and without synthetic dialogues. The evaluation shows that using prompt-following language models to generate synthetic dialogues could help training better TOD systems.

ABSTRACT. Quantified modal logics have numerous applications in mathematics, computer science, AI, philosophy, and further fields. For AI applications, in particular in knowledge representation and reasoning, modal logic formalisms often employ multiple modalities with different properties and intricate interaction schemes. In this paper, a format for representing such non-trivial modal logic set-ups, and reasoning problems within these logics, is presented. Automated reasoning for this large family of modal logics is then enabled via a translation to classical higher-order logic.

ABSTRACT. A key challenge in machine learning is to design interpretable models that can reduce their inputs to the best subset for making transparent predictions, especially in the clinical domain. In this work, we propose a certifiably optimal feature selection procedure for logistic regression from a mixed-integer conic optimization perspective that can take an auxiliary cost to obtain features into account. Based on an extensive review of the literature, we carefully create a synthetic dataset generator for clinical prognostic model research. This allows us to systematically evaluate different heuristic and optimal cardinality- and budget-constrained feature selection procedures. The analysis shows key limitations of the methods for the low-data regime and when confronted with label noise. Our paper not only provides empirical recommendations for suitable methods and dataset designs, but also paves the way for future research in the area of meta-learning.

ABSTRACT. Qualitative constraint networks are widely used to represent knowledge bases in qualitative spatial and temporal reasoning (QSTR). However, inconsistencies may arise in various application contexts, such as merging data from different sources. In such cases, identifying a consistent constraint network that deviates as little as possible from the over-constrained network is of great interest. This problem of repairing an inconsistent network is a challenging optimization problem, as determining network consistency is already NP-hard for most qualitative constraint languages (also called qualitative calculi). But not all repairs are created equal: Unlike previous work, we consider a practical use case in which facts feature different likelihoods of being true. In this paper, we propose an approach to address this problem by extending qualitative constraint networks with a likelihood factor that can, for example, be derived from the credibility of different data sources. Specifically, we present an encoding for Partial MaxSAT and a Monte Carlo Tree Search (MCTS) implementation for solving qualitative constraint networks with likelihoods optimally and efficiently. Our experimental evaluation demonstrates the effectiveness of our approach, showing that approximate search techniques can offer a promising trade-off between computational budget and optimality.

ABSTRACT. Dialogue systems are an important and very active research area with many practical applications. However, researchers and practitioners new to the field may have difficulty with the categorisation, number and terminology of existing free and commercial systems. Our paper aims to achieve two main objectives. Firstly, based on our structured literature review, we provide a categorisation of dialogue systems according to the objective, modality, domain, architecture, and model, and provide information on the correlations among these categories. Secondly, we summarise and compare frameworks and applications of intelligent virtual assistants, commercial frameworks, research dialogue systems, and large language models according to these categories and provide system recommendations for researchers new to the field.

ABSTRACT. AI models that can recognize and understand the semantics of graphical user interfaces (GUIs) enable a variety of use cases ranging from accessibility to automation. Recent efforts in this domain have pursued the development of a set of foundation models: generic GUI understanding models that can be used off-the-shelf to solve a variety of GUI-related tasks, including ones that they were not trained on. In order to develop such foundation models, meaningful downstream tasks and baselines for GUI-related use cases will be required. In this paper, we present interactive link prediction as a downstream task for GUI understanding models and provide baselines as well as testing tools to effectively and efficiently evaluate predictive GUI understanding models. In interactive link prediction, the task is to predict whether tapping on an element on one screen of a mobile application (source element) navigates the user to a second screen (target screen). If this task is solved sufficiently, it can demonstrate an understanding of the relationship between elements and components across screens and enable various applications in GUI design automation and assistance. To encourage and support research on interactive link prediction, this paper contributes (1) a pre-processed large-scale dataset of links in mobile applications (18,830 links from 5,362 applications) derived from the popular RICO dataset, (2) performance baselines from five heuristic-based and two learning-based GUI understanding models, (3) a small-scale dataset of links in mobile GUI prototypes including ratings from an online study with 36 end-users for out-of-sample testing, and (4) a Figma plugin that can leverage link prediction models to automate and assist mobile GUI prototyping.

ABSTRACT. We present FREDo, a few-shot document-level relation extraction (FSDLRE) benchmark. As opposed to existing benchmarks which are built on sentence-level relation extraction corpora, we argue that document-level corpora provide more realism, particularly regarding none-of-the-above (NOTA) distributions. Therefore, we propose a set of FSDLRE tasks and construct a benchmark based on two existing supervised learning data sets, DocRED and sciERC. We adapt the state-of-the-art sentence-level method MNAV to the document-level and develop it further for improved domain adaptation. We find FSDLRE to be a challenging setting with interesting new characteristics such as the ability to sample NOTA instances from the support set. The data, code, and trained models are available online at https://github.com/nicpopovic/FREDo

ABSTRACT. Pseudo-Labeling is a simple and effective approach to semi-supervised learning. It requires criteria that guide the selection of pseudo-labeled data. The latter have been shown to crucially affect pseudo-labeling’s generalization performance. Several such criteria exist and were proven to work reasonably well in practice. However, their performance often depends on the initial model fit on labeled data. Early overfitting can be propagated to the final model by choosing instances with over- confident but wrong predictions, often called confirmation bias.

In two recent works, we demonstrate that pseudo label selection (PLS) can be naturally embedded into decision theory. This paves the way for BPLS, a Bayesian framework for PLS that mitigates the issue of confirmation bias. At its heart is a novel selection criterion: an analytical approximation of the posterior predictive of pseudo-samples and labeled data. We derive this selection criterion by proving Bayes optimality of this “pseudo posterior predictive”. We empirically assess BPLS for generalized linear, non-parametric generalized additive models, and Bayesian neural networks on simulated and real-world data. When faced with data prone to overfitting and thus a high chance of confirmation bias, BPLS outperforms traditional PLS methods. Moreover, BPLS brings along the benefit of allowing for the incorporation of prior information not only in predicting pseudo-labels but also in selecting pseudo-labeled data.

The decision-theoretic embedding further allows us to render PLS more robust towards the involved modeling assumptions. To achieve this goal, we introduce a multi-objective utility function. We demonstrate that the latter can be constructed to account for different sources of uncertainty and explore three examples: model selection, accumulation of errors and covariate shift.