ABSTRACT. The ‘collective’ part of collective intelligence can feel simultaneously uplifting (“we all contribute!”) and surprising (“I thought you needed to be an expert?”). People often have this same pair of feelings about human-centered design. A partial resolution I (and many of us) offer to these reactions is, “it depends on what you mean by expert. Each of us is an expert in our own lives, which can offers a unique perspective. Also, it’s handy to anchor insights in a concrete setting.” One belief that animates both fields is that we’re not restricted to choosing between expert innovation and collective innovation as they exist today. Experts can take a cue from anthropology and embed themselves in a domain to get more situated insights. And we can create and share knowledge and tools that help a wider group of people innovate.

For the past 6 years, I’ve worked in online education as both a researcher and practitioner, trying to scale the learning that happens in a design studio to the globe. I’ll share insights from my group’s empirical research and software platforms working toward this goal. A traditional design degree (or PhD or MD) provides focused, multi-year training in a discipline. Some of what’s taught is necessarily cumulative, building on what came before. However, online learning materials of many types show that bite-sized learning is often possible and really useful.

How might collective intelligence benefit by weaving focused learning modules (both domain knowledge and process strategies) into an innovation architecture? I’ll share insights and challenges that have emerged from my group’s work — including peer review, scientific discovery, and creativity support—that provide careful process guidance and place focused learning experiences at the point where they’re needed (as opposed to, say, in your ninth grade biology class). This helps collective intelligence participants gain "micro-expertise" and make more creative, practical, and innovate contributions. With such complex sociotechnical systems, a lot of the behavior is emergent, scale-dependent, and importantly different around the globe. This makes moving from the lab to the wild especially important. So along the way I’ll reflect on how the web has dramatically improved our ability to do this Design at Large: creating research that is used around the world for people’s own goals, and improving our knowledge through experiments on these platforms that compare alternatives.

ABSTRACT. Crowdsourcing has become an important phenomenon in firms’ innovation, marketing, and fundraising toolkits and research in this area has been growing rapidly for the last ten years. In fact, in 2013, three well-known journals awarded the Best Paper Award of 2012 to articles dealing with crowdsourcing (AMP, AMR, and JPIM). Most crowdsourcing and crowd work involve participants from the general public—or specific communities—who come together, whether explicitly or implicitly, to help the organizer or seeker solve the problem at hand. This panel has been designed for people interested in the topic to learn more about how crowdsourcing works in different contexts, interact with other researchers already active in different fields, and start new collaborations to help advance the state of the art.

ABSTRACT. Long-form questions and answers on community question answering (CQA) websites and forums such as stackoverflow.com are a valuable resource. Unlike questions and answers in traditional question answering research challenges and datasets, the questions that are asked on such platforms are quite different. Questions have multi-sentence elaborations, can vary from being an embryonic curiosity to a fully fleshed out problem, can contain multiple intents, and can seek advice and opinion beyond facts. Consequently, answers are also different: they are longer and more diverse. For complex and subjective questions, there usually does not exist any authority or notions of correctness, and diverging answers can be helpful to the asker and the community. Motivated by a desire to better understand the quality of long-form questions and answers, we designed an annotation task to collect data about as many commonsense properties of questions and answers, such as question interestingness and answer helpfulness. Just like users of Q&A sites and forums, which are not always experts, our annotation task requires raters to use their commonsense judgment. Our work contributes to the less-explored domain of collecting non-expert's subjective judgments by releasing QUEST, a dataset that contains about 687,000 annotations on 12,096 unique question-answer pairs from 30 different Q&A sites by 3 independent raters. We describe the dataset, iterations of task design, potential implications and usage, as well as limitations and challenges in crowdsourcing commonsense judgments.

ABSTRACT. Collective intelligence projects build the framework for absorption, filtering, summarizing, explaining and comparing of knowledge and ideas, creation of possible solutions for problems and their evaluation, finally – for taking decisions. One of the definitions of collective intelligence says it is the ability to solve problems exceeding the skills of a single person. If in a social structure mutual cooperation is missing, this structure has a limited ability to solve a certain group of problems: in such cases every individual looks for solutions on his own, therefore, neither positive or negative interaction exists. Collective intelligence, however, is emerging when cooperation, competition and mutual observation lead to the new, original solutions of problems or accelerate the process and increase the ability to solve complex problems (Szuba, 2001).

In most cases, this kind of cooperation occurs in business projects, but similar initiatives concerning public affairs are equally important. So far, despite various efforts there are no fully satisfactory results of comparative examination of CI initiatives in the two aforementioned spheres. Therefore, the principal goal of my presentation is to analyze possible research methods which may be helpful in capturing differences in the behavior of CI participants in both these fields.

ABSTRACT. Online Citizen Science applications for data classification, such as Galaxy Zoo and Penguin Watch, are developed to accommodate a broad spectrum of users from various backgrounds and different levels of interest, involvement and expertise. The use of these applications by volunteers neither requires nor assumes any prior scientific knowledge or skills in order to participate, in a way to invite and be open to users regardless of their background. Volunteers of online citizen science applications are a multidimensional user group which can be separated in a variety of different other user groups, with differences in motivations and perceived outcomes from their participation. From a usability point of view, this raises questions in how to attract those groups of participants not necessarily concerned with the scientific aspects of the activities in tasks that are sometimes repetitive or concentrated on very specific micro-tasks. This paper analyses feedback from first-time users of various classification applications (namely: Galaxy Zoo, Penguin Watch, Bat Detective from Zooniverse, and Gendered and Tech magazines from Crowdcrafting), and proposes to establish a set of guidelines for the User Experience (UX) design of online citizen science data classification applications to accommodate a variety of user groups. The guidelines include, amongst others, information and content presentation, controls placement and help buttons, and the addition of various levels of difficulty, in regards to functionality, usability, and look and feel by using User Centered Design (UCD) methodologies.

ABSTRACT. Open source software forms much of our digital infrastructure. It, however, contains vulnerabilities which have been exploited, attracted public attention, and caused large financial damages. This paper proposes a solution to shortcomings in the current economic situation of open source software development. The main idea is to introduce price signals into the peer production of software. This is achieved through a trading market for futures contracts on the status of software issues. Users, who value secure software, gain the possibility to predict outcomes and incentivize work, strengthening collaboration and information sharing in open source software development. The design of such a trading market is discussed and a prototype introduced. The feasibility of the trading market design is validated in a proof-of-concept implementation and simulation.

ABSTRACT. Does Artificial Swarm Intelligence enable human groups to converge on optimal decisions at higher rates than traditional methods for aggregating group input? This study explores the issue rigorously and finds that "human swarms" can be significantly more effective in enabling networked populations to converge on Social Optima as compared to plurality voting, Borda Count rankings, and Condorcet pairwise voting. Across a test set of 100 questions, the traditional voting methods reached socially optimal solutions 60% of the time. The artificial swarming systems converged on socially optimal solutions 82% of the time. This is a highly significant result (p=.001) and suggests that human swarming may be an effective path not only for amplifying the intelligence of human populations, but for enabling human groups with conflicting interests to find solutions that maximize their collective opinions, preferences, interests, and/or welfare.

ABSTRACT. Cyber-physical systems (CPS) are systems that span the physical and cyber-world by linking objects and process from these spaces. In a typical CPS data is collected from the physical world via sensors and computation resources from the cyber-space are used to integrate and analyze this data in order to decide on optimal feedback processes which can be put in place by physical actuators. CPS have started to diffuse into many areas, including mission-critical public transportation, energy services, and industrial production and manufacturing processes. While CPS affect the lives of people that rely on them on a daily basis, they so far only interact with humans as passive consumers. The results of a recent study about adaptation in CPS revealed an emerging trend to add an additional "social"' layer in a CPS architecture to address human and social factors. This trend shows the growing recognition of the importance of the social dimension of such CPS and of the need to evolve them into cyber-physical social systems (CPSS). CPSS consist not only of software and raw sensing and actuating hardware, but are fundamentally grounded in the behaviour of human actors, who both generate data and make informed decisions. As CPSS extend CPS with a social dimension, the question of the relation between CPSS and self-organizational, crowd-powered systems and Collective Intelligence (CI) systems naturally arises. What CI aspects do CPSS exhibit? Can we consider them as an emerging type of CI system or should they be rather perceived as systems of systems that also include a CI system? To answer these and other questions, we have recently performed a systematic mapping study of CPSS. In this paper we report on the study and some of our initial findings.

ABSTRACT. Crowdsourcing is recently used to automate complex tasks when computational systems alone fail. The literature includes several contributions concerning natural language processing, e.g., language translation [Zaidan and Callison-Burch 2011; Minder and Bernstein 2012a; 2012b], also in combination with active learning [Green et al. 2015] and interactive model training [Zacharias et al. 2018]. In this work, we investigate (1) whether a (paid) crowd, that is acquired from a multilingual website’s community, is capable of translating coherent content from English to their mother tongue (we consider Arabic native speakers); and (2) in which cases state-of-the-art machine translation models can compete with human translations for automation in order to reduce task completion times and costs. The envisioned goal is a hybrid machine translation service that incrementally adapts machine translation models to new domains by employing human computation to make machine translation more competitive (see Figure 1). Recently, approaches for domain adoption of neural machine translation systems include filtering of generic corpora based on sentence embeddings of in-domain samples [Wang et al. 2017] have been proposed, as well as the fine-tuning with mixed batches containing domain and outof-domain samples [Chu et al. 2017] and with different regularization methods [Barone et al. 2017]. As a first step towards this goal, we conduct an experiment using a simple two-staged human computation algorithm for translating a subset of the IWSLT parallel corpus including English transcriptions of TED talks and reference translations in Arabic with a specifically acquired crowd. We compare the output with the state-of-the-art machine translation system Google Translate as a baseline.

ABSTRACT. Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair, then sentences that contain that pair are likely to express the relation. In this paper, we use the results of a crowdsourcing relation extraction task to identify two problems with DS data quality: the widely varying degree of false positives across different relations, and the observed causal connection between relations that are not considered by the DS method. The crowdsourcing data aggregation is performed using ambiguity-aware CrowdTruth metrics, that are used to capture and interpret inter-annotator disagreement. We also present preliminary results of using the crowd to enhance DS training data for a relation classification model, without requiring the crowd to annotate the entire set.

ABSTRACT. While digital media offer many opportunities to improve people’s lives, the ability to use the Internet effectively and efficiently is not self-evident even among those who grew up with technologies. Rather, there is considerable variation in Internet skills across the population and these differences tend to be linked to people’s sociodemographic characteristics such as socioeconomic status. Drawing on several data sets, this talk will discuss who is most likely to participate online from joining social media platforms to editing Wikipedia entries. The talk will also offer insights on the potential biases that can stem from relying on certain types of data sets in big data studies.

ABSTRACT. We demonstrate that density and centralization, two conceptually distinct but difficult to separate dimensions of social networks, interact to influence the extent to which group members feel that they share a social identity. The interaction of density and centralization determines the number of role equivalent classes (RECs), sets of individuals who share similar patterns of social network ties. We propose that as the number of RECs increases, group members are less likely to share a social identity. In a laboratory experiment, we manipulated group communication networks to isolate the main and interactive effects of density and centralization. As predicted, the interaction of density and centralization weakened shared social identity, and the number of RECs had the same negative effect on shared social identity. Further, increases in shared social identity led to stronger transactive memory systems and better group performance.

ABSTRACT. Virtual reality (VR) provides a real-time form of embodied interaction that significantly affects team collaboration. Despite such potential, only a few studies have investigated how to utilize VR for a team-level collaborative training that members perform different hands-on tasks in the same physical space. In this paper, the author explores the possibility of how VR environments can be used to improve the quality of collaborative training in a physically limited co-working space, by examining Starbucks cafes as a case. The author suggests a conceptual design of VR environments that support collaborative training, and then discusses ideas of possible interaction strategies related to the suggested design.

ABSTRACT. As teams have become increasingly integrated with machines, including intelligent machines, it is important to understand how machines can be used to enhance team performance. Thus understanding what level of machine intelligence is most useful for enhancing collaboration is an important issue. One way to achieve this is to develop a framework for understanding various types of human-machine teams and compare their effectiveness using a standardized metric. The goal of this paper is to use a typology of the roles machines can play in human-machine teams [Malone 2018] as a basis for designing and testing two specific machine interventions. We test these interventions using a measure of collective intelligence [Woolley et al. 2010; Engel et al. 2015] with 106 teams of Amazon Mechanical Turk workers. Our results indicate that teams’ collective intelligence varied as a result of different ways machines were used to help manage team coordination process. Specifically, a passive tool that supports task assignment contributed to collective intelligence more than a reactive chatbot assistant that offers temporally-based strategies. Our study has implications for conceptualizing and designing machine intelligences for human-machine teaming.

ABSTRACT. Online peer production networks – networks that create artifacts like Wikipedia – are increasingly important for society. But they are strangely organized: They are notable for the absence of the explicit hierarchical command structures and functional departments frequently seen in companies Models from mathematical biology may be useful for analyzing such models, because in biology growth is often limited by competition and changes in the environment. Through evolution, animals have learned how to react to such changes. In particular, social insects have evolved algorithms for coordination. Because of this, dynamic models of task allocation of social insects might provide an alternative framework for studying coordination in peer production. Analyses of Wikipedia articles suggest that edits exhibit burstiness, and this burstiness may function as a signal, attracting further edits.

ABSTRACT. Innovation is often driven by finding analogies across domains, but the growing number of papers makes it difficult to find relevant ideas in a single discipline, let alone analogies in other domains. To provide computational support for finding analogies across domains, we introduce a mixed-initiative system, where humans annotate aspects of documents that denote their background (the high-level problems being addressed), purpose (the specific problems being addressed), mechanism (how they achieved their purpose), and findings (what they learned/achieved), and a computational model constructs a semantic representation from these annotations that can be used to find analogies among the documents. We demonstrate this system finds more analogies amongst research papers than baseline techniques; that annotators and annotations can generalize beyond domain; and that the resulting analogies found are useful to experts. These results demonstrate a novel path towards collective intelligence systems for analogical search.

ABSTRACT. Collective learning is the process of a group transforming a stream of shared or partially shared information into knowledge or insight about the world. The strongest form of success for collective learning is for a group integrate all available information at every point in time, for all members of the group to be able to act upon that knowledge, and for the group to continue rationally collecting new information as needed. One of the key goals of existing work, such as in the study of social learning and cumulative cultural evolution, is identifying which circumstances lead collective learning to succeed or fail. However, one limitation of most existing mathematical work has been a focus on asymptotic analysis, i.e., the long-run behavior of groups of social learners. It is useful to be able to say that a process eventually arrives at the right answer, but such a claim leaves open the question of how the information a group has seen at a given point is related to the behavior of the group at that time. A recent approach provides a way to address these questions. This approach focuses on modeling the population statistics of a group as a Bayesian learning process. This perspective treats the group as Bayesian regardless of whether individual agents are. The key prediction of this model is that population statistics track the totality of information presented to the group at each point in time. Here we present a new instance of this class of models, and test the rational population statistics prediction of this model in a behavioral laboratory experiment for the first time.