REA-NDM-FONCSI 2021: REA-NDM-FONCSI JOINT INITIATIVE - BOUNCING FORWARD FROM GLOBAL CRISES AND CHALLENGES
PROGRAM FOR MONDAY, JUNE 21ST
Days:
next day
all days

View: session overviewtalk overview

08:00-08:15 Session 2: Start of the day: Resilience Engineering and Naturalistic Decision Making contributions - Trends and opportunities

Start - Day 1

Location: Main Room
08:00
Resilience Engineering and Desicion Making Contributions, Trends and Opportunities
PRESENTER: Ivonne Herrera

ABSTRACT. This work explores in a systematic way the contributions to the symposium looking where these perspective complement or diverge. Topics includes managemen of trade-offs, learning, monitoring performance, readiness to respond across diverse critical infrastructures.

08:15-09:00 Session 3: “So what should we do?” Challenges and opportunities for Resilience Engineering and Safety-II in practice

Steven Shorrock

Resilience engineering and Safety-II have emerged as credible perspectives and approaches to tackle some of the fundamental problems faced by societies, organisations and teams. Building on complexity science, systems theory, human factors engineering, and other established disciplines, the approaches have both strong theoretical and practical validity. But - as with prior disciplines - both come with particular challenges for practitioners, who act in a milieu that is not always conducive to straightforward practical application. This talk will explore some of these challenges - and opportunities - with experiences from practitioners in aviation, healthcare and WebOps, at different levels, from intergovernmental organisations to team activity.

Location: Main Room
“So what should we do?” Challenges and opportunities for Resilience Engineering and Safety-II in practice

ABSTRACT. Resilience engineering and Safety-II have emerged as credible perspectives and approaches to tackle some of the fundamental problems faced by societies, organisations and teams. Building on complexity science, systems theory, human factors engineering, and other established disciplines, the approaches have both strong theoretical and practical validity. But - as with prior disciplines - both come with particular challenges for practitioners, who act in a milieu that is not always conducive to straightforward practical application. This talk will explore some of these challenges - and opportunities - with experiences from practitioners in aviation, healthcare and WebOps, at different levels, from intergovernmental organisations to team activity.

09:00-09:45 Session 4: Societal resilience, preparedness and reconfiguration

Lightning talk and fishbowl discussion

Location: Main Room
09:00
Rethinking mass casualty distribution – Embedding a resilient hospital selection algorithm into a mass casualty distribution simulation model
PRESENTER: Sheuwen Chuang

ABSTRACT. Considering hospitals’ resilience potential in addition to the shortest-distance and medical-care adequacy policy to distribute mass casualties (MC) is an imperative practice for mass casualty distribution decision making. This study developed a novel hospital selection algorithm composed of driving time from the disaster site to hospital, care adequacy, and mobilization ability to determine the best hospital choice for MC distribution. Next, we developed an MC distribution simulation model embedding the algorithm to generate optimized distribution decisions for various MC incident scenarios. The simulation model was tested by using the Formosa Fun Coast Dust Explosion. Regarding super overload on some responsible emergency hospitals in the FFCDE event, the model with mobilization ability shows a better-balanced distribution of mass casualties to the initial receiving hospitals than without the ability. The study findings can contribute to surge capacity planning and resource assessment of emergency medical services for future disasters.

09:10
How Nursing Care Stations and Care Facilities in Japan Coped with COVID-19 Pandemic: A Survey of Challenges, Preparations, and Responses
PRESENTER: Taro Kanno

ABSTRACT. This study presents a questionnaire survey, designed by a resource-centric framework, about how Japanese home-visit nursing stations and nursing care facilities prepared and coped with the COVID-19 pandemic. We analyzed the answers using qualitative analysis to develop categories of good and typical practices to cope with pandemic situations. Based on the analysis results and discussions, we developed five categories of preparedness and countermeasures for resource management: protection, utilization, stockpiling, procuring, and repairs. Furthermore, we provided the categories with practice examples for human resources, which can be used as a reference when preparing new business continuity plans (BCPs).

09:20
Citizen Audits for Resilience and Preparedness (CARP)
PRESENTER: Simon Gill

ABSTRACT. When facing shocks and stresses we aim for communities to not just bounce-back and survive, but bounce-forward and thrive. The principle in place has been to enable ‘preparedness’ to facilitate effective planning and response and ‘resilience’ to support anticipation, monitoring, response and learning. However, the coronavirus pandemic has exposed deep problems in how we approach resilience and preparedness at a community level.

Deliberative democratic processes have risen in popularity across the world in the form of citizen assemblies. Such assemblies have been successful and have been applied in areas such as abortion and gay marriage legislation (the classic Irish example) but also issues in transport, social policy, political appointments, and electoral reform. These assemblies work in parallel to formal governance mechanisms and act as guidance for decision making that is based on consensus among a large group of citizens drawn at random from the community.

Social audits are a form of citizen participation that looks at government performance and accountability. The objective of a social audit is to monitor and evaluate government performance - making public officials accountable for their actions and decisions. A social audit exercise is a mechanism of oversight: that is, the control that citizens can exert on their government officials to ensure that they act transparently, responsibly, and effectively.

This paper outlines a study of citizen assemblies and social audits and explores an application called a citizen audit for resilience and preparedness (CARP). This evaluates the capabilities and capacities of a community to respond to stresses and shocks, the process overseen and performed by a randomly selected group of citizens. Resilience and preparedness are characteristics that should emerge from a community rather than something imposed on it, a citizen audit could go some way towards building an active relationship between government bodies and its community.

Also discussed is a means of measuring success of the initiative, based on the regular reviewing of individual well-being, which can be run before and after a disruption.

10:00-11:00 Session 5: Emerging technologies and collaborative work - Part A Day 1

Lightning talk and fish bowl discussion

Chair:
Location: Main Room
10:00
A Human-Centred Process Model for Explainable AI

ABSTRACT. With the increase in inscrutable Artificial Intelligence systems being used to support human decision makers, there has been much interest in what it means for these systems to provide ‘explanation’. In this paper, the concern is with applying a simple formalism that can express a minimal set of features that can be used to define an explanation. It is argued that few contemporary AI systems support this minimal set. Advice is provided on how future developments in explainable AI systems could adhere to this minimal set.

10:10
Competition and Conflict Between Frames in Using Machine Learning
PRESENTER: Hebah Bubakr

ABSTRACT. Artificial intelligence and Machine learning (AI/ML) systems promise to improve organisational decision making by avoiding bias because the machine ought to remain unaffected by moods, prejudices or personal opinions when interpreting the data. However, this promise rests on the fact that these tools are independent of the biases of their developers.  The purpose of this paper is to investigate what bias means to developers of AI / ML systems, and how they interpret bias through the result of the system. Our concern is with the relationship between developer - algorithm - data - output.  In this paper, we applied the Data/Frame Model (DFM) to understand what decisions are made by developers of AI/ML.  We propose that developers work with three distinct frames. First, they need to define a suitable dataset that will answer specific questions and also be amenable to analysis. We term this the ‘dataset frame’ and it includes factors such as size, representativeness and coverage, type of questions that could be addressed using these data. Second, having selected datasets, participants then explored different algorithms to test the selected datasets. We term this the ‘Algorithm Frame’. Third, once the algorithm produces answers, then these are reviewed. We term this the ‘Interpretation Frame’ which includes both judgement on the performance of the algorithm (so overlaps with the ‘algorithm frame’), plus judgement on the interpretation of the output to the original questions, and also judgement of the implications of this interpretation.  Our conclusions suggest that developers of AI / ML might take a narrow perspective on ‘bias’ (as a statistical problem rather than a social or ethical problem).  This is not because they are unaware of wider, ethical concerns but because the requirements relating to the management of data and the implementation of algorithms might narrow their focus to technical challenges.  Consequently, biased outcomes can be produced unconsciously because developers are simply not attending to these broader concerns. This suggests that the ‘interpretation frame’ ought to be elaborated to encompass the implications arising from possible interpretations of the algorithms’ output.

10:20
Interdependency Analysis for collaborative robot applications through FRAM analysis
PRESENTER: Arie Adriaensen

ABSTRACT. Although collaborative robots (cobot) applications are increasingly utilised to physically collaborate in realtime with human operators, current cobot safety mainly focuses on a techno-centric perspective in terms of physical separation and managing the net result of kinetic energy in the cobot system. Human Factors (HF) research or its subcategory Cognitive Systems Engineering (CSE) are largely absent in cobot safety. To complement the current techno-centric approach, this project introduces a socio-technical safety and resilience analysis perspective for human-robot interaction by applying systemic safety analysis methods, with a specific focus on observability, predictability, and directability (OPD) requirements for JCS in relation to cobot operations. For the purpose of this project, cobots are broadly defined by any robot system with the potential for foreseeable or intentional physical contact between robots and humans, not necessarily restricted to system operators. The project sets out a generic cobot safety framework that is based on a FRAM analysis of the work system to assess OPD requirements in JCS.

10:30
Development of a Debriefing Tool for Performance Evaluation in Maritime Training Simulations
PRESENTER: Josué França

ABSTRACT. This work presents, based on non-technical skills concepts, BRM (Bridge Resource Management) fundamentals and maritime safety regulations requirements, a proposed design of a debriefing tool for performance evaluation in maritime training simulations. This debriefing tool is formed by a set of questions, comprehending general questions, assessment of the voyage and potential for improvement. It is part of the training simulations and was designed to encourage the discussion regarding the skills, attitudes and interactions that were experienced by the students during the training, aiming to identify and stimulate the development of these and other skills necessary for a productive and safe work on board the vessels.

11:15-12:00 Session 6: Showcase from around the world - Scandinavia

Stian Antonsen; Gesa Praetorius; Ivonne Herrera; Martina Ragosta

A conversation around

  • What happen since last symposium
  • what are your current projects
  • Current questions and possible solutions
  • What industry or other collaborations with end users you have
  • If relevant PhD topics of your students
Discussant:
Location: Main Room
13:00-13:45 Session 7: Organisational resilience

Lightning talk, fishbowl discussion, and workshop

Location: Main Room
13:00
Adaptation in Team Performance for Sensemaking: Are We Ready For The Challenge
PRESENTER: Julie Gore

ABSTRACT. This paper reports an on-going project to explore the challenge of measuring adaptive performance in teams for improved sensemaking. The paper describes an initial rapid review of literature which highlights: (i) the conceptual emphasis of the existing literature, (ii) nascent measures for examining individual adaptive skill and adaptivity in teams, and (iii) a lack of empirical evidence for the development or efficacy of training interventions to improve or develop adaptive team performance. We build on previous understanding about adaptive performance. Nascent measures of individual and team adaptivity are identified and evaluated. Initial experiences of efforts to experimentally measure adaptive team performance for sensemaking will be presented if available.

13:10
Nine Principles for Supporting Distributed Sensemaking
PRESENTER: Simon Attfield

ABSTRACT. The future military operating Environment (FOE) is likely to be one in which operational military units will be less dependent on traditional command and control (C2) structures, operating more as semi-autonomous, self-organising teams. The devolution of C2 functions places a greater emphasis on the need for local peer-to-peer sensemaking and decision making. The work in this paper is a result of a project looking at the idea of sensemaking as a distributed phenomenon in military settings. We discuss this changing context and, as a prelude to the principles, offer a characterisation of distributed sensemaking as a form of distributed cognition which involves the harnessing of disparate information resources. Based on a review of sensemaking and team sensemaking sources from areas including Organisational Studies, Computer Supported Collaborative Work, Naturalistic Decision Making and Human Computer Interaction, we propose nine principles for supporting distributed sensemaking based on specific identified challenges and potential interventions. The nine principles broadly relate to the provision of information, interpretation and communication.

13:20
Environmental, Health & Safety management system: Attributes and Barriers
PRESENTER: Vanessa Bertoni

ABSTRACT. Environmental, Health & Safety integrated management systems are widespread in manufacturing industries, despite limitations regarding the system efficiency and performance, especially concerning technical and behavioral factors. The Functional Safety tool can collaborate with better results in the implementation of an integrated management system. This study aims to identify attributes, and barriers in the studies of integrating functional safety and Environmental, Health, & Safety with sustainability and management systems. A systematic literature review was carried out in three phases: i) selection of studies on the topic, ii) bibliometric analysis, and iii) content analysis. This review, done through 702 searched documents from the Scopus, Science Direct, and Web of Science databases, resulted in 84 documents after exclusion criteria and the inclusion of 16 documents via the Snowball method. As the main deliverables, this study listed 30 attributes for the implementation of an integrated management system. Moreover, 25 raised barriers to implementing a system that integrates environmental, health & safety, and functional safety can be used to support practitioners in the implementation of manufactures. Finally, as the main contribution, there is a survey of managerial implications on how to implement this integrated management system in manufacturing industries.

14:00-15:00 Session 8A: Lead talks from implementation - sharing experiences
Location: Main Room
14:00
Implementing Safety Differently at Lewis Tree Service
PRESENTER: Asher Balkin

ABSTRACT. Surprise is ubiquitous yet seldom acknowledged nor prepared for in ordinary work situations. We explore the practical application of fundamental Resilience Engineering premises “surprise will happen” and “work is variable” to line clearance work.

“I was felling large, dead locus tree surrounded by slippery, moss covered ground. The tree started to fall in intended direction, but the top broke out and was coming right for me. I fell as I scrambled to get out of the way. I barely escaped on my hands and knees…”

At Lewis Tree Service, we perform the high-risk work of removing trees near power lines including trees on downed power lines after storms.

We discuss how safety programs must change as the attributes of work become more variable. Those who are implementing “safety differently”, “safety II”, or Resilience Engineering will gain practical take-aways that could be applied in most settings. • We are mining close calls. Proactively soliciting stories of serious close calls that happened in the past through targeted questioning. • We are writing and telling stories to build collective memory. In collaboration with a storytelling coach, we crafted a method that includes interviewing people after an event to support developing and sharing people-centered stories. • We create the space to practice through “drill day”. Safety designs drills that are about practicing on the “blue line”, enabling successful adapting to real work variability. • We perform After Action Reviews which enable us to better understand the shape of surprise. • Leaders changed safety conversations by asking different questions to develop a learning culture (help notice tradeoff decisions, take system view, reinforce new view safety, demonstrate caring and empathy, show appreciation for workers, recognize humble, engaged leadership, and prompt new thinking/actions). • We explicitly explore role of uncertainty in decision making and include a discussion of mechanisms to mitigate its effects. We invented an “uncertainty gauge” to support conversations that explore a worker’s ability to control how the tree or limb will come down and their different perceptions of uncertainty. “The things that kill workers seem to be the things that are the most difficult to control.” “Risk is in the eye of the beholder. Risk may be defined as the degree to which a worker is facing uncertainty.” Conklin • We conclude with insights from the past two years of S-II and RE implementation which will be useful to other safety practitioners and researchers.

14:30
Combining insights from biology, ecology and human systems

ABSTRACT. Highlights from recent work

14:00-15:00 Session 8B: Workshop: Defining an Integrative Framework of Sensemaking and Sustainability for Building Organizational and Community Resilience

Interactive workshop

Location: Workshops
14:00
Defining an Integrative Framework of Sensemaking and Sustainability for Building Organizational and Community Resilience
PRESENTER: Mark Chubb

ABSTRACT. Organizations and communities are embracing the principles and practices of sustainability and resilience in response to wicked issues such as climate change, income inequality, technological disruption, economic displacement, and emerging infectious diseases. Many of them find stakeholder aspirations and interests conflict even when they are well aligned on the need for decisive action to meet these challenges.

This workshop establishes a framework for and seeks engagement from interested experts to further develop an approach to sensemaking and sustainability to guide public participation processes among diverse stakeholders to facilitate constructive dialogue. The workshop seeks to establish specific criteria for each dimension of the framework to help users define individual and organizational roles and responsibilities in developing community resilience.

The proposed framework builds on the concepts of public value and co-production of public goods. It consists of five key elements of resilience to guide sustainable community engagement for organizational and community resilience. Workshop participants will employ the framework to examine diverse stakeholder interests and activities related to key resilience challenges facing communities.

• Open – How can individuals and organizations share data and insights with one another without compromising data integrity, personal privacy, or competitive advantage?

• Varied – What critical infrastructure pathways, key resources, and information sources support confidence, reliability, and redundancy under crisis conditions?

• Simple – What systems or methods are available or required to manage complexity, support situational awareness, and promote a common operating picture?

• Local – What resources exist within the community to sustain life and maintain critical functions during extreme events? How will individuals and organizations fill key gaps?

• Connected – How are individuals and organizations integrated within regional, national, and international networks and cooperative agreements that can provide critical insights or key resources before, during or after a crisis?

This workshop is an opportunity for experienced emergency management practitioners, public policy analysts, decision- support researchers, and crisis communication experts among others to expand and develop the proposed framework for public participation and community engagement. Participants in this workshop will contribute to an enhanced understanding of this framework, its application to various policy and practice challenges, and how it can be developed and applied as a tool for community engagement, comprehensive emergency planning, and policy analysis. Participants will be invited to collaborate in further development and deployment of this framework as a community engagement strategy.

15:15-16:00 Session 9: Societal and community resilience

Lightning talk, fish bowl discussion, and panel

Location: Main Room
15:15
Understanding the applicability of resilience practices: proposed approach in the investigation of solutions to support societal resilience
PRESENTER: Matthieu Branlat

ABSTRACT. To improve its capacities, an organisation might want to learn from tools and practices successful elsewhere. Focused on societal resilience and disaster management, project ENGAGE proposes a path to understand the conditions for applying such practices in another context. The approach is based on acknowledging that effective solutions will necessarily be related to how societal resilience manifests in a given location. We ask, for example, what makes certain solutions viable in one local context but not (or only partially so) in another? To make the link between the solutions and societal resilience, we first consider what elements of the local context are factors that support or hinder societal resilience. We then distinguish between aspects of society on which one can act upon directly (the target aspects) from aspects that are part of the local context and cannot easily be changed (the contextual aspects). This distinction between target and contextual dimensions allows us to state the underlying idea of the approach: a solution aims to have an impact on a population, but its impact is influenced (positively or negatively) by the context of use. The project aims to improve knowledge by identifying and relating such elements and dynamics, and to improve practices by providing enriched descriptions of solutions that authorities and emergency organizations can implement to better support societal resilience in local contexts. In the end, a solution involving a population in disaster management might be successful in one location but only be implementable elsewhere under some conditions. The overall ideas discussed, which underly the work in project ENGAGE, provide a path to understand why such a solution might be successful and how sensitive it is to its context of use.

15:25
A tale of two “cities”: comparing the State of Ohio and The Ohio State University’s ability to enact massive change
PRESENTER: Michael Rayo

ABSTRACT. We will share our front-row-seat perspective of how the State of Ohio (population 11.7 million) and the community of The Ohio State University (population of ~100,000) generated new capabilities to meet the demands of the pandemic. It reveals the barriers and facilitators for change (i.e., adaptation) and generating new capabilities amid a climate in which the need for change was urgent and unambiguous. Will will discuss how these organizations' adaptive capacity was shaped by how and when these organizations accepted extra external resources, their willingness to engage with their broader communities, and the continuous, inexorable pressures both faced to dissolve these new, "extra" capabilities.

16:00-20:00 Session 10: Emergent Talents Programme

Emergent Talents:

  • Natalie Sanford
  • Shanee Honig
  • Nichole Pereira
  • Karl Hybinette
  • Lida David
  • Claudia Guerra Disconzi
  • Bruna Gayer
  • Ann-Therese Hedqvist
  • Atif Ashraf
  • Arie Adriaensen
  • Clément Cornière 
  • Elleke Ketelaars 
  • Floris Van den Oever 
  • Gertrude Spiteri 
  • David Grimm 
  • Matt Woodward 
  • Myke Cohen 
  • Sahinya Susindar 
  • Samuel Muir 
  • Catherine Stevens
16:15-17:00 Session 11: Workshop: From critical infrastructure to society as a whole: Can Naturalistic Decision Making and Resilience Engineering scale up?

Interactive workshop

Discussant:
Location: Workshops
16:15
From critical infrastructure to society as a whole: Can Naturalistic Decision Making and Resilience Engineering scale up?
PRESENTER: Matthieu Branlat

ABSTRACT. In light of events such as the Fukushima accident or the current pandemics, disaster management has been shifting in recent years from a “critical infrastructure” to a “societal” focus. In RE and NDM, we are used to deal with the former, the more professional or formal side of management of disruptive events, through our concepts and methods: we investigate, learn from and try to better support expert behaviour or resilient organisations/HROs. The potential and limits of applying our analytical tools to informal social actors remains less studied. The proposed panel aims to address questions such as the following: - Are we, as fields of research, equipped to look at the larger societal context and dynamics? - How can we account for non-professional individuals and groups actions? - Should we, as a field, be involved in this larger scale, or should we only ensure we can interact with other fields who look more broadly? In the latter case, what can we bring to such discussions? - What additional competences do we need to bring in or what fields do we need to be inspired by in order to be able to ask better questions on societal scales?

17:15-19:00 Session 12A: Resilient healthcare and NDM

Panel, lightning talk, and fish bowl discussion: administrated by USA node

Location: Main Room
17:15
Resilient healthcare during the COVID-19 pandemic: a case study in a Brazilian hospital

ABSTRACT. The COVID-19 pandemic has challenged the resilience of healthcare services in several countries. This exploratory study presents the lessons learned by a leading private hospital in Southern Brazil, using the lens of resilience engineering.

17:25
Learning to Adapt, as we Adapt: A Hospital’s Experience of Learning During the Pandemic
PRESENTER: Sudeep Hegde

ABSTRACT. The COVID-19 pandemic is a case of fundamental surprise, a crisis that emerged suddenly, without warning, and which had no preceding pattern. Like most organizations, hospitals were caught off-guard with few to no pre-existing protocols, norms, standards or guidelines in place. Furthermore, public health and institutional leadership did not have enough available information to make informed decisions in the beginning. Hospitals’ management and operations have been challenged with elevated patient volumes; infection prevention; lack of resources; decisions about mode of care-delivery. Hospitals have been hit with a paradox: while it’s increasingly important to safely deliver care, it’s more difficult to do so. While there have been multiple challenges, frontline and management staff have devised numerous solutions to harrowing working conditions. Recent literature has surfaced on how various organizations have coped with the COVID-19 pandemic finding solutions from the ground up through surge planning, patient monitoring dashboards, telehealth and remote medical services, and workforce protection, strengthening value-based payment. In all of the examples, adaptation at multiple institutional levels has been key. Despite declining cases of COVID-19 and immunizations, infection control measures will continue for the foreseeable future. There is a need and an opportunity to learn about what has worked during this past year, and to ensure preparedness as hospitals transition beyond the current crisis. Previous work exists on the use of knowledge-elicitation techniques to proactively learn about successful adaptative practices. In particular, the Resilience Engineering Tool to Improve Patient Safety (RETIPS) has been used on a pilot-basis to learn about adaptive capabilities of anesthesia residents3. RETIPS is a semi-structured self-reporting tool designed to elicit narratives of adaptation in everyday clinical work. The current study tailored RETIPS to the case of a radiology unit to learn from its operational staff about what went well as they adapted to the pandemic. Fifty-eight reports were received between July 29th and October 12th 2020 with over 90% received during the month of August. Participants’ roles included radiology leadership (21%), technologist (19%), nursing (13%), radiologists (10%), and sedation/anesthesia (10%). Participants described examples of lessons-learned, and indicated the success factors, challenges, and resources pertinent to their examples. The thematic analysis of the descriptions of their examples of an ‘adjustment or coping strategy or preventive practice’ identified several prominent themes. The lessons from this experience need to be learned in order to sustain these capabilities going forward, even beyond the pandemic itself.

17:35
The Workload Implications of Resilient Healthcare: The Role of Social Interactions

ABSTRACT. The extra effort of clinicians´ to provide care to their patients, which is a manifestation of resilience, usually goes unnoticed due to successful outcomes that occur most of the time. Therefore, clinicians’ self-sacrifice and overreliance on resilience become “normal work”. This study investigates the workload implications of resilient healthcare. The study setting was the radiology department of a major teaching hospital. The centrality of each clinician in four ability-based social networks (i.e., the resilience abilities of monitoring, responding, anticipating, and learning) was considered a proxy of each actor’s contribution to the overall system resilience. As such, a resilience score was calculated for each actor in each ability-based network, combining five indicators theoretically connected to the actor’s resilience: in-degree, closeness, and betweenness, which are derived from social network analysis, and availability and reliability, which are non-network attributes assessed through Likert-style questions. In turn, the individual workload was assessed based on the NASA-TLX questionnaire, which produces indicators related to six dimensions of workload (in addition to an overall score): mental, physical, temporal, performance, effort, and frustration. Both questionnaires – social network analysis and NASA-TLX – were answered by 155 out of the 220 staff of the radiology department. Follow-up interviews with 10 respondents were conducted to understand the influence of contextual factors on the actors´ workload and their resilience score based on network centrality. Preliminary data analysis indicates that the overall workload of the actors with the higher and lower resilience scores is similar, although there are significant differences in the workload dimensions (e.g., actors with higher resilience scores are subject to higher temporal and mental demand in comparison to those with lower resilience scores). As another finding, mental demand was the major workload dimension significantly correlated (95% confidence interval) to the resilience score. Furthermore, actors that take the initiative to help their coworkers without being requested had a higher overall workload than those that are more passive – the frequency at which actors take the initiative to help others was assessed by a specific question in the social network analysis survey. Overall, our findings indicated that social interactions related to the four resilience abilities mattered to the workload of the professionals surveyed. The adopted research method is expected to be replicable to other settings. It might be a new approach for the analysis of the human costs of resilient performance. The workload is proposed as a proxy of that human cost.

 

17:45
Running out of resilience: coping with the COVID-19 pandemic in Brazilian intensive care units

ABSTRACT. Intensive care units (ICUs) have been the most visible facet of healthcare services affected by the COVID-19 pandemic. This paper presents an exploratory investigation of how ICUs in Brazil have coped with the pandemic. A resilience engineering perspective is adopted, using a mixed-method research design. The quantitative stage comprised an exploratory survey for assessing the uptake of five work system guidelines that are logically related to resilience. The guidelines are: provide slack resources; give visibility to processes and outcomes; support diversity of perspectives when making decisions; monitor and understand the gap between work-as-imagined and work-as-done; and monitor unintended consequences of improvements and changes. These guidelines were described as statements in the survey, accompanied by ICU examples. There was a sliding bar with two endpoints: fully disagree (zero) and fully agree (100). For example, one of the statements related to the guideline on slack was as follows: there are extra or standby human resources that can be quickly deployed, and these are available in sufficient quantity to cope with unforeseen events. There were 33 valid responses, all from different ICUs. All respondents were invited to be interviewed and seven agreed with that. These interviews shed light on the rationale for the survey responses. The qualitative stage followed with interviews with two public health officials and non-participant observations of the meetings of a municipal COVID-19 crisis management committee in a capital city. Joint data analysis from the quantitative and qualitative stages supported the identification of five lessons learned, each corresponding to one of the aforementioned guidelines. Results indicated the guideline on slack resources as the most relevant as it addressed the key pandemic issue of matching capacity to demand. The provision of sufficient and experienced staff was found to be more challenging than the provision of facilities and supplies. Running out of resilience was part of everyday work in the ICUs, as evidenced by: (i) the lack of effective treatments, which means that despite all efforts the mortality rates were very high; (ii) exhausted professionals who could not cope with the stressful working conditions; and (iii) chronic and acute mismatches between capacity and demand, which implied that many patients received sub-standard care. ICU resilience was inextricably dependent on the resilience of society at large. An integrated approach to resilient healthcare is necessary, encompassing the micro, meso, and macro levels, besides accounting for the broader societal resilience.

17:15-18:00 Session 12B: Workshop - How has the COVID-19 pandemic changed how child welfare workers assess the safety of children? How has the pandemic changed other critical practices and decisions in child welfare?

Interactive workshop

(US UK)

Location: Workshops
17:15
Safety in Child Welfare: Challenges, Innovations, and Opportunities
PRESENTER: Emily Newsome

ABSTRACT. Child welfare organizations are tasked with maintaining the safety of children under complex circumstances. As such, they must be highly adaptable and resilient, while also supporting child welfare practitioners who make decisions that affect the lives of children and families. This panel will bring together experienced practitioners, researchers, and thought leaders in child welfare to explore decision making in child welfare. Topics will include: challenges of safety assessments and decisions in child welfare, how practitioners and systems have responded to the COVID-19 pandemic, and innovations for systems in which these decisions are made. Panellist backgrounds include experts in social work practice; managers of public child welfare systems in the U.S.; and researchers exploring innovative ideas for helping shift organizational cultures toward a culture of learning and promoting expertise.

07:30 Session 1: Teaser videos

This session contains all the teaser videos that were uploaded by contributors

They can be viewed here at any time during the four days, and also used during the sessions by the authors

Location: Workshops
What is Resilience Engineering
What is Naturalistic Decision Making
PRESENTER: Gary Klein
Interdependency Analysis for collaborative robot applications through FRAM analysis
PRESENTER: Arie Adriaensen

ABSTRACT. Although collaborative robots (cobot) applications are increasingly utilised to physically collaborate in realtime with human operators, current cobot safety mainly focuses on a techno-centric perspective in terms of physical separation and managing the net result of kinetic energy in the cobot system. Human Factors (HF) research or its subcategory Cognitive Systems Engineering (CSE) are largely absent in cobot safety. To complement the current techno-centric approach, this project introduces a socio-technical safety and resilience analysis perspective for human-robot interaction by applying systemic safety analysis methods, with a specific focus on observability, predictability, and directability (OPD) requirements for JCS in relation to cobot operations. For the purpose of this project, cobots are broadly defined by any robot system with the potential for foreseeable or intentional physical contact between robots and humans, not necessarily restricted to system operators. The project sets out a generic cobot safety framework that is based on a FRAM analysis of the work system to assess OPD requirements in JCS.

Uncovering resilience abilities in maintenance teams for buildings with Functional Resonance Analysis Method
PRESENTER: Ivenio De Souza

ABSTRACT. Building maintenance comprises a set of complex socio-technical activities, with many interacting agents. Traditional tools for safety management make the results of such evaluations distant from real situations. Resilience Engineering (RE) argues that improvements in safety performance concern the ability to recognize and adapt to handle unanticipated perturbations. This study aims at increasing the understanding of everyday building maintenance activities for air conditioning systems to uncover resilience abilities developed by the maintenance team to deal with everyday challenges. The Functional Resonance Analysis Method (FRAM) has been applied to model these activities. The data collection comprised interviews and observations. As a contribution, this study outlined the potential of the FRAM model as the basis of in-depth and systematic analysis of daily performance, highlighting resilience abilities aligned with RE principles.

Responses to overload as foundation for a resilient project and organizational health metric
PRESENTER: Marisa Bigelow

ABSTRACT. It can be difficult to understand changes to team performance in the moment and at a larger organizational scale, especially when adapting to unique evolving pressures and opportunities. To translate theory into practice and evaluate resilience at a project level, we are measuring four responses to saturation on a regular cadence and creating new project monitoring representations for communication and comparison at all organizational levels. The ongoing work exemplifies the convergence of contextualized decision-making with a resilience engineering framework for analysis.

Environmental, Health & Safety management system: Attributes and Barriers
PRESENTER: Vanessa Bertoni

ABSTRACT. Environmental, Health & Safety integrated management systems are widespread in manufacturing industries, despite limitations regarding the system efficiency and performance, especially concerning technical and behavioral factors. The Functional Safety tool can collaborate with better results in the implementation of an integrated management system. This study aims to identify attributes, and barriers in the studies of integrating functional safety and Environmental, Health, & Safety with sustainability and management systems. A systematic literature review was carried out in three phases: i) selection of studies on the topic, ii) bibliometric analysis, and iii) content analysis. This review, done through 702 searched documents from the Scopus, Science Direct, and Web of Science databases, resulted in 84 documents after exclusion criteria and the inclusion of 16 documents via the Snowball method. As the main deliverables, this study listed 30 attributes for the implementation of an integrated management system. Moreover, 25 raised barriers to implementing a system that integrates environmental, health & safety, and functional safety can be used to support practitioners in the implementation of manufactures. Finally, as the main contribution, there is a survey of managerial implications on how to implement this integrated management system in manufacturing industries.

Applying the Principles of Resilience Engineering to Small and Medium Sized Businesses

ABSTRACT. Small to medium sized businesses are a significant contributor to the global economy. However, they are widely accepted to be more fragile than larger businesses, especially in terms of financial security and the coronavirus pandemic has exposed a need to build resilient performance of SMEs. This presentation explores how SMEs can move beyond the traditional, minimal approach to risk management, utilising the principles and techniques of resilience engineering to bounce forward from future shocks.

The Impact of Learning from Everyday Work and Decision Making on Aviation Maintenance

ABSTRACT. Aviation maintenance engineers are routinely required to work around challenges in the maintenance environment to deliver aircraft safely and efficiently. A process called Human Hazard Analysis is being used by the offshore helicopter community to explore the gaps between Work-As-Done on aircraft in-service with the Work-As-Imagined and Work-As-Prescribed during the design process. This presentation outlines the results of these workshops so far but also their limitations. Recent research into maintenance engineer decision-making has the potential to address some of these weaknesses. The results of these studies is outlined with corresponding recommendations on how this might be capitalised by the industry.

Surprises and adaptations in software development projects
PRESENTER: Michael Smith

ABSTRACT. The design and development of software systems is a complex process. Even a relatively small project may involve distributed teams of several different types of professions, dozens of different tools, and hundreds of different documents. Prescriptive system development processes, of which there are many, are focused on how to proactively reduce potential variation. By contrast, there has been little work exploring how to adapt to unanticipated variation - how to cope with surprises - in software project management. Surprises in the systems development process can impact the cost of development, the scope or quality of the product, the time of delivery, and the relationships between stakeholders. They can have positive or negative impacts, opening new opportunities or hindering progress forward. Responding to a surprise constitutes a test of the ability of the project team and parent organization to recognize and adapt. The sensitivity to meaningful variation and potential risks, the flexibility of work processes and resource allocation, and the expertise of the responsible practitioners all play important roles in the adaptation process. In this session we will begin by discussing patterns in problem detection (Klein), anomaly response (Woods & Hollnagel), and organizational surprises (Cunha, Clegg & Kamoche). We will share some examples from our own work at Mile Two and analyze them. The aim of the analysis will be to identify macrocognitive functions related to coping with surprises in this context, and how project or organizational resilience is evidenced in these examples. To make this an interactive session, at the start of the conference we will ask participants to share electronically their own accounts of surprises in systems development projects. We will include these in our analyses - ahead of time for earlier contributions, and in real time for late or concurrent contributions. We will use an online whiteboard or similar tool to actively show the real-time analysis.

 

Please submit your examples of surprises in software development projects via this form: https://forms.gle/oaZGRgfqmyQuKNb2A

Resilient Analysis Grid: A quantitative approach of healthcare provider’s perspective during COVID-19 pandemic
PRESENTER: Vanessa Bertoni

ABSTRACT. There are growing concerns about how healthcare systems can adapt in times of crisis. The overarching challenge lies in how resilience engineering could be used to analyze and improve the performance of healthcare systems concerning the Covid-19. This study aims to describe the relationship between resilience potentials and health and safety aspects and its consequences on quality and resilience in healthcare systems. This study has a quantitative methodological approach using a survey with the Resilience Analysis Grid as an approach to analyzing organizational resilience based on the idea that four potentials (responding, monitoring, learning, and anticipating) influence patient safety, occupational health and safety, and resilient system performance. As for this study results, anticipating and monitoring, overall resilience, and occupational health and safety are the variables that need more attention in healthcare systems. This study has a dyad of contributions, as a practice, evaluate the resilience in a pandemic time, and as theoretical, the identification of the importance of resilience four potential connections in healthcare systems

Development of a Debriefing Tool for Performance Evaluation in Maritime Training Simulations
PRESENTER: Josué França

ABSTRACT. This work presents, based on non-technical skills concepts, BRM (Bridge Resource Management) fundamentals and maritime safety regulations requirements, a proposed design of a debriefing tool for performance evaluation in maritime training simulations. This debriefing tool is formed by a set of questions, comprehending general questions, assessment of the voyage and potential for improvement. It is part of the training simulations and was designed to encourage the discussion regarding the skills, attitudes and interactions that were experienced by the students during the training, aiming to identify and stimulate the development of these and other skills necessary for a productive and safe work on board the vessels.

Team Decision Making in Designing Software for Joint Cognitive Systems
PRESENTER: John Flach

ABSTRACT. This paper describes how Mile Two has attempted to integrate CSE with other disciplines in order to design software to support decision making in joint cognitive systems. The paper discusses three aspects of our approach: 1) Recognizing the value and limitation of different design perspectives; 2) Cross-training; and 3) Involving all disciplines at all stage of the design process - co-creation. The paper will discuss the successes and difficulties associated with implementing a total team process across different stages of design and development.

Adaptation in Team Performance for Sensemaking: Are We Ready For The Challenge
PRESENTER: Julie Gore

ABSTRACT. This paper reports an on-going project to explore the challenge of measuring adaptive performance in teams for improved sensemaking. The paper describes an initial rapid review of literature which highlights: (i) the conceptual emphasis of the existing literature, (ii) nascent measures for examining individual adaptive skill and adaptivity in teams, and (iii) a lack of empirical evidence for the development or efficacy of training interventions to improve or develop adaptive team performance. We build on previous understanding about adaptive performance. Nascent measures of individual and team adaptivity are identified and evaluated. Initial experiences of efforts to experimentally measure adaptive team performance for sensemaking will be presented if available.

Citizen Audits for Resilience and Preparedness (CARP)
PRESENTER: Simon Gill

ABSTRACT. When facing shocks and stresses we aim for communities to not just bounce-back and survive, but bounce-forward and thrive. The principle in place has been to enable ‘preparedness’ to facilitate effective planning and response and ‘resilience’ to support anticipation, monitoring, response and learning. However, the coronavirus pandemic has exposed deep problems in how we approach resilience and preparedness at a community level.

Deliberative democratic processes have risen in popularity across the world in the form of citizen assemblies. Such assemblies have been successful and have been applied in areas such as abortion and gay marriage legislation (the classic Irish example) but also issues in transport, social policy, political appointments, and electoral reform. These assemblies work in parallel to formal governance mechanisms and act as guidance for decision making that is based on consensus among a large group of citizens drawn at random from the community.

Social audits are a form of citizen participation that looks at government performance and accountability. The objective of a social audit is to monitor and evaluate government performance - making public officials accountable for their actions and decisions. A social audit exercise is a mechanism of oversight: that is, the control that citizens can exert on their government officials to ensure that they act transparently, responsibly, and effectively.

This paper outlines a study of citizen assemblies and social audits and explores an application called a citizen audit for resilience and preparedness (CARP). This evaluates the capabilities and capacities of a community to respond to stresses and shocks, the process overseen and performed by a randomly selected group of citizens. Resilience and preparedness are characteristics that should emerge from a community rather than something imposed on it, a citizen audit could go some way towards building an active relationship between government bodies and its community.

Also discussed is a means of measuring success of the initiative, based on the regular reviewing of individual well-being, which can be run before and after a disruption.

The Power of Curiosity

ABSTRACT. Definition: Curiosity is the motivation to bridge a gap. The literature suggests this can be an information gap or a mental model gap. I think a better formulation is to consider a plausibility gap. Features of curiosity: Curiosity is superficial and transient and intense and impulsive. Superficial — it can arise, change focus, and end abruptly. (Waiting in a checkout line in a supermarket, we may become intensely curious about the latest news regarding a movie star’s marital woes — this curiosity disappears as soon as we step forward and away from the tabloids.) Transient — it reflects attention, and when attention captured by something else, curiosity disappears. Intense — people will work to satisfy their curiosity despite being hungry or thirsty Impulsive — something can evoke curiosity and stop us in our tracks. We may be tempted to buy that tabloid in the supermarket as long as we are looking at it Blocking curiosity: The research literature identifies common practices that can interfere with curiosity. Many of these inhibitors are initiated by instructors. Ask closed questions — calling for Yes/No answers. Focus on procedures. Emphasize memorization. Claim that there are right answers. Quickly correct students’ mistakes. Provide After-Action reviews that contain mostly statements and few questions. Focus on what the trainee is doing, and not why. Overwhelm students with details. Introduce too much complexity too early. (Too much confusion/complexity/ambiguity inhibits curiosity. The student stops being curious and instead tries to memorize all the material. Further, presenting too many open variables and loose ends can inhibit curiosity. Additional instructional behaviors that can inhibit curiosity include the following: Discourage questions from the students/trainees — make it clear that the class needs to get through all the material. Discourage class discussions. Provide complete explanations, leaving no space for student to engage in self-explanation. Provide explanations that make students feel stupid — use ridicule to make students afraid to appear dumb. Promoting curiosity. The collection of common inhibitors is fairly discouraging. However, the literature also identifies a number of factors that instructors and trainers can use to harness curiosity. These include the following: Pose a question; present contrasts and counterfactuals/hypotheticals; provide a sequence with an unknown resolution [e.g., who will win an election, or a race?). In addition, teachers can try to switch out of a mindset of criticizing students and instead wonder what caused the student to make a mistake.

Resilient healthcare during the COVID-19 pandemic: a case study in a Brazilian hospital

ABSTRACT. The COVID-19 pandemic has challenged the resilience of healthcare services in several countries. This exploratory study presents the lessons learned by a leading private hospital in Southern Brazil, using the lens of resilience engineering.

Modelling driver decisions to improve port traffic management during critical situations
PRESENTER: Ross Phillips

ABSTRACT. According to Flach & Voorhorst (2020) effective management of real-world situations can be achieved by treating human decisions according to meaning processing approaches to cognition. We describe a System Theoretic Process Analysis (STPA) of traffic management in a busy port area of the Netherlands. Reflecting on the meaning processing approach to cognition, we then generate and test a new set of prompts that the STPA analyst can use as they generate scenarios to explain why human actors would carry out unsafe control actions. The new prompts help the analyst consider the human controller as an individual embedded and in a dynamic ecology, with perceptions and actions emerging from interactions with that ecology. They help consider people as perceivers of affordances rather than whole processes; and as aware of the meaning of actions for several valued goals. On applying the new prompts to the port traffic management case, new insight was generated concerning how truck drivers will act to perceive (and not just perceive to act), seek out workarounds in the face of terminal closures, and attend to information that helps them decide whether diverting or ignoring diversions is most likely to lead to punctual delivery. While extending STPA analysis to account for the meaning processing approach, demands greater resource and understanding, there will be occasions when a less superficial approach to human control is desired by STPA practitioners.

Defining an Integrative Framework of Sensemaking and Sustainability for Building Organizational and Community Resilience
PRESENTER: Mark Chubb

ABSTRACT. Organizations and communities are embracing the principles and practices of sustainability and resilience in response to wicked issues such as climate change, income inequality, technological disruption, economic displacement, and emerging infectious diseases. Many of them find stakeholder aspirations and interests conflict even when they are well aligned on the need for decisive action to meet these challenges.

This workshop establishes a framework for and seeks engagement from interested experts to further develop an approach to sensemaking and sustainability to guide public participation processes among diverse stakeholders to facilitate constructive dialogue. The workshop seeks to establish specific criteria for each dimension of the framework to help users define individual and organizational roles and responsibilities in developing community resilience.

The proposed framework builds on the concepts of public value and co-production of public goods. It consists of five key elements of resilience to guide sustainable community engagement for organizational and community resilience. Workshop participants will employ the framework to examine diverse stakeholder interests and activities related to key resilience challenges facing communities.

• Open – How can individuals and organizations share data and insights with one another without compromising data integrity, personal privacy, or competitive advantage?

• Varied – What critical infrastructure pathways, key resources, and information sources support confidence, reliability, and redundancy under crisis conditions?

• Simple – What systems or methods are available or required to manage complexity, support situational awareness, and promote a common operating picture?

• Local – What resources exist within the community to sustain life and maintain critical functions during extreme events? How will individuals and organizations fill key gaps?

• Connected – How are individuals and organizations integrated within regional, national, and international networks and cooperative agreements that can provide critical insights or key resources before, during or after a crisis?

This workshop is an opportunity for experienced emergency management practitioners, public policy analysts, decision- support researchers, and crisis communication experts among others to expand and develop the proposed framework for public participation and community engagement. Participants in this workshop will contribute to an enhanced understanding of this framework, its application to various policy and practice challenges, and how it can be developed and applied as a tool for community engagement, comprehensive emergency planning, and policy analysis. Participants will be invited to collaborate in further development and deployment of this framework as a community engagement strategy.

Operational Resilience Management

ABSTRACT. Progressing digitalization and networking of systems and organization opens new potentials and opportunities. On the downside, the increasing complexity and interdependencies amplify the challenge to withstand adverse events. This puts pressure on classic risk management processes, which are getting to its limits when coping with such complex structures. This observation resulted into the emergence of the resilience concept, which strongly focuses on inherent capabilities of systems or organizations. In this context, several fundamental works formed a basic understanding of resilience and shaped the way for its actual application. The present work continues this path by proposing a framework to integrate resilience management into the actual operation of systems and organizations by providing an operative process for the provision and management of the fundamental resilience capabilities of responding, monitoring, anticipation and learning. Digital concepts, e.g. Digital Twins and the related data and information base, bear huge potential for further enhancement of resilience capabilities by anticipating management. Thus, they are deeply integrated into the framework to provide the capability of a self-adjusting resilience management and risk mitigation process.

Supporting Resilience in Large-scale Multi-domain Operations as Learning Laboratories
PRESENTER: Martin Voshell

ABSTRACT. For military agencies worldwide, adversary threat capabilities continuously place new demands on weapon system training and mission and package employment that reveal critical integration challenges and new contexts for expertise and continuous learning. By looking at recent large-scale multi-domain air warfare exercises through the lens of the learning laboratory framework, this paper describes how next-generation data integration of multiple domains must be used to study and develop analytic insights into modeling resilience in complex mission environments especially in the context of integrated Live, Virtual, Constructive (LVC) training events.

Training for Macrocognitive Skills Awareness in Cybersecurity Professionals
PRESENTER: Hifsa Khan

ABSTRACT. Cybersecurity professionals are becoming increasingly important in all sectors due to the increase of security breaches in fields such as healthcare, government, and others (Hemphill & Longstreet, 2006). Cyber-defenders in particular carry out daunting and critical work to defend information systems from ever-increasing vulnerabilities (Shah et al., 2018). While cybersecurity experts are in high demand, there is not a clear educational path for emerging professionals to fill the gap in the market (Crumpler & Lewis, 2019). Furthermore, although research is advancing the state-of-the-art in the automation of network defense, little research has been conducted on the role of human cognition in the success of cybersecurity defenders, which remains critical in this complex environment. As a result of this, early-career cybersecurity defenders may not be well prepared for their roles initially (Crumpler & Lewis, 2019). In our presentation, we will present macrocognitive themes identified in our forthcoming publication (Schuster, in press) and describe their implications for research and practice. We will also propose a simple awareness training that introduces macrocognitive and metacognitive concepts to novice cybersecurity defenders to improve their performance and retention. In the proposed awareness training, we will first identify the macrocognitive concepts of interest using (Schuster, in press) as a guide and focusing on concepts that have the most support in the literature for cybersecurity and related fields. Next, we will define the skills and knowledge to be trained. We will illustrate our approach with an example. Finally, we will describe how we will evaluate our training at the reaction and learning levels defined by Kirkpatrick (1959, 1996). The proposed study serves two aims. First, it could provide evidence for the efficacy of metacognitive training for macrocognitive skills. Second, a distinguishing feature of the proposed study is the capture of qualitative feedback from subject-matter expert (SME) participants. This can inform research in cybersecurity training for macrocognition.

Panel: What Good Are Models?
PRESENTER: Laura Militello

ABSTRACT. In this session, a panel of experts will explore the question, "What can we do with models of decision making?" Experts will discuss how they have applied naturalistic decision making models in research and practice, with a focus on use of models in healthcare. Experts will share how they have adapted theoretical models to practical purposes, such as using models to guide the design of data collection methods, understanding real-time work in dynamic and critical settings, as a lens through which to view and understand patient safety events, and to guide the design of training.

Workshop: Use of Tactical Decision Games in Decision-Making Training in the Military and Law Enforcement

ABSTRACT. Topic: Tactical decision making in the military and law enforcement. Method: Tactical decision game (TDG) workshops. Participants will participate in an interactive, online TDG to exercise decision making under naturalistic conditions. Students will be presented a scenario requiring a decision. Each scenario consists of an oral description and a diagram. Students will be required to come up with a decision in the form of the orders they would issue to subordinates and will then participate in a guided cognitive critique of the decisions. The session will conclude with a discussion of the methodology. We propose to run at least one military session and one law enforcement session. Time limit: 90 minutes each. Participant limit: 20.

Health Care Cue-Detection Exercise
PRESENTER: Joseph Borders

ABSTRACT. A Cue-Detect ShadowBox exercise. Participants will participate in an interactive, online video exercise to train critical cue-detection skills in a cardiac care scenario. Participants will watch a training scenario video together to detect critical cues in patient condition and medical team performance. After making their own observations, participants will then be exposed to the cues that a panel of expert nurses were able to detect in the same video. Through a guided discussion, participants will then reflect what they gained from being exposed to the expert insights. The session will conclude with a discussion of the methodology and the results of a study conducted at the University of Texas at Arlington’s School of Nursing and Health Innovation. Time limit: 60 minutes. Participant limit: 30.

How can we help novice child protection social workers to see situations like experienced practitioners? A randomised controlled trial evaluation of the ShadowBox™ method using pre-recorded video feedback
PRESENTER: Andrew Whittaker

ABSTRACT. 1. Background:

Protecting children for abuse and neglect is a complex area of decision-making but frequent staff turnover has meant that many frontline child protection social workers are often relatively inexperienced. The ShadoxBoxTM method is an educational intervention that enables novice practitioners to gain decision-making skills quickly.

2. Aim:

This study was an RCT evaluation of an educational intervention for novice social workers that used the ShadoxBoxTM method adapted to include pre-recorded video feedback from an expert panel to test whether novice decision making would become more similar to experienced practitioners.

3. Methods:

The study was an RCT in which participants completed complex scenarios in a computer lab in control and intervention groups. The training method involved participants receiving feedback from highly experienced practitioners at each decision point within the scenarios. The intervention group received pre-recorded video feedback from a panel of highly experienced practitioners while the control group received no feedback.

4. Participants and Setting

Participants (n=83) were trainee social workers from a London university randomly allocated to control and intervention groups. Data was collected on computer stations using Qualtrics.

5. Results

Undertaking the scenarios improved both groups but greater improvement was seen in participants who had received video feedback from a panel of experienced practitioners. The results were promising, with participant accuracy increasing by 44% (from 31% to 75%) in the intervention group compared to an increase of 31% (from 32% to 63%) in the control group.

Considerable improvements were noted in both intervention and control groups, which suggest that scenario-based interventions can be a promising educational method as learning is rooted in real life scenarios and participants have the opportunity to reflect upon their decisions.

The qualitative findings are that novice participants make predictable errors, including: • Making shallow assumptions • Focusing exclusively on the parents rather than the child. • Jumping to early conclusions with insufficient information.

6. Conclusions

• ShadoxBoxTM training appears to be a promising intervention for improving decision making. • Novices benefitted from having concentrated exposure to complex scenarios focused on assessing risk and making professional judgements. • When this was augmented by direct feedback from a panel of highly experienced practitioners, these benefits were increased considerably. • The complexity of the scenarios also exposed the novices to real life pressures rather than the simplified versions used in decision research.

A Systems-Resilience Approach to Technology Transition in High-Consequence Work Systems
PRESENTER: Kelly Neville

ABSTRACT. The rate at which new information technology fails to successfully transition is disconcertingly high. Many technologies only manage to transition through a forced and difficult process, and often produce system disruptions that pose serious threats to high-consequence systems (e.g., Finkelstein & Dowell, 1996; Goldstein, 2005; Patterson, Militello, Su, & Sarkar, 2016; Sherwood, Neville, McLean, Walwanis, & Bolton, 2020; Trist & Bamforth, 1951; White, Wastell, Broadhurst, & Hall, 2010; Wears, Cook, & Perry, 2006). We propose a systems-centered approach to technology transition and system modernization grounded in complex-systems science and resilience engineering. In an exploratory research effort, we developed an initial version of the Transform with Resilience during Upgrades to Socio-Technical Systems (TRUSTS) Framework. This framework, derived from a literature review, specifies general characteristics of complex systems that allow them to behave with resilience in complex, high-stakes operating environments. We have begun translating the framework into tools for guiding technology transition and system modernization. The toolset will help technology-development and transition teams evaluate and engineer into their designs the system-resilience impacts of new technologies starting with the concept development stage and through deployment. Our primary goal is to use the TRUSTS Framework and tools to significantly improve the success rate of technology transition and count the transition as a success not just because it is being used but because it has transitioned without compromising the receiving system’s ability to perform and adapt in high-demand conditions. Complementary efforts will focus on applying and adapting the TRUSTS Framework and toolset to benefit transition and modernization challenges faced in multiple high-consequence domains, including air transportation, healthcare, and national security.

Designing for “Agility” in Envisioned Worlds: Concepts for Collaborative Intelligence in Human-Machine Teams
PRESENTER: Neelam Naikar

ABSTRACT. This paper explores two challenging design problems—that of designing “agile” sociotechnical systems, which can contend with very high levels of instability, uncertainty, and unpredictability, and that of designing future or envisioned systems, which do not yet exist. We suggest a design approach focusing on constraints or boundaries of successful operation, which can accommodate many possibilities for action, thereby providing a systematic basis for creating “agility” in the future system for dealing with dynamic, ambiguous environments. Given recent advances in artificial intelligence, we demonstrate some of these concepts with the decision ladder template from cognitive work analysis, showing how it may be applied in the design of “agile” human-machine intelligence. This design approach, which moves beyond standard approaches to function allocation, whether fixed or dynamic, toward ‘team player’, joint cognitive systems, and collaborative approaches to human-machine performance, has significant implications for research in artificial intelligence.

WAx on, WAx off: understanding the dynamics of cyber-socio-technical systems

ABSTRACT. The WAx is a conceptual framework that addresses the needs for understanding, modeling, and analyzing cyber-socio-technical systems, seen as the archetypal maximum expression of complexity achievable by collective systems formed by humans and technological artifacts (Patriarca et al., 2021).

The scope of current socio-technical systems is expanded by cyber elements, forcing reconsideration and re-evaluation of fundamental properties. The introduction of a kind of artifacts and agents (i.e., cyber-agents) objectively different in capability and rapidity of response, raises terrifically the number of possible non-linear interactions within modern systems. To this extent, Cyber-Socio-Technical Systems (CSTSs) sometimes include open AI issues (e.g., explainability, interpretability, and trust in technological artifacts) and exacerbate features related to adaptation, evolution, and, in general, unpredictability of complex adaptive systems. The understanding, analysis and, ultimately, the engineering of a CSTS, fundamental in the immediate future in terms of safety and security, passes through the definition of an appropriate systemic dimension of investigation. Risks of social manipulation, new types of warfare, or changes in power structures potentially attributable to cyber-artifacts come to mind in this sense. Relying on these observations, the WAx is structured to be impractical for reductionism and simplistic approaches that are patently unsuitable for CSTS risk and safety management. In a broader sense, the framework remains a conceptual support for reflection, useful for encouraging discussion and insight into the different system's performance. As such, it casts a light to remove biases on the dynamics involved in the different varieties of work, whether performed by humans or cybernetic agents.

Safety in Child Welfare: Challenges, Innovations, and Opportunities
PRESENTER: Emily Newsome

ABSTRACT. Child welfare organizations are tasked with maintaining the safety of children under complex circumstances. As such, they must be highly adaptable and resilient, while also supporting child welfare practitioners who make decisions that affect the lives of children and families. This panel will bring together experienced practitioners, researchers, and thought leaders in child welfare to explore decision making in child welfare. Topics will include: challenges of safety assessments and decisions in child welfare, how practitioners and systems have responded to the COVID-19 pandemic, and innovations for systems in which these decisions are made. Panellist backgrounds include experts in social work practice; managers of public child welfare systems in the U.S.; and researchers exploring innovative ideas for helping shift organizational cultures toward a culture of learning and promoting expertise.

Panel: Advanced Automation panel
PRESENTER: Emilie Roth

ABSTRACT. We plan to assemble panellists from a range of U.S. government agencies such as the U.S. Army, the U.S. Food & Drug Administration, the Federal Aviation Administration, and Sandia National Labs to share the issues and challenges they face in developing, applying, studying and regulating advanced automation, artificial intelligence, machine learning in complex settings. These technologies are proposed for use in military operations, civilian aviation, railways, patient and physician-facing medical devices, and many others. While some themes are common across contexts, challenges vary depending on context, intended use, and intended user. Panelists will discuss the challenges related to integrating advanced automation in their domains, risks they anticipate, and human capabilities that should be preserved and supported.

Running out of resilience: coping with the COVID-19 pandemic in Brazilian intensive care units

ABSTRACT. Intensive care units (ICUs) have been the most visible facet of healthcare services affected by the COVID-19 pandemic. This paper presents an exploratory investigation of how ICUs in Brazil have coped with the pandemic. A resilience engineering perspective is adopted, using a mixed-method research design. The quantitative stage comprised an exploratory survey for assessing the uptake of five work system guidelines that are logically related to resilience. The guidelines are: provide slack resources; give visibility to processes and outcomes; support diversity of perspectives when making decisions; monitor and understand the gap between work-as-imagined and work-as-done; and monitor unintended consequences of improvements and changes. These guidelines were described as statements in the survey, accompanied by ICU examples. There was a sliding bar with two endpoints: fully disagree (zero) and fully agree (100). For example, one of the statements related to the guideline on slack was as follows: there are extra or standby human resources that can be quickly deployed, and these are available in sufficient quantity to cope with unforeseen events. There were 33 valid responses, all from different ICUs. All respondents were invited to be interviewed and seven agreed with that. These interviews shed light on the rationale for the survey responses. The qualitative stage followed with interviews with two public health officials and non-participant observations of the meetings of a municipal COVID-19 crisis management committee in a capital city. Joint data analysis from the quantitative and qualitative stages supported the identification of five lessons learned, each corresponding to one of the aforementioned guidelines. Results indicated the guideline on slack resources as the most relevant as it addressed the key pandemic issue of matching capacity to demand. The provision of sufficient and experienced staff was found to be more challenging than the provision of facilities and supplies. Running out of resilience was part of everyday work in the ICUs, as evidenced by: (i) the lack of effective treatments, which means that despite all efforts the mortality rates were very high; (ii) exhausted professionals who could not cope with the stressful working conditions; and (iii) chronic and acute mismatches between capacity and demand, which implied that many patients received sub-standard care. ICU resilience was inextricably dependent on the resilience of society at large. An integrated approach to resilient healthcare is necessary, encompassing the micro, meso, and macro levels, besides accounting for the broader societal resilience.

The Workload Implications of Resilient Healthcare: The Role of Social Interactions

ABSTRACT. The extra effort of clinicians´ to provide care to their patients, which is a manifestation of resilience, usually goes unnoticed due to successful outcomes that occur most of the time. Therefore, clinicians’ self-sacrifice and overreliance on resilience become “normal work”. This study investigates the workload implications of resilient healthcare. The study setting was the radiology department of a major teaching hospital. The centrality of each clinician in four ability-based social networks (i.e., the resilience abilities of monitoring, responding, anticipating, and learning) was considered a proxy of each actor’s contribution to the overall system resilience. As such, a resilience score was calculated for each actor in each ability-based network, combining five indicators theoretically connected to the actor’s resilience: in-degree, closeness, and betweenness, which are derived from social network analysis, and availability and reliability, which are non-network attributes assessed through Likert-style questions. In turn, the individual workload was assessed based on the NASA-TLX questionnaire, which produces indicators related to six dimensions of workload (in addition to an overall score): mental, physical, temporal, performance, effort, and frustration. Both questionnaires – social network analysis and NASA-TLX – were answered by 155 out of the 220 staff of the radiology department. Follow-up interviews with 10 respondents were conducted to understand the influence of contextual factors on the actors´ workload and their resilience score based on network centrality. Preliminary data analysis indicates that the overall workload of the actors with the higher and lower resilience scores is similar, although there are significant differences in the workload dimensions (e.g., actors with higher resilience scores are subject to higher temporal and mental demand in comparison to those with lower resilience scores). As another finding, mental demand was the major workload dimension significantly correlated (95% confidence interval) to the resilience score. Furthermore, actors that take the initiative to help their coworkers without being requested had a higher overall workload than those that are more passive – the frequency at which actors take the initiative to help others was assessed by a specific question in the social network analysis survey. Overall, our findings indicated that social interactions related to the four resilience abilities mattered to the workload of the professionals surveyed. The adopted research method is expected to be replicable to other settings. It might be a new approach for the analysis of the human costs of resilient performance. The workload is proposed as a proxy of that human cost.

 

How Experts Know That Their Intuition is Right? Naturalistic Study on Executive Search Consultants

ABSTRACT. Previous studies on the role of intuition in experts’ decision-making allowed to discover strategy experts use to make correct decisions in situations in the absence of clear facts. This study aims to discover how expert executive search consultants know when to trust their intuition and how intuition influences their process of reasoning and decision-making. Six executive search consultants from Warsaw and London took part in the study (average experience = 13 years). Results of the study indicated that consultants use intuition to provide an initial appraisal of a candidate. Intuitive answer was found to be accompanied with a Feeling of Rightness (FOR) whose intensity was prompting the level of confidence in one’s intuition. Based on results, a decision-making model was created that depicts how FOR influences consultants’ process of reasoning and decision-making.

Naturalistic Decision Making - Thinking about the future
PRESENTER: Phil Tetlock

ABSTRACT. An Adversarial Collaboration to Explore the Value of Unnatural Cognitive Acts

One of my favorite scientific papers is the Kahneman and Klein 2009 article: A failure to disagree. Together they scoped out the conditions under which it is reasonable to expect the emergence of deeply intuitive, pattern-recognition forms of expertise and those under which it is not. The key moderator variable was the task environment: the degree to which people get clear rapid feedback on the correctness of their hunches. Intelligence analysts and geopolitical pundits rarely get clear, rapid feedback.

In my 2005 book Expert Political Judgment (EPJ), I studied professionals who work in learning-unfriendly settings: experts on world politics who offer opinions on big topics. The professionals I studied in EPJ were highly accomplished and intelligent people who could build powerful cases for the positions they staked out. But expertise did not translate into predictive power.

My 2015 book Superforecasting explores what happens when you introduce a profoundly disruptive mode of thinking, subjective-probability geopolitical forecasting tournaments that give participants rapid clear feedback on the correctness of their views. Progress in geopolitical forecasting required disrupting experts’ preferred natural modes of expressing uncertainty: make explicit probability judgments of events that many thought were beyond quantitative probability estimation.

We showed that people could learn to make increasingly well calibrated and discriminating judgments. There were big and systematic individual differences among forecasters. It was possible to develop training systems that boosted performance in a series of randomized controlled experiments.

So, my modest proposal to this conference is that in learning unfriendly environments it is sometimes beneficial to introduce disruptive interventions that pressure experts to engage in unnatural cognitive acts.

How Nursing Care Stations and Care Facilities in Japan Coped with COVID-19 Pandemic: A Survey of Challenges, Preparations, and Responses
PRESENTER: Taro Kanno

ABSTRACT. This study presents a questionnaire survey, designed by a resource-centric framework, about how Japanese home-visit nursing stations and nursing care facilities prepared and coped with the COVID-19 pandemic. We analyzed the answers using qualitative analysis to develop categories of good and typical practices to cope with pandemic situations. Based on the analysis results and discussions, we developed five categories of preparedness and countermeasures for resource management: protection, utilization, stockpiling, procuring, and repairs. Furthermore, we provided the categories with practice examples for human resources, which can be used as a reference when preparing new business continuity plans (BCPs).

Nine Principles for Supporting Distributed Sensemaking
PRESENTER: Simon Attfield

ABSTRACT. The future military operating Environment (FOE) is likely to be one in which operational military units will be less dependent on traditional command and control (C2) structures, operating more as semi-autonomous, self-organising teams. The devolution of C2 functions places a greater emphasis on the need for local peer-to-peer sensemaking and decision making. The work in this paper is a result of a project looking at the idea of sensemaking as a distributed phenomenon in military settings. We discuss this changing context and, as a prelude to the principles, offer a characterisation of distributed sensemaking as a form of distributed cognition which involves the harnessing of disparate information resources. Based on a review of sensemaking and team sensemaking sources from areas including Organisational Studies, Computer Supported Collaborative Work, Naturalistic Decision Making and Human Computer Interaction, we propose nine principles for supporting distributed sensemaking based on specific identified challenges and potential interventions. The nine principles broadly relate to the provision of information, interpretation and communication.

Using the Video-Occlusion Research Method to Guide the Design of a Perceptual-Cognitive Training Program for Law Enforcement Officers

ABSTRACT. LEOs often must make rapid decisions and actions to protect themselves and control a situation or a subject. While different agencies teach different techniques and philosophies for use-of-force and arrest-and-control, they are all enhanced by accurate and fast recognition of potentially attacking motions made by subjects. Whether subjects are civilians being questioned or suspects being detained, officers must respond with an appropriate scale and type of action. Slow or inappropriate actions can have highly negative results for subjects, officers, and their agencies. Unfortunately, training of rapid decision-action performance requires a high volume of practice that is difficult to acquire in training contexts that typically focus on “what you do” more than “what you see.”

One possible solution is to target the Recognition component for concentrated practice using the Video-Occlusion method developed in sports science. Video-occlusion was validated by reliably differentiating expert performers from less expert performers. It was then developed into video-occlusion training applications. For decades researchers have observed that video-occlusion might contribute to training rapid decision-action performance in military and law enforcement contexts. This study compared the performance of experienced law enforcement officers and less experienced officers on a video-occlusion test of Attack Recognition. Findings validated the video-occlusion method and recommended specific occlusion points for recognition-only video-occlusion training of attack recognition that can be delivered on tablets and cell phones as a way to accelerate expertise.

Implications for World Model Development Based on Perceptual Cues and Strategies Used by an Experienced Pilot to Negotiate Wires Obstacles
PRESENTER: Katie Ernst

ABSTRACT. Helicopter or other rotorcraft operations fly almost continuously in close proximity to hazards and are just beginning to incorporate advanced automation that will rely on dynamic world models. Our work highlights the perceptual cues and strategies of human rotorcraft pilots in navigating complex airspace and identifies opportunities to support human knowledge and resilience to improve overall mission effectiveness and safety.