ABSTRACT. The notion of system resilience is increasingly invoked in healthcare as the societal impacts of the COVID-19 pandemic (and its flow-on effects) push us into progressively unfamiliar territory. Around the world,  health systems have been forced to respond to the rapidity of unfolding events by rethinking conventional (regulatory and normative) approaches to safety and system performance. In some instances, this has been accompanied by a (perhaps temporary) tolerance for more adaptive forms of work and decentralised decision-making. There are of course many positive stories of ingenuity and success embedded within this narrative. Yet, in countries hard hit by COVID-19, the overall experience of health professionals has been far from positive. Aside from the significant human toll of the pandemic on clinical workers, these times of volatility and uncertainty have engendered unsustainable levels of adaptive pressure on healthcare teams - creating anxiety, fear and often extending them far beyond their clinical capabilities in order to respond resiliently to emergent issues. Whilst frontline teams are inarguably best placed to respond resiliently to emerging threats, we might also consider how the field of resilience engineering (RE) can guide systems and workers as they reconfigure services and operations. Further, can RE theory and practice inform how frontline teams might 'gracefully disrupt' when extension of services is no longer viable due to transformed demands - and what are the capabilities that must be cultivated to do so effectively at scale? This talk engages with these key questions through the lens of work delivered by the Queensland Health Bridge Labs program which in 2020 brought together expertise in safety innovation, human factors and ergonomics and creative design to address various challenges at the blunt and sharp end of healthcare.

ABSTRACT. The study presented in this article aimed to investigate how regulatory demands have shape safety management and safety management systems (SMS) in shipping companies. A case study focused on five companies operating in the Swedish maritime industry was carried out. Ten semi-structured interviews with respondents in management positions on shore and onboard were carried out during spring 2020. Originally the case study was supposed to include observations as well, but these were suspended due to COVID-19. The interviews were transcribed and analyzed with the help of the four resilience cornerstones; respond, anticipate, monitor, and learn. The results show that the companies, as well as the industry itself should be considered as complex systems, in which an additional layer of complexity is added by international and national rules and regulations. The work put into managing safety has become of such complex nature that for the most part, safety work is put into administrative tasks to ensure compliance with regulations. The results also highlight concerns among the management onboard that more time is dedicated to formulate routines and procedures than to actually making the work safer for those performing it. To compensate for the experienced gaps between work-as-imagined, or prescribed, and work-as-done questions based on the Resilience Assessment Grid (RAG) are presented. These can serve as a complement to the current safety management strategies, and may also give raise to a deeper cooperation between shore-based and ship-based personnel in shipping companies. This may enable a participatory approach in which safety management can be shaped bottom-up with as onboard personnel contributing essential knowledge on needs and preconditions to ensure safe operations while also taking the complexity of the system into account.

ABSTRACT. Aviation maintenance engineers are routinely required to work around challenges in the maintenance environment to deliver aircraft safely and efficiently. A process called Human Hazard Analysis is being used by the offshore helicopter community to explore the gaps between Work-As-Done on aircraft in-service with the Work-As-Imagined and Work-As-Prescribed during the design process. This presentation outlines the results of these workshops so far but also their limitations. Recent research into maintenance engineer decision-making has the potential to address some of these weaknesses. The results of these studies is outlined with corresponding recommendations on how this might be capitalised by the industry.

ABSTRACT. This article relates the construction and findings of an intervention at an airport following a safety incident that uncovered a deviation of communication practice between two air traffic controllers for handling small aircraft traffic on a secondary runway. The intervention was carried out by coaching and empowering a local team so that they would work with the air traffic controllers to understand the reasons for the deviation, involving the training of controllers to observe normal work practices in the control tower and its effects on local safety management practices. The results in terms of adapting scientific observation methods to a local issue and subsequent behavioural changes in attitudes towards the deviation of formal procedure are discussed.

ABSTRACT. Adaptation can be defined as making appropriate responses to changing situations while adaptability enables people to recognize and adapt fluidly to unexpected characteristics of a situation. Paradoxically, people at the front-line of organizations can intuitively recognize and understand cases of adaptation but there has been little guidance so far on how to learn, practice, evolve and transfer their experiences. The most common perspective is that people constitute a general adaptive capacity for organizations that compensate for shortfalls and problems at any level. Commercial aircraft operations and Air Traffic Control (ATC) are complex and tightly coupled systems moving passengers and cargo with safety and regularity worldwide. Although they are highly scripted environments, adaptation remains one of the most-valued concepts. At the operational level, commercial aviation and ATC share many common work demands of the classic Naturalistic Decision Making environments such as: rapidly escalating situations, multiple information resources, information-based uncertainty, time pressure, errors with high consequences, multifaceted decisions, trade-offs between interacting and conflicting goals, multiple stakeholders, hidden dynamic interdependencies and physiological stressors due to round the clock operations. The levels of safety in commercial aviation and ATC are high, but the adaptive expertise of their people is constrained because, firstly they are rarely exposed to exceptional situations and secondly relevant guidance is minimal. However, there exist many examples of extraordinary episodes of adaptation such as: the “Miracle of Hudson river” (i.e. US Airways Flight 1549 successful ditching at Hudson River in 2009 without any casualties) and the management of air traffic during ATC Zero on 11th of September 2001 when controllers managed safely the grounding of thousands of aircraft in the US skies at a short notice. In this line of reasoning, we explored the concept of adaptation in the commercial aircraft operations and the ATC through the lens of Resilience Engineering and Naturalistic Decision Making. We developed a model that includes cognitive and team processes such as: Steering, Sensemaking, Common Operating Picture, Coordination and Transfer of Control, Managing Changes and the Planning-Doing-Checking cycle. We analyzed cases of adaptive failures and successes and we developed a set of adaptation elements (i.e. a toolbox for adaptation). Our aim was to explore the nature of adaptation as a source of resilience in the commercial aircraft operations and the ATC and develop a practical toolbox with wider applications in diverse organizations and challenging situations.

ABSTRACT. The tech industry has rapidly grown into being a domain that touches many parts of our lives, meaning that it has the potential to have very high consequence incidents, while the increased speed of operations can create an ever increasing time pressure; at the same time the tech industry is historically one that is resistant to importing knowledge from some other disciplines and domains. In this session, I'll share the tactics and strategies that have worked for me in introducing resilience engineering concepts to teams at high tech companies.

In the session, I'll draw upon my background as an engineer and incident responder in big tech corporations, a former EMT, and someone who studies resilience engineering and produces analyses for others. I'll discuss: Common challenges in tech, such as: Resistance to importing knowledge from other disciplines Relative newness of many companies and the industry as a whole The possibility of high consequences at very fast speeds, coupled with little regulation or guidance Encouragement to eschew traditional approaches Strategies that have been successful such as: Introducing the ideas of a sociotechnical system Exploring ways that automation can fail Building off of increasingly accepted work such as blameless approaches to incident learning Common ground in incident response I'll also explore subjects that tend to meet more resistance including: That increased automation doesn't necessarily mean decreased incidents or accidents Automation as a team player doesn't mean waiting for increased AI.

ABSTRACT. Small to medium sized businesses are a significant contributor to the global economy. However, they are widely accepted to be more fragile than larger businesses, especially in terms of financial security and the coronavirus pandemic has exposed a need to build resilient performance of SMEs. This presentation explores how SMEs can move beyond the traditional, minimal approach to risk management, utilising the principles and techniques of resilience engineering to bounce forward from future shocks.

ABSTRACT. An Adversarial Collaboration to Explore the Value of Unnatural Cognitive Acts

One of my favorite scientific papers is the Kahneman and Klein 2009 article: A failure to disagree. Together they scoped out the conditions under which it is reasonable to expect the emergence of deeply intuitive, pattern-recognition forms of expertise and those under which it is not. The key moderator variable was the task environment: the degree to which people get clear rapid feedback on the correctness of their hunches. Intelligence analysts and geopolitical pundits rarely get clear, rapid feedback.

In my 2005 book Expert Political Judgment (EPJ), I studied professionals who work in learning-unfriendly settings: experts on world politics who offer opinions on big topics. The professionals I studied in EPJ were highly accomplished and intelligent people who could build powerful cases for the positions they staked out. But expertise did not translate into predictive power.

My 2015 book Superforecasting explores what happens when you introduce a profoundly disruptive mode of thinking, subjective-probability geopolitical forecasting tournaments that give participants rapid clear feedback on the correctness of their views. Progress in geopolitical forecasting required disrupting experts’ preferred natural modes of expressing uncertainty: make explicit probability judgments of events that many thought were beyond quantitative probability estimation.

We showed that people could learn to make increasingly well calibrated and discriminating judgments. There were big and systematic individual differences among forecasters. It was possible to develop training systems that boosted performance in a series of randomized controlled experiments.

So, my modest proposal to this conference is that in learning unfriendly environments it is sometimes beneficial to introduce disruptive interventions that pressure experts to engage in unnatural cognitive acts.

ABSTRACT. Research methodology and specific research methods for Resilience Engineering (RE) research remains sparse within the published literature. We present a mixed methods approach to data collection and analysis to support a new Safety-II/RE-informed data stream within a commercial airline’s safety management system. This project found mixed methods to be an extremely versatile research approach that affords internal and external validity as well as numerous opportunities for secondary data analysis; additionally, corporate leadership were eager to leverage the frequent opportunities afforded by this multi-staged (sequential and convergent) research design for establishing and maintaining concurrence between various datasets to further validate the produced research conclusions. The designs of this project are specifically useful to those organizations which are unwilling or unable to completely abandon Safety-I work but are looking to improve operational safety through the incorporation of more modern approaches.

ABSTRACT. Project Confidus received initial funding from the Scottish Oil & Gas Institute of Technology (OGTC) and emerges from the growing perception that most organisations invest substantial resources in reporting-based communication systems and yet, draw little benefit from such investments. It focuses on shifting from essentially one-way and highly fragmented communication practices, towards multi-way conversations. These conversations provide a much wider context on issues and therefore, generate enhanced potential for learning. The engagement in conversations produces reflection and sharing of knowledge with and between the various organisational levels. The use of dedicated technology aims to overcome time and geographical barriers and foster improved hierarchical, as well as peer-to-peer exchanges across remote assets.

ABSTRACT. Safety-II and Resilience Engineering professionals have many options for transforming industry safety programs, leveraging their expertise in the decision of which abstract theoretical model(s) to instantiate in a particular system. We present a case study and lessons learned from one large commercial airline’s effort to utilize Hollnagel’s Resilience Assessment Grid (RAG) in the transformation of their safety management system. Specifically, we will discuss the nuances of *how* the Learning and Improvement Team (LIT) transformed the RAG model into LIT’s LPAC model. We will share: our initial mindset as well as how it changed over the first two years; our deep dive into the epistemology of pilot work and how that was influential in developing LIT’s model and methodology; finding balance between corporate goals, organizational capabilities, safety theory, and rigorous methodology; and critiques we’ve received since going public with the program.

ABSTRACT. Cybersecurity professionals are becoming increasingly important in all sectors due to the increase of security breaches in fields such as healthcare, government, and others (Hemphill & Longstreet, 2006). Cyber-defenders in particular carry out daunting and critical work to defend information systems from ever-increasing vulnerabilities (Shah et al., 2018). While cybersecurity experts are in high demand, there is not a clear educational path for emerging professionals to fill the gap in the market (Crumpler & Lewis, 2019). Furthermore, although research is advancing the state-of-the-art in the automation of network defense, little research has been conducted on the role of human cognition in the success of cybersecurity defenders, which remains critical in this complex environment. As a result of this, early-career cybersecurity defenders may not be well prepared for their roles initially (Crumpler & Lewis, 2019). In our presentation, we will present macrocognitive themes identified in our forthcoming publication (Schuster, in press) and describe their implications for research and practice. We will also propose a simple awareness training that introduces macrocognitive and metacognitive concepts to novice cybersecurity defenders to improve their performance and retention. In the proposed awareness training, we will first identify the macrocognitive concepts of interest using (Schuster, in press) as a guide and focusing on concepts that have the most support in the literature for cybersecurity and related fields. Next, we will define the skills and knowledge to be trained. We will illustrate our approach with an example. Finally, we will describe how we will evaluate our training at the reaction and learning levels defined by Kirkpatrick (1959, 1996). The proposed study serves two aims. First, it could provide evidence for the efficacy of metacognitive training for macrocognitive skills. Second, a distinguishing feature of the proposed study is the capture of qualitative feedback from subject-matter expert (SME) participants. This can inform research in cybersecurity training for macrocognition.

ABSTRACT. For military agencies worldwide, adversary threat capabilities continuously place new demands on weapon system training and mission and package employment that reveal critical integration challenges and new contexts for expertise and continuous learning. By looking at recent large-scale multi-domain air warfare exercises through the lens of the learning laboratory framework, this paper describes how next-generation data integration of multiple domains must be used to study and develop analytic insights into modeling resilience in complex mission environments especially in the context of integrated Live, Virtual, Constructive (LVC) training events.

ABSTRACT. The rate at which new information technology fails to successfully transition is disconcertingly high. Many technologies only manage to transition through a forced and difficult process, and often produce system disruptions that pose serious threats to high-consequence systems (e.g., Finkelstein & Dowell, 1996; Goldstein, 2005; Patterson, Militello, Su, & Sarkar, 2016; Sherwood, Neville, McLean, Walwanis, & Bolton, 2020; Trist & Bamforth, 1951; White, Wastell, Broadhurst, & Hall, 2010; Wears, Cook, & Perry, 2006). We propose a systems-centered approach to technology transition and system modernization grounded in complex-systems science and resilience engineering. In an exploratory research effort, we developed an initial version of the Transform with Resilience during Upgrades to Socio-Technical Systems (TRUSTS) Framework. This framework, derived from a literature review, specifies general characteristics of complex systems that allow them to behave with resilience in complex, high-stakes operating environments. We have begun translating the framework into tools for guiding technology transition and system modernization. The toolset will help technology-development and transition teams evaluate and engineer into their designs the system-resilience impacts of new technologies starting with the concept development stage and through deployment. Our primary goal is to use the TRUSTS Framework and tools to significantly improve the success rate of technology transition and count the transition as a success not just because it is being used but because it has transitioned without compromising the receiving system’s ability to perform and adapt in high-demand conditions. Complementary efforts will focus on applying and adapting the TRUSTS Framework and toolset to benefit transition and modernization challenges faced in multiple high-consequence domains, including air transportation, healthcare, and national security.

ABSTRACT. John MERRIHEW, Veloxiti; Patty McDermott, the MITRE Corporation; Kerstin Haring, University of Denver

People query intelligent assistants on personal devices to help with ever-expanding tasks and decisions throughout their day, yet it is difficult to imagine these assistants being reliable or effective enough to support decision makers in NDM-like settings. What is the current state of research applying this technology to classic NDM situations, and what are the potential payoffs and problematic pitfalls of a JARVIS-like software assistant for high-stakes decision makers? Are Intelligent Assistants being implemented today meeting needs for flexible and adaptive support of humans in dynamic, high-risk environments? In this panel we will convene an open and honest discussion on this topic. Recognizing the wealth of guidance that has emerged in applied research over the past few decades of work on agents, associates, and assistants, Panelists will describe their ongoing work in this area, highlighting ways forward and tough challenges that remain.

ABSTRACT. We plan to assemble panellists from a range of U.S. government agencies such as the U.S. Army, the U.S. Food & Drug Administration, the Federal Aviation Administration, and Sandia National Labs to share the issues and challenges they face in developing, applying, studying and regulating advanced automation, artificial intelligence, machine learning in complex settings. These technologies are proposed for use in military operations, civilian aviation, railways, patient and physician-facing medical devices, and many others. While some themes are common across contexts, challenges vary depending on context, intended use, and intended user. Panelists will discuss the challenges related to integrating advanced automation in their domains, risks they anticipate, and human capabilities that should be preserved and supported.

ABSTRACT. Trust is a buzzword in autonomous system design, with sponsors and managers often asking for it by name. The waters get murky as practitioners navigate the differences between trust and calibrated trust and the assessment techniques associated with each construct. In this session there is no audience - by attending you are part of a (large) roundtable so come lend your expertise and opinions! Through guided facilitation, the group will explore research on trust and calibrated trust; discuss whether the emphasis on ‘trusting the system’ is overblown; and share lessons learned on evaluating user trust and assessing whether users understand and use the system appropriately. A variety of facilitation techniques will be used to keep participants engaged and ensure that a diverse set of voices is heard. Attendees will walk away with concrete techniques for evaluating trust as well as advice for employing them. They will also have the opportunity to interact and connect with a wide variety of trust researchers that they may not have otherwise met.

ABSTRACT. Although collaborative robots (cobot) applications are increasingly utilised to physically collaborate in realtime with human operators, current cobot safety mainly focuses on a techno-centric perspective in terms of physical separation and managing the net result of kinetic energy in the cobot system. Human Factors (HF) research or its subcategory Cognitive Systems Engineering (CSE) are largely absent in cobot safety. To complement the current techno-centric approach, this project introduces a socio-technical safety and resilience analysis perspective for human-robot interaction by applying systemic safety analysis methods, with a specific focus on observability, predictability, and directability (OPD) requirements for JCS in relation to cobot operations. For the purpose of this project, cobots are broadly defined by any robot system with the potential for foreseeable or intentional physical contact between robots and humans, not necessarily restricted to system operators. The project sets out a generic cobot safety framework that is based on a FRAM analysis of the work system to assess OPD requirements in JCS.

ABSTRACT. Building maintenance comprises a set of complex socio-technical activities, with many interacting agents. Traditional tools for safety management make the results of such evaluations distant from real situations. Resilience Engineering (RE) argues that improvements in safety performance concern the ability to recognize and adapt to handle unanticipated perturbations. This study aims at increasing the understanding of everyday building maintenance activities for air conditioning systems to uncover resilience abilities developed by the maintenance team to deal with everyday challenges. The Functional Resonance Analysis Method (FRAM) has been applied to model these activities. The data collection comprised interviews and observations. As a contribution, this study outlined the potential of the FRAM model as the basis of in-depth and systematic analysis of daily performance, highlighting resilience abilities aligned with RE principles.

ABSTRACT. It can be difficult to understand changes to team performance in the moment and at a larger organizational scale, especially when adapting to unique evolving pressures and opportunities. To translate theory into practice and evaluate resilience at a project level, we are measuring four responses to saturation on a regular cadence and creating new project monitoring representations for communication and comparison at all organizational levels. The ongoing work exemplifies the convergence of contextualized decision-making with a resilience engineering framework for analysis.

ABSTRACT. Environmental, Health & Safety integrated management systems are widespread in manufacturing industries, despite limitations regarding the system efficiency and performance, especially concerning technical and behavioral factors. The Functional Safety tool can collaborate with better results in the implementation of an integrated management system. This study aims to identify attributes, and barriers in the studies of integrating functional safety and Environmental, Health, & Safety with sustainability and management systems. A systematic literature review was carried out in three phases: i) selection of studies on the topic, ii) bibliometric analysis, and iii) content analysis. This review, done through 702 searched documents from the Scopus, Science Direct, and Web of Science databases, resulted in 84 documents after exclusion criteria and the inclusion of 16 documents via the Snowball method. As the main deliverables, this study listed 30 attributes for the implementation of an integrated management system. Moreover, 25 raised barriers to implementing a system that integrates environmental, health & safety, and functional safety can be used to support practitioners in the implementation of manufactures. Finally, as the main contribution, there is a survey of managerial implications on how to implement this integrated management system in manufacturing industries.

ABSTRACT. Small to medium sized businesses are a significant contributor to the global economy. However, they are widely accepted to be more fragile than larger businesses, especially in terms of financial security and the coronavirus pandemic has exposed a need to build resilient performance of SMEs. This presentation explores how SMEs can move beyond the traditional, minimal approach to risk management, utilising the principles and techniques of resilience engineering to bounce forward from future shocks.

ABSTRACT. Aviation maintenance engineers are routinely required to work around challenges in the maintenance environment to deliver aircraft safely and efficiently. A process called Human Hazard Analysis is being used by the offshore helicopter community to explore the gaps between Work-As-Done on aircraft in-service with the Work-As-Imagined and Work-As-Prescribed during the design process. This presentation outlines the results of these workshops so far but also their limitations. Recent research into maintenance engineer decision-making has the potential to address some of these weaknesses. The results of these studies is outlined with corresponding recommendations on how this might be capitalised by the industry.

ABSTRACT. The design and development of software systems is a complex process. Even a relatively small project may involve distributed teams of several different types of professions, dozens of different tools, and hundreds of different documents. Prescriptive system development processes, of which there are many, are focused on how to proactively reduce potential variation. By contrast, there has been little work exploring how to adapt to unanticipated variation - how to cope with surprises - in software project management. Surprises in the systems development process can impact the cost of development, the scope or quality of the product, the time of delivery, and the relationships between stakeholders. They can have positive or negative impacts, opening new opportunities or hindering progress forward. Responding to a surprise constitutes a test of the ability of the project team and parent organization to recognize and adapt. The sensitivity to meaningful variation and potential risks, the flexibility of work processes and resource allocation, and the expertise of the responsible practitioners all play important roles in the adaptation process. In this session we will begin by discussing patterns in problem detection (Klein), anomaly response (Woods & Hollnagel), and organizational surprises (Cunha, Clegg & Kamoche). We will share some examples from our own work at Mile Two and analyze them. The aim of the analysis will be to identify macrocognitive functions related to coping with surprises in this context, and how project or organizational resilience is evidenced in these examples. To make this an interactive session, at the start of the conference we will ask participants to share electronically their own accounts of surprises in systems development projects. We will include these in our analyses - ahead of time for earlier contributions, and in real time for late or concurrent contributions. We will use an online whiteboard or similar tool to actively show the real-time analysis.

Please submit your examples of surprises in software development projects via this form: https://forms.gle/oaZGRgfqmyQuKNb2A

ABSTRACT. There are growing concerns about how healthcare systems can adapt in times of crisis. The overarching challenge lies in how resilience engineering could be used to analyze and improve the performance of healthcare systems concerning the Covid-19. This study aims to describe the relationship between resilience potentials and health and safety aspects and its consequences on quality and resilience in healthcare systems. This study has a quantitative methodological approach using a survey with the Resilience Analysis Grid as an approach to analyzing organizational resilience based on the idea that four potentials (responding, monitoring, learning, and anticipating) influence patient safety, occupational health and safety, and resilient system performance. As for this study results, anticipating and monitoring, overall resilience, and occupational health and safety are the variables that need more attention in healthcare systems. This study has a dyad of contributions, as a practice, evaluate the resilience in a pandemic time, and as theoretical, the identification of the importance of resilience four potential connections in healthcare systems

ABSTRACT. This work presents, based on non-technical skills concepts, BRM (Bridge Resource Management) fundamentals and maritime safety regulations requirements, a proposed design of a debriefing tool for performance evaluation in maritime training simulations. This debriefing tool is formed by a set of questions, comprehending general questions, assessment of the voyage and potential for improvement. It is part of the training simulations and was designed to encourage the discussion regarding the skills, attitudes and interactions that were experienced by the students during the training, aiming to identify and stimulate the development of these and other skills necessary for a productive and safe work on board the vessels.

ABSTRACT. This paper describes how Mile Two has attempted to integrate CSE with other disciplines in order to design software to support decision making in joint cognitive systems. The paper discusses three aspects of our approach: 1) Recognizing the value and limitation of different design perspectives; 2) Cross-training; and 3) Involving all disciplines at all stage of the design process - co-creation. The paper will discuss the successes and difficulties associated with implementing a total team process across different stages of design and development.

ABSTRACT. This paper reports an on-going project to explore the challenge of measuring adaptive performance in teams for improved sensemaking. The paper describes an initial rapid review of literature which highlights: (i) the conceptual emphasis of the existing literature, (ii) nascent measures for examining individual adaptive skill and adaptivity in teams, and (iii) a lack of empirical evidence for the development or efficacy of training interventions to improve or develop adaptive team performance. We build on previous understanding about adaptive performance. Nascent measures of individual and team adaptivity are identified and evaluated. Initial experiences of efforts to experimentally measure adaptive team performance for sensemaking will be presented if available.

ABSTRACT. When facing shocks and stresses we aim for communities to not just bounce-back and survive, but bounce-forward and thrive. The principle in place has been to enable ‘preparedness’ to facilitate effective planning and response and ‘resilience’ to support anticipation, monitoring, response and learning. However, the coronavirus pandemic has exposed deep problems in how we approach resilience and preparedness at a community level.

Deliberative democratic processes have risen in popularity across the world in the form of citizen assemblies. Such assemblies have been successful and have been applied in areas such as abortion and gay marriage legislation (the classic Irish example) but also issues in transport, social policy, political appointments, and electoral reform. These assemblies work in parallel to formal governance mechanisms and act as guidance for decision making that is based on consensus among a large group of citizens drawn at random from the community.

Social audits are a form of citizen participation that looks at government performance and accountability. The objective of a social audit is to monitor and evaluate government performance - making public officials accountable for their actions and decisions. A social audit exercise is a mechanism of oversight: that is, the control that citizens can exert on their government officials to ensure that they act transparently, responsibly, and effectively.

This paper outlines a study of citizen assemblies and social audits and explores an application called a citizen audit for resilience and preparedness (CARP). This evaluates the capabilities and capacities of a community to respond to stresses and shocks, the process overseen and performed by a randomly selected group of citizens. Resilience and preparedness are characteristics that should emerge from a community rather than something imposed on it, a citizen audit could go some way towards building an active relationship between government bodies and its community.

Also discussed is a means of measuring success of the initiative, based on the regular reviewing of individual well-being, which can be run before and after a disruption.

ABSTRACT. Definition: Curiosity is the motivation to bridge a gap. The literature suggests this can be an information gap or a mental model gap. I think a better formulation is to consider a plausibility gap. Features of curiosity: Curiosity is superficial and transient and intense and impulsive. Superficial — it can arise, change focus, and end abruptly. (Waiting in a checkout line in a supermarket, we may become intensely curious about the latest news regarding a movie star’s marital woes — this curiosity disappears as soon as we step forward and away from the tabloids.) Transient — it reflects attention, and when attention captured by something else, curiosity disappears. Intense — people will work to satisfy their curiosity despite being hungry or thirsty Impulsive — something can evoke curiosity and stop us in our tracks. We may be tempted to buy that tabloid in the supermarket as long as we are looking at it Blocking curiosity: The research literature identifies common practices that can interfere with curiosity. Many of these inhibitors are initiated by instructors. Ask closed questions — calling for Yes/No answers. Focus on procedures. Emphasize memorization. Claim that there are right answers. Quickly correct students’ mistakes. Provide After-Action reviews that contain mostly statements and few questions. Focus on what the trainee is doing, and not why. Overwhelm students with details. Introduce too much complexity too early. (Too much confusion/complexity/ambiguity inhibits curiosity. The student stops being curious and instead tries to memorize all the material. Further, presenting too many open variables and loose ends can inhibit curiosity. Additional instructional behaviors that can inhibit curiosity include the following: Discourage questions from the students/trainees — make it clear that the class needs to get through all the material. Discourage class discussions. Provide complete explanations, leaving no space for student to engage in self-explanation. Provide explanations that make students feel stupid — use ridicule to make students afraid to appear dumb. Promoting curiosity. The collection of common inhibitors is fairly discouraging. However, the literature also identifies a number of factors that instructors and trainers can use to harness curiosity. These include the following: Pose a question; present contrasts and counterfactuals/hypotheticals; provide a sequence with an unknown resolution [e.g., who will win an election, or a race?). In addition, teachers can try to switch out of a mindset of criticizing students and instead wonder what caused the student to make a mistake.

ABSTRACT. The COVID-19 pandemic has challenged the resilience of healthcare services in several countries. This exploratory study presents the lessons learned by a leading private hospital in Southern Brazil, using the lens of resilience engineering.

ABSTRACT. According to Flach & Voorhorst (2020) effective management of real-world situations can be achieved by treating human decisions according to meaning processing approaches to cognition. We describe a System Theoretic Process Analysis (STPA) of traffic management in a busy port area of the Netherlands. Reflecting on the meaning processing approach to cognition, we then generate and test a new set of prompts that the STPA analyst can use as they generate scenarios to explain why human actors would carry out unsafe control actions. The new prompts help the analyst consider the human controller as an individual embedded and in a dynamic ecology, with perceptions and actions emerging from interactions with that ecology. They help consider people as perceivers of affordances rather than whole processes; and as aware of the meaning of actions for several valued goals. On applying the new prompts to the port traffic management case, new insight was generated concerning how truck drivers will act to perceive (and not just perceive to act), seek out workarounds in the face of terminal closures, and attend to information that helps them decide whether diverting or ignoring diversions is most likely to lead to punctual delivery. While extending STPA analysis to account for the meaning processing approach, demands greater resource and understanding, there will be occasions when a less superficial approach to human control is desired by STPA practitioners.

ABSTRACT. Organizations and communities are embracing the principles and practices of sustainability and resilience in response to wicked issues such as climate change, income inequality, technological disruption, economic displacement, and emerging infectious diseases. Many of them find stakeholder aspirations and interests conflict even when they are well aligned on the need for decisive action to meet these challenges.

This workshop establishes a framework for and seeks engagement from interested experts to further develop an approach to sensemaking and sustainability to guide public participation processes among diverse stakeholders to facilitate constructive dialogue. The workshop seeks to establish specific criteria for each dimension of the framework to help users define individual and organizational roles and responsibilities in developing community resilience.

The proposed framework builds on the concepts of public value and co-production of public goods. It consists of five key elements of resilience to guide sustainable community engagement for organizational and community resilience. Workshop participants will employ the framework to examine diverse stakeholder interests and activities related to key resilience challenges facing communities.

• Open – How can individuals and organizations share data and insights with one another without compromising data integrity, personal privacy, or competitive advantage?

• Varied – What critical infrastructure pathways, key resources, and information sources support confidence, reliability, and redundancy under crisis conditions?

• Simple – What systems or methods are available or required to manage complexity, support situational awareness, and promote a common operating picture?

• Local – What resources exist within the community to sustain life and maintain critical functions during extreme events? How will individuals and organizations fill key gaps?

• Connected – How are individuals and organizations integrated within regional, national, and international networks and cooperative agreements that can provide critical insights or key resources before, during or after a crisis?

This workshop is an opportunity for experienced emergency management practitioners, public policy analysts, decision- support researchers, and crisis communication experts among others to expand and develop the proposed framework for public participation and community engagement. Participants in this workshop will contribute to an enhanced understanding of this framework, its application to various policy and practice challenges, and how it can be developed and applied as a tool for community engagement, comprehensive emergency planning, and policy analysis. Participants will be invited to collaborate in further development and deployment of this framework as a community engagement strategy.

ABSTRACT. Progressing digitalization and networking of systems and organization opens new potentials and opportunities. On the downside, the increasing complexity and interdependencies amplify the challenge to withstand adverse events. This puts pressure on classic risk management processes, which are getting to its limits when coping with such complex structures. This observation resulted into the emergence of the resilience concept, which strongly focuses on inherent capabilities of systems or organizations. In this context, several fundamental works formed a basic understanding of resilience and shaped the way for its actual application. The present work continues this path by proposing a framework to integrate resilience management into the actual operation of systems and organizations by providing an operative process for the provision and management of the fundamental resilience capabilities of responding, monitoring, anticipation and learning. Digital concepts, e.g. Digital Twins and the related data and information base, bear huge potential for further enhancement of resilience capabilities by anticipating management. Thus, they are deeply integrated into the framework to provide the capability of a self-adjusting resilience management and risk mitigation process.

ABSTRACT. For military agencies worldwide, adversary threat capabilities continuously place new demands on weapon system training and mission and package employment that reveal critical integration challenges and new contexts for expertise and continuous learning. By looking at recent large-scale multi-domain air warfare exercises through the lens of the learning laboratory framework, this paper describes how next-generation data integration of multiple domains must be used to study and develop analytic insights into modeling resilience in complex mission environments especially in the context of integrated Live, Virtual, Constructive (LVC) training events.

ABSTRACT. Cybersecurity professionals are becoming increasingly important in all sectors due to the increase of security breaches in fields such as healthcare, government, and others (Hemphill & Longstreet, 2006). Cyber-defenders in particular carry out daunting and critical work to defend information systems from ever-increasing vulnerabilities (Shah et al., 2018). While cybersecurity experts are in high demand, there is not a clear educational path for emerging professionals to fill the gap in the market (Crumpler & Lewis, 2019). Furthermore, although research is advancing the state-of-the-art in the automation of network defense, little research has been conducted on the role of human cognition in the success of cybersecurity defenders, which remains critical in this complex environment. As a result of this, early-career cybersecurity defenders may not be well prepared for their roles initially (Crumpler & Lewis, 2019). In our presentation, we will present macrocognitive themes identified in our forthcoming publication (Schuster, in press) and describe their implications for research and practice. We will also propose a simple awareness training that introduces macrocognitive and metacognitive concepts to novice cybersecurity defenders to improve their performance and retention. In the proposed awareness training, we will first identify the macrocognitive concepts of interest using (Schuster, in press) as a guide and focusing on concepts that have the most support in the literature for cybersecurity and related fields. Next, we will define the skills and knowledge to be trained. We will illustrate our approach with an example. Finally, we will describe how we will evaluate our training at the reaction and learning levels defined by Kirkpatrick (1959, 1996). The proposed study serves two aims. First, it could provide evidence for the efficacy of metacognitive training for macrocognitive skills. Second, a distinguishing feature of the proposed study is the capture of qualitative feedback from subject-matter expert (SME) participants. This can inform research in cybersecurity training for macrocognition.

ABSTRACT. In this session, a panel of experts will explore the question, "What can we do with models of decision making?" Experts will discuss how they have applied naturalistic decision making models in research and practice, with a focus on use of models in healthcare. Experts will share how they have adapted theoretical models to practical purposes, such as using models to guide the design of data collection methods, understanding real-time work in dynamic and critical settings, as a lens through which to view and understand patient safety events, and to guide the design of training.

ABSTRACT. Topic: Tactical decision making in the military and law enforcement. Method: Tactical decision game (TDG) workshops. Participants will participate in an interactive, online TDG to exercise decision making under naturalistic conditions. Students will be presented a scenario requiring a decision. Each scenario consists of an oral description and a diagram. Students will be required to come up with a decision in the form of the orders they would issue to subordinates and will then participate in a guided cognitive critique of the decisions. The session will conclude with a discussion of the methodology. We propose to run at least one military session and one law enforcement session. Time limit: 90 minutes each. Participant limit: 20.

ABSTRACT. A Cue-Detect ShadowBox exercise. Participants will participate in an interactive, online video exercise to train critical cue-detection skills in a cardiac care scenario. Participants will watch a training scenario video together to detect critical cues in patient condition and medical team performance. After making their own observations, participants will then be exposed to the cues that a panel of expert nurses were able to detect in the same video. Through a guided discussion, participants will then reflect what they gained from being exposed to the expert insights. The session will conclude with a discussion of the methodology and the results of a study conducted at the University of Texas at Arlington’s School of Nursing and Health Innovation. Time limit: 60 minutes. Participant limit: 30.

ABSTRACT. 1. Background:

Protecting children for abuse and neglect is a complex area of decision-making but frequent staff turnover has meant that many frontline child protection social workers are often relatively inexperienced. The ShadoxBoxTM method is an educational intervention that enables novice practitioners to gain decision-making skills quickly.

2. Aim:

This study was an RCT evaluation of an educational intervention for novice social workers that used the ShadoxBoxTM method adapted to include pre-recorded video feedback from an expert panel to test whether novice decision making would become more similar to experienced practitioners.

3. Methods:

The study was an RCT in which participants completed complex scenarios in a computer lab in control and intervention groups. The training method involved participants receiving feedback from highly experienced practitioners at each decision point within the scenarios. The intervention group received pre-recorded video feedback from a panel of highly experienced practitioners while the control group received no feedback.

4. Participants and Setting

Participants (n=83) were trainee social workers from a London university randomly allocated to control and intervention groups. Data was collected on computer stations using Qualtrics.

5. Results

Undertaking the scenarios improved both groups but greater improvement was seen in participants who had received video feedback from a panel of experienced practitioners. The results were promising, with participant accuracy increasing by 44% (from 31% to 75%) in the intervention group compared to an increase of 31% (from 32% to 63%) in the control group.

Considerable improvements were noted in both intervention and control groups, which suggest that scenario-based interventions can be a promising educational method as learning is rooted in real life scenarios and participants have the opportunity to reflect upon their decisions.

The qualitative findings are that novice participants make predictable errors, including: • Making shallow assumptions • Focusing exclusively on the parents rather than the child. • Jumping to early conclusions with insufficient information.

6. Conclusions

• ShadoxBoxTM training appears to be a promising intervention for improving decision making. • Novices benefitted from having concentrated exposure to complex scenarios focused on assessing risk and making professional judgements. • When this was augmented by direct feedback from a panel of highly experienced practitioners, these benefits were increased considerably. • The complexity of the scenarios also exposed the novices to real life pressures rather than the simplified versions used in decision research.

ABSTRACT. The rate at which new information technology fails to successfully transition is disconcertingly high. Many technologies only manage to transition through a forced and difficult process, and often produce system disruptions that pose serious threats to high-consequence systems (e.g., Finkelstein & Dowell, 1996; Goldstein, 2005; Patterson, Militello, Su, & Sarkar, 2016; Sherwood, Neville, McLean, Walwanis, & Bolton, 2020; Trist & Bamforth, 1951; White, Wastell, Broadhurst, & Hall, 2010; Wears, Cook, & Perry, 2006). We propose a systems-centered approach to technology transition and system modernization grounded in complex-systems science and resilience engineering. In an exploratory research effort, we developed an initial version of the Transform with Resilience during Upgrades to Socio-Technical Systems (TRUSTS) Framework. This framework, derived from a literature review, specifies general characteristics of complex systems that allow them to behave with resilience in complex, high-stakes operating environments. We have begun translating the framework into tools for guiding technology transition and system modernization. The toolset will help technology-development and transition teams evaluate and engineer into their designs the system-resilience impacts of new technologies starting with the concept development stage and through deployment. Our primary goal is to use the TRUSTS Framework and tools to significantly improve the success rate of technology transition and count the transition as a success not just because it is being used but because it has transitioned without compromising the receiving system’s ability to perform and adapt in high-demand conditions. Complementary efforts will focus on applying and adapting the TRUSTS Framework and toolset to benefit transition and modernization challenges faced in multiple high-consequence domains, including air transportation, healthcare, and national security.

ABSTRACT. This paper explores two challenging design problems—that of designing “agile” sociotechnical systems, which can contend with very high levels of instability, uncertainty, and unpredictability, and that of designing future or envisioned systems, which do not yet exist. We suggest a design approach focusing on constraints or boundaries of successful operation, which can accommodate many possibilities for action, thereby providing a systematic basis for creating “agility” in the future system for dealing with dynamic, ambiguous environments. Given recent advances in artificial intelligence, we demonstrate some of these concepts with the decision ladder template from cognitive work analysis, showing how it may be applied in the design of “agile” human-machine intelligence. This design approach, which moves beyond standard approaches to function allocation, whether fixed or dynamic, toward ‘team player’, joint cognitive systems, and collaborative approaches to human-machine performance, has significant implications for research in artificial intelligence.

ABSTRACT. The WAx is a conceptual framework that addresses the needs for understanding, modeling, and analyzing cyber-socio-technical systems, seen as the archetypal maximum expression of complexity achievable by collective systems formed by humans and technological artifacts (Patriarca et al., 2021).

The scope of current socio-technical systems is expanded by cyber elements, forcing reconsideration and re-evaluation of fundamental properties. The introduction of a kind of artifacts and agents (i.e., cyber-agents) objectively different in capability and rapidity of response, raises terrifically the number of possible non-linear interactions within modern systems. To this extent, Cyber-Socio-Technical Systems (CSTSs) sometimes include open AI issues (e.g., explainability, interpretability, and trust in technological artifacts) and exacerbate features related to adaptation, evolution, and, in general, unpredictability of complex adaptive systems. The understanding, analysis and, ultimately, the engineering of a CSTS, fundamental in the immediate future in terms of safety and security, passes through the definition of an appropriate systemic dimension of investigation. Risks of social manipulation, new types of warfare, or changes in power structures potentially attributable to cyber-artifacts come to mind in this sense. Relying on these observations, the WAx is structured to be impractical for reductionism and simplistic approaches that are patently unsuitable for CSTS risk and safety management. In a broader sense, the framework remains a conceptual support for reflection, useful for encouraging discussion and insight into the different system's performance. As such, it casts a light to remove biases on the dynamics involved in the different varieties of work, whether performed by humans or cybernetic agents.

ABSTRACT. Child welfare organizations are tasked with maintaining the safety of children under complex circumstances. As such, they must be highly adaptable and resilient, while also supporting child welfare practitioners who make decisions that affect the lives of children and families. This panel will bring together experienced practitioners, researchers, and thought leaders in child welfare to explore decision making in child welfare. Topics will include: challenges of safety assessments and decisions in child welfare, how practitioners and systems have responded to the COVID-19 pandemic, and innovations for systems in which these decisions are made. Panellist backgrounds include experts in social work practice; managers of public child welfare systems in the U.S.; and researchers exploring innovative ideas for helping shift organizational cultures toward a culture of learning and promoting expertise.

ABSTRACT. We plan to assemble panellists from a range of U.S. government agencies such as the U.S. Army, the U.S. Food & Drug Administration, the Federal Aviation Administration, and Sandia National Labs to share the issues and challenges they face in developing, applying, studying and regulating advanced automation, artificial intelligence, machine learning in complex settings. These technologies are proposed for use in military operations, civilian aviation, railways, patient and physician-facing medical devices, and many others. While some themes are common across contexts, challenges vary depending on context, intended use, and intended user. Panelists will discuss the challenges related to integrating advanced automation in their domains, risks they anticipate, and human capabilities that should be preserved and supported.

ABSTRACT. Intensive care units (ICUs) have been the most visible facet of healthcare services affected by the COVID-19 pandemic. This paper presents an exploratory investigation of how ICUs in Brazil have coped with the pandemic. A resilience engineering perspective is adopted, using a mixed-method research design. The quantitative stage comprised an exploratory survey for assessing the uptake of five work system guidelines that are logically related to resilience. The guidelines are: provide slack resources; give visibility to processes and outcomes; support diversity of perspectives when making decisions; monitor and understand the gap between work-as-imagined and work-as-done; and monitor unintended consequences of improvements and changes. These guidelines were described as statements in the survey, accompanied by ICU examples. There was a sliding bar with two endpoints: fully disagree (zero) and fully agree (100). For example, one of the statements related to the guideline on slack was as follows: there are extra or standby human resources that can be quickly deployed, and these are available in sufficient quantity to cope with unforeseen events. There were 33 valid responses, all from different ICUs. All respondents were invited to be interviewed and seven agreed with that. These interviews shed light on the rationale for the survey responses. The qualitative stage followed with interviews with two public health officials and non-participant observations of the meetings of a municipal COVID-19 crisis management committee in a capital city. Joint data analysis from the quantitative and qualitative stages supported the identification of five lessons learned, each corresponding to one of the aforementioned guidelines. Results indicated the guideline on slack resources as the most relevant as it addressed the key pandemic issue of matching capacity to demand. The provision of sufficient and experienced staff was found to be more challenging than the provision of facilities and supplies. Running out of resilience was part of everyday work in the ICUs, as evidenced by: (i) the lack of effective treatments, which means that despite all efforts the mortality rates were very high; (ii) exhausted professionals who could not cope with the stressful working conditions; and (iii) chronic and acute mismatches between capacity and demand, which implied that many patients received sub-standard care. ICU resilience was inextricably dependent on the resilience of society at large. An integrated approach to resilient healthcare is necessary, encompassing the micro, meso, and macro levels, besides accounting for the broader societal resilience.

ABSTRACT. The extra effort of clinicians´ to provide care to their patients, which is a manifestation of resilience, usually goes unnoticed due to successful outcomes that occur most of the time. Therefore, clinicians’ self-sacrifice and overreliance on resilience become “normal work”. This study investigates the workload implications of resilient healthcare. The study setting was the radiology department of a major teaching hospital. The centrality of each clinician in four ability-based social networks (i.e., the resilience abilities of monitoring, responding, anticipating, and learning) was considered a proxy of each actor’s contribution to the overall system resilience. As such, a resilience score was calculated for each actor in each ability-based network, combining five indicators theoretically connected to the actor’s resilience: in-degree, closeness, and betweenness, which are derived from social network analysis, and availability and reliability, which are non-network attributes assessed through Likert-style questions. In turn, the individual workload was assessed based on the NASA-TLX questionnaire, which produces indicators related to six dimensions of workload (in addition to an overall score): mental, physical, temporal, performance, effort, and frustration. Both questionnaires – social network analysis and NASA-TLX – were answered by 155 out of the 220 staff of the radiology department. Follow-up interviews with 10 respondents were conducted to understand the influence of contextual factors on the actors´ workload and their resilience score based on network centrality. Preliminary data analysis indicates that the overall workload of the actors with the higher and lower resilience scores is similar, although there are significant differences in the workload dimensions (e.g., actors with higher resilience scores are subject to higher temporal and mental demand in comparison to those with lower resilience scores). As another finding, mental demand was the major workload dimension significantly correlated (95% confidence interval) to the resilience score. Furthermore, actors that take the initiative to help their coworkers without being requested had a higher overall workload than those that are more passive – the frequency at which actors take the initiative to help others was assessed by a specific question in the social network analysis survey. Overall, our findings indicated that social interactions related to the four resilience abilities mattered to the workload of the professionals surveyed. The adopted research method is expected to be replicable to other settings. It might be a new approach for the analysis of the human costs of resilient performance. The workload is proposed as a proxy of that human cost.

ABSTRACT. Previous studies on the role of intuition in experts’ decision-making allowed to discover strategy experts use to make correct decisions in situations in the absence of clear facts. This study aims to discover how expert executive search consultants know when to trust their intuition and how intuition influences their process of reasoning and decision-making. Six executive search consultants from Warsaw and London took part in the study (average experience = 13 years). Results of the study indicated that consultants use intuition to provide an initial appraisal of a candidate. Intuitive answer was found to be accompanied with a Feeling of Rightness (FOR) whose intensity was prompting the level of confidence in one’s intuition. Based on results, a decision-making model was created that depicts how FOR influences consultants’ process of reasoning and decision-making.

ABSTRACT. An Adversarial Collaboration to Explore the Value of Unnatural Cognitive Acts

One of my favorite scientific papers is the Kahneman and Klein 2009 article: A failure to disagree. Together they scoped out the conditions under which it is reasonable to expect the emergence of deeply intuitive, pattern-recognition forms of expertise and those under which it is not. The key moderator variable was the task environment: the degree to which people get clear rapid feedback on the correctness of their hunches. Intelligence analysts and geopolitical pundits rarely get clear, rapid feedback.

In my 2005 book Expert Political Judgment (EPJ), I studied professionals who work in learning-unfriendly settings: experts on world politics who offer opinions on big topics. The professionals I studied in EPJ were highly accomplished and intelligent people who could build powerful cases for the positions they staked out. But expertise did not translate into predictive power.

My 2015 book Superforecasting explores what happens when you introduce a profoundly disruptive mode of thinking, subjective-probability geopolitical forecasting tournaments that give participants rapid clear feedback on the correctness of their views. Progress in geopolitical forecasting required disrupting experts’ preferred natural modes of expressing uncertainty: make explicit probability judgments of events that many thought were beyond quantitative probability estimation.

We showed that people could learn to make increasingly well calibrated and discriminating judgments. There were big and systematic individual differences among forecasters. It was possible to develop training systems that boosted performance in a series of randomized controlled experiments.

So, my modest proposal to this conference is that in learning unfriendly environments it is sometimes beneficial to introduce disruptive interventions that pressure experts to engage in unnatural cognitive acts.

ABSTRACT. This study presents a questionnaire survey, designed by a resource-centric framework, about how Japanese home-visit nursing stations and nursing care facilities prepared and coped with the COVID-19 pandemic. We analyzed the answers using qualitative analysis to develop categories of good and typical practices to cope with pandemic situations. Based on the analysis results and discussions, we developed five categories of preparedness and countermeasures for resource management: protection, utilization, stockpiling, procuring, and repairs. Furthermore, we provided the categories with practice examples for human resources, which can be used as a reference when preparing new business continuity plans (BCPs).

ABSTRACT. The future military operating Environment (FOE) is likely to be one in which operational military units will be less dependent on traditional command and control (C2) structures, operating more as semi-autonomous, self-organising teams. The devolution of C2 functions places a greater emphasis on the need for local peer-to-peer sensemaking and decision making. The work in this paper is a result of a project looking at the idea of sensemaking as a distributed phenomenon in military settings. We discuss this changing context and, as a prelude to the principles, offer a characterisation of distributed sensemaking as a form of distributed cognition which involves the harnessing of disparate information resources. Based on a review of sensemaking and team sensemaking sources from areas including Organisational Studies, Computer Supported Collaborative Work, Naturalistic Decision Making and Human Computer Interaction, we propose nine principles for supporting distributed sensemaking based on specific identified challenges and potential interventions. The nine principles broadly relate to the provision of information, interpretation and communication.

ABSTRACT. LEOs often must make rapid decisions and actions to protect themselves and control a situation or a subject. While different agencies teach different techniques and philosophies for use-of-force and arrest-and-control, they are all enhanced by accurate and fast recognition of potentially attacking motions made by subjects. Whether subjects are civilians being questioned or suspects being detained, officers must respond with an appropriate scale and type of action. Slow or inappropriate actions can have highly negative results for subjects, officers, and their agencies. Unfortunately, training of rapid decision-action performance requires a high volume of practice that is difficult to acquire in training contexts that typically focus on “what you do” more than “what you see.”

One possible solution is to target the Recognition component for concentrated practice using the Video-Occlusion method developed in sports science. Video-occlusion was validated by reliably differentiating expert performers from less expert performers. It was then developed into video-occlusion training applications. For decades researchers have observed that video-occlusion might contribute to training rapid decision-action performance in military and law enforcement contexts. This study compared the performance of experienced law enforcement officers and less experienced officers on a video-occlusion test of Attack Recognition. Findings validated the video-occlusion method and recommended specific occlusion points for recognition-only video-occlusion training of attack recognition that can be delivered on tablets and cell phones as a way to accelerate expertise.

ABSTRACT. Helicopter or other rotorcraft operations fly almost continuously in close proximity to hazards and are just beginning to incorporate advanced automation that will rely on dynamic world models. Our work highlights the perceptual cues and strategies of human rotorcraft pilots in navigating complex airspace and identifies opportunities to support human knowledge and resilience to improve overall mission effectiveness and safety.