ICCRTS-KSCO 2017: 22ND INTERNATIONAL C2 RESEARCH AND TECHNOLOGY SYMPOSIUM / 9TH KNOWLEDGE SYSTEMS FOR COALITION OPERATIONS CONFERENCE
PROGRAM FOR WEDNESDAY, NOVEMBER 8TH
Days:
previous day
all days

View: session overviewtalk overview

09:00-09:45 Session 16: "Organized Complexity in Networks: Insights and Opportunities" - Dr. David L. Alderson, Associate Professor, Operations Research Director, Center for Infrastructure Defense Naval Postgraduate School

Organized Complexity in Networks: Insights and Opportunities

Recent efforts to develop a universal view of complex networks have created both excitement and confusion about the way in which knowledge of network structure can be used to understand, control, or design system behavior.   In this talk, I contrast the predominant perspective in the network science literature (that abstracts away domain-specific function and instead focuses on graph-theoretic measures of system structure and dynamics) with that of engineers and practitioners (who emphasize the importance of network performance, constraints, and trade-offs).  After presenting this view of “organized complexity,” I describe ongoing work and opportunities for understanding the robustness, fragility, and resilience in critical infrastructures and command and control systems.

Location: Room A
09:45-10:30 Session 17: NATO Research Panel

This panel will review the activities and products of selected recently completed, on-going, and new C2 and Cyber related research groups organized under the auspices of NATO’s Science and Technology Organization.

SAS-104 – Bjorn JE Johansson, Swedish Defence Research Agency

SAS-new group – David S. Alberts, Institute for Defense Analyses

IST-144 – Elizabeth Bowman, Army Research Laboratory

SAS-ET-DS – Elizabeth Bowman, Army Research Laboratory  

MSG-136 – Frank T. Johnsen

IST-090/118/050  Frank T. Johnsen, Peter-Paul Meiler, Marco Manso

IST-147 – Niranjan Suri

US SOCOM  NATO ACT Collaboration – Ken Teske, US SOCOM

Location: Room A
10:30-12:00 Session 18A: Poster Session
Location: Room 1
10:30
SPAWAR Unified DevOps Orchestration Engine (SUDOE) – A Tool to Simplify Orchestration of Virtual Machines and Containers
SPEAKER: Samia Ali

ABSTRACT. In the face of growing global uncertainty, the Navy, led by Chief of Naval Operations Admiral John Richardson, has structured its strategy around a “Design for Maintaining Maritime Superiority” in which the Navy “will make our best initial assessment of the environment, formulate a way ahead, and move out.” Inherent in this “design” is the ability to “continually assess the environment.” In short, the uncertainty that the Navy faces necessitates deliberate course corrections and the use of constantly evolving strategy, tactics, and tools. Central among these tools is the need to have agile command and control (C2) software systems and updates that can be rapidly deployed on any of the plethora of naval systems.

In recent years, SPAWAR Systems Center Pacific (SSC PAC) has invested significant resources into improving the Navy’s software distribution chain through the implementation of the SPAWAR Unified DevOps Orchestration Engine (SUDOE). Traditionally, the Navy has relied on highly trained engineers to perform time-consuming software installations on ships and other duty stations. SUDOE, on the other hand, follows a set of best practices and utilizes a suite of open source and custom built tools for provisioning management, code scanning, code compilation, automated software orchestration, and user friendly infrastructure controls that can be implemented remotely. SUDOE has been tailored and successfully applied to DoD projects of varying requirements and complexity -- ranging from Programs of Records to small internally funded research projects. Implementation of the SUDOE has decreased time and cost, while increasing security and quality.

10:30
IMPROVE THE BATTLEFIELD SITUATION AWARENESS THROUGH AN EFFECTIVE IM/IX FRAMEWORK
SPEAKER: unknown

ABSTRACT. The Hyperion prototype, an integration framework in support to information management and exploitation, is designed and developed under the Canadian Army (CA) G2 Capability Development to investigate options for addressing key issues in order to enable Land Intelligence Modernization. The Hyperion prototype aims to support the integration, alignment, management and exploitation of (1) data extracted from large amounts of documents (structured data) and (2) raw data (unstructured data) from many disparate external ISR sources. It is as well an exploratory platform for investigating, experimenting, testing and integrating new technologies. Technological choices made while designing the Hyperion backend are mainly: (1) no imposed data model or structure; (2) service oriented ecosystem; and (3) a system that leverages big data technologies. The fundamental improvement offered by Hyperion is the flexibility it provides to users for managing information as they see fit. Legacy systems suffer from schema rigidity or schema fragility. That is, once data relationships are set, they remain so and cannot easily be changed in conventional data management systems. Ontologies provide the flexibility and expressiveness required by the changing nature of military operations. In theory, ontologies provide solutions to many problems and that is why ontology concepts were selected to be the backbone of Hyperion. Using ontologies, Hyperion intends to be an all-source information repository that can search and access seamlessly current and archived data, provide contextualized search, and import data from legacy systems. This paper will discuss the Hyperion prototype, and will investigate some cases of its exploitation.

10:30
Knowledge Representation to Capture & Retrieve Experiential Learning in Military Settings
SPEAKER: Sarah Markham

ABSTRACT. No military plan survives execution intact. Planner rely on their experiential learning in addition to formal military education when creating a plan to fulfil a Commander’s Intent. The strategic environment is characterised by uncertainty, complexity, and rapid change, which requires persistent engagement, adaptation and learning. Passage of time between the creation of a plan and the execution of operations may result in changes to the initial conditions with respect to enemy COA, status of friendly forces, etc. Once execution begins, the predictability of enemy actions decreases the further along the timeline. This decreases the chances that the original plan will be correct and increases the likelihood that changes will have to be made. Much experiential learning is gained during this process. To optimise military operations, it is crucial that this experiential learning is captured in a manner in which it can be easily retrieved. We propose an intelligent aide to fulfil this purpose, modelled on graph-theoretic plan visualisation and an ontological representation of military operations. An embedded scripting framework is utilised to capture the experiential learning which may be shared and discussed in After Action Reviews. In addition we include an abstract argumentation tool to aid application of experiential learning to future plan development.

10:30-12:00 Session 18B: Demonstration Session
Location: Room A
10:30
Advanced NLP Analytics
10:30
Toplogy Control of Command and Control Networks
SPEAKER: Scott Rager
10:30
Multigenre Experimentation within the Network Science Research Laboratory
SPEAKER: Lisa Scott
12:00-13:00Lunch Break
13:00-15:00 Session 19A: Track 2d - C2 Concepts, Theory, Policy, and Approaches
Location: Room A
13:00
Agility, Automation, and Autonomy
SPEAKER: David Alberts

ABSTRACT. Success in military, civil-military, and other complex endeavors depends upon the ability of command and control arrangements and approaches to effectively employ automation, manage autonomy and achieve requisite levels of effectiveness, efficiency, and agility. A failure of C2 to accomplish this can lead to adverse consequences and potential mission failure. The need for agility, and for C2 Agility in particular has been well articulated. A growing body of evidence supports the importance of enhancing the agility of organizations faced with dynamic situations that are characterized by complexity and uncertainty. The importance of agility increases dramatically when organizations are operating in highly contested cyber environments. The automation of a variety of C2-related tasks has been prevalent for some time. Yet there has not been a clear recognition of what this increased automation means in terms of the constraints it places upon commanders and C2 systems. These constraints may be, under certain circumstances, impediments to effectiveness, efficiency and agility and thus, to mission accomplishment. The emergence of autonomous systems is a more recent development. From a C2 perspective, as is the case with automation, the design and operation of ‘autonomous’ systems involve delegations of decision rights whose impacts and consequences are currently not well-understood. This paper will explore the nature of the C2-related inter-dependencies between and among agility, automation, and autonomy and provide an integrated conceptual framework and set of metrics that will organize efforts to better understand these relationships and their impact on C2 and mission effectiveness.

13:30
Mission Command and Agile C2

ABSTRACT. Mission Command is a C2 framework that involves varying degrees of decentralization, potentially investing lower levels of a hierarchy—which remains essentially intact—with considerable autonomy. It involves a broadening in the allocation of decision rights. We examine Mission Command from a historical and organizational perspective, as well as analyzing its relation to the modern theory of C2 Agility. Mission Command has roots in Prussia, arguably beginning as far back as Frederick the Great. The Prussians and Germans successfully employed Mission Command in conflicts ranging from the Franco Prussian War to the early stages of World War 2. There was also a parallel naval tradition similar to Mission Command, embraced by the United States Navy in the early 20th Century but going back at least to Admiral Nelson. Since the 1950s modern military establishments have adopted Mission Command with varying degrees of success. Mission Command can be either hindered or enhanced by pervasive advanced communications and networking technology, depending on organizational culture. It requires high levels of trust within an organization, and is consistent with modern management theories that emphasize empowering subordinates. Mission Command is in some respects an inherently agile command concept. However, if Mission Command is interpreted rigidly as only broadening decision rights without concomitantly liberalizing Patterns of Interaction and Distribution of Information, its agility and effectiveness are likely to be hampered.

14:00
Research on C2 AI Problem Decomposition
SPEAKER: Xin Jin

ABSTRACT. The rapid development of Artificial Intelligence technology has led to the victory of AlphaGo, making people see the hope of intelligent military command and control. However, battlefield is quite complex, while chessboard is significantly abstracted. It is impossible to copy its method directly, as machine learning from observations directly to orders, or from orders directly to results are too difficult. This paper attempts to decompose the problem of intelligent command and control into more specific problems, and try to use different artificial intelligence methods to solve different specific problems, through the local technology breakthrough to achieve the overall level of intelligence to enhance, with a certain guiding significance to development of intelligent command and control.

14:30
The lack of convergence between C2 theory and practice

ABSTRACT. Fifteen years ago the authors presented a paper that discussed the possible impact of novel ideas and technologies for command and control (C2), such as the network centric approach and radical views on the design of command posts. Some of the fundamental ideas of what was in vogue at the time were questioned and the ``old'' was put in contrast with the ``new.'' Looking back at our thoughts as well as others that from that time and the progression of theory within the field of C2, we suggest that the status quo, what actually has been achieved, may be the worst of the two straw men worlds that we suggested at the time. While theories on C2 and C2 systems constantly are moving in the direction of system science concepts like emergence and adaptivity, most actual military practice is signified by mechanistic and reductionistic ideals. However, there are differences even within the military profession. In the paper, naval and army forces are discussed to illustrate how contextual factors have driven naval C2 in a direction that differs from the C2 of land-warfare forces. We suggest that it is necessary to: 1.) further develop adaptive approaches to the organization and conduct of military operations, 2.) develop enabling instead of controlling technologies, 3.) switch focus from structure to function, 4.) support information sharing instead of control of information, 5.) develop tools for assessing adaptive capacity in socio-technical systems.

13:00-15:00 Session 19B: Track 1b - Operational Issues: Coalition Command and Control
Chair:
Location: Room 1
13:00
Spectrum Sharing Initiatives: Federal Response to Escalating Military and Commercial Demands

ABSTRACT. Communications spectrum—that is, the set of electromagnetic frequencies suitable for communications and radar—is a precious and increasingly scarce resource. Effective sharing of spectrum, enabled by both policy and technology, will be crucial in future military operations, including coalition operations, in both peacetime and wartime. This paper reviews recent United States Federal Government activities to promote sharing of the electromagnetic communications spectrum. Spectrum sharing is not new; the orderly sharing of spectrum has been the primary purpose of spectrum management since the beginning of the practical use of radio. But spectrum sharing is being reinvented due to the escalating demand for spectrum from both commercial and government users, the inability of conventional coarse-grained and manually-based spectrum sharing methods to meet the demand, and the advent of new technologies that enable automated and fine-grained sharing of spectrum along frequency, geography, time, and other dimensions. We discuss 13 U.S. Federal Government spectrum sharing initiatives—representing both conventional and advanced sharing approaches—that are either completed or well underway. The initiatives cover the higher end of the radio spectrum, from the TV white space initiative in the Very High Frequency (VHF) and Ultra High Frequency (UHF) bands to the 60 GHz unlicensed, 70-80-90 GHz millimeter wave, and level probing radar initiatives in the Extremely High Frequency (EHF) band

13:30
Pre-Shared Key-Enabled CSfC: A Positively Disruptive Technology and Enabler for Next Generation MPE Secure Communications

ABSTRACT. Communication Security (COMSEC) is a core component of mission critical operations that not only becomes more complex, but exposes technical and operational capability gaps as environments become more complex in terms of the number and type of participants, locations and durations. Seeking an alternative to relying solely on traditional (expensive and operationally cumbersome) Type-1 Controlled Cryptographic Items (CCI), the US Joint Staff J6, Director for C4/Cyber released the US DoD Mission Partner Environment (MPE) Joining Instructions that allows for the employment of Commercial Solutions for Classified (CSfC) – a layered, COTS-based approach that has been quietly employed within the US DoD for a number of years to provide COMSEC for Classified – for access and cryptographic interfaces for MPE systems to include US Battlefield Information Collection and Exploitation System (BICES).

Two major (unidentified in the MPE Joining Instructions) problems exist with the as-is CSfC approach. First, the traditional, certificate-based CSfC approach is not quantum computer resistant so it is already an ‘end of life’ technology. Second, CSfC is a US-only approach because it is entirely based on US policies, key management and approval processes that do not have authority and/or support outside the US.

Addressing these issues, this paper proposes a positively disruptive technology (quantum computer resistant Pre-Shared Key (PSK) based CSfC) that not only address US-provided MPE / BICES CSfC issues, but also allows external entities a self-deterministic option whereby they can employ US-provided and/or develop their own PSK-based CSf[ac]C COMSEC (PCacC) where [ac] could be any coalition-like entity.

14:00
Protecting Information Sharing Systems with Commercial Solutions for Classified Encryption
SPEAKER: Mark Miller

ABSTRACT. Abstract With the increase in research and development of Quantum computers to break current encryption devices, there’s a critical need to ensure information sharing systems are protected. The Department of Defense (DoD) and all Mission Partners should develop Commercial Solutions for Classified (CSfC) capabilities and options as an alternative to using traditional Type-1 encryption for some users and scenarios. CSfC provides the user the ability to secure their information systems and networks in the Cyber arena to enhance mitigation of future quantum computing threats. Applying and evolving a framework approach discussed in the 19th ICCRTS, will show the importance of comprehending each mission partner’s CSfC efforts, whether from Special Operations and Conventional Forces, or a Ministry, Bureau, or Agency to protect against Cyber threats. To accomplish this, we look at four principles: Common vision, goals and objectives for the mission; Common understanding of the situation; Coordination of efforts to ensure coherency; and Common measures of progress to change course or direction as needed. We must analyze DoD’s and Mission Partner’s Cyber approaches, best practices, and lessons learned to identify common goals, areas of interest, capabilities, and common categories of effort to address common focus areas for CSfC.

14:30
Improving C2 Alignment and Integration
SPEAKER: Ken Teske

ABSTRACT. Coalition and Mission Partner Command and Control (C2) must develop options and capabilities that enhance inter-dependence and further their alignment to increase everyone’s effectiveness and understanding, while reducing mission risk. Alignment as well as integration continue to be an absolute necessity and must look at the following four principles: Common view with goals and objectives, Common understanding of capabilities and lexicons, Alignment of efforts to ensure coherency, and Assessment to change course or direction as needed. Both alignment and integration will help address the challenges associated with operating in today’s ever changing environment. Applying and evolving a framework approach that was discussed in the 19th ICCRTS, paper (003) and 21st ICCRTS, paper (001) demonstrates the importance of comprehending that each mission partner’s input weather from Special Operations, Conventional Forces, from a Ministry, Department, Bureau, or Agency is important to better understand and address C2 problems, issues, and potential solutions. We must use their lessons learned, best practices, approaches, and strategies identifying common goals, areas of interest, capabilities, and common categories of effort applied by each of the organizations as a focus to advance C2. This framework methodology will continue to directly impact the way C2 is conducted both today and in the future.

13:00-15:00 Session 19C: Track 5b - Highly Connected, Automated, and Autonomous Forces
Location: Room 2
13:00
Exploring Trust and Autonomy in a Hybrid Human/Robotic Force
SPEAKER: Michael Hieb

ABSTRACT. Automated agents are increasingly entering Command and Control (C2) organizational structures of both first responders and armed forces. As such agents become more sophisticated, they may become integral parts of mission teams and be tasked with commanding other agents and even humans. This raises critical issues associated with the allocation of decision rights to automated agents, and the levels of trust that will be required. Problems include measurement of trust in the context of mixed human and robotic (“hybrid”) teams, and the propagation and updating of decision rights throughout such teams. Current work on C2 for autonomy is mostly aimed at single entities and simple control schemes. Our approach is to use a commercial mission simulation, VR-Forces by MAK technologies, to perform virtual assessments of trust in a mission scenario involving driving an ambulance through an urban war zone to a field hospital. Our initial experiment examines how trust evolves when humans are commanded by either a human or autonomous robot. As it turns out, humans are more willing to obey an autonomous robot than a human in this specific experiment. While our experiment does not begin to answer all relevant questions surrounding trust in a hybrid team, it does point to the possibility of using well-controlled virtual environments to thoroughly explore different aspects of trust. This experimentation can then guide design of automation and C2 structures.

13:30
Leveraging Command and Control with Automated Planning

ABSTRACT. Nowadays, military forces have been employed in many types of operations, from conventional combat to population support. Furthermore, massive data and information volatility have hindered the authority decision-making process, once he seeks for the maximum synergy among his available resources. Hence, making quick and right decisions is one ability which has been required from military authorities. During the C2 loop, decision-makers must efficiently use the available resources regarding the missions, under restricted boundaries of time and costs. The very first issue to solve in large scale Hybrid Forces is how to select the proper agents and their capabilities. First of all, to be able to select them, we need to be able to check mission feasibility before action begins. If the available capabilities are not enough for accomplishing the mission, using them anyway would produce time and resources misuse, leading to decision-making inferiority. In a further step, given the levels of autonomy, we need to be able to detect and decentralize independent objectives. Finally, the selection of just the essential resources combined with the mission and objective evaluation performs an interesting aid to decision-making loop, which leads to decision-making superiority. Towards this contribution, this work exploits Artificial Intelligence research areas in order to propose a new planning model which can be applied not only in military operations but also in academic field. This model is based on infeasibility and independent detection, agent selection and task delegation which may lead to a lower volume of messages and a quicker response time.

14:00
Improving Integration between Tactical and HQ Levels by making SOA applicable on the Battlefield

ABSTRACT. NATO Research Task Group IST-118 provided concrete recommendations for the application at the tactical level of a subset of the Service Oriented Architecture (SOA) based HQ-level core services from the NATO C3 Taxonomy, based on systematic testing and evaluation.

The SOA paradigm is used by NATO to achieve interoperability at the (HQ) information infrastructure level. Currently applied technologies (e.g. Web services) were not designed for tactical networks. This frustrates interoperability and integration between tactical and HQ levels and is not cost-effective. IST-118 investigated the application of the core services to the tactical domain and provided recommendations for their deployment, based on experiences and experiments. These recommendations support the development of SOA at the tactical level, thus improving integration between tactical and HQ levels. This approach also diminishes the need to develop and implement separate HQ and tactical versions of the same functionalities, thus reducing cost, both of required materiel, R&D and training.

We showed the advantage of cross-layer middleware, enabling adaptation of the services’ communication behavior to the special needs of tactical networks and enabling parameterization of the network to fulfil the services‘ communication requirements. We also pursued video- and text-based services, contributing to their use on the battlefield. Technology demonstration events show that we reached TRL 4 (close to TRL 5).

Topics: 5 Highly Connected, Automated, and Autonomous Forces (Main topic). 6 Interoperability, Integration and Security (Related topic). 8 Methodology, Experimentation, Analysis, Assessment and Metrics (Related topic). 9 Battlefields of the Future (Related topic).

14:30
MIP-Solutions for Civil-Military Cooperation
SPEAKER: unknown

ABSTRACT. The Multilateral Interoperability Programme (MIP) – a multinational military standardization endeavour with 24 member nations, EDA and NATO – provides solutions for achieving semantic interoperability between military C2 systems. The latest products are the MIP 4 Information Exchange Specification (MIP4IES) and the constantly refined MIP Information Model (MIM). MIP solutions address military information exchange, primarily in land-based operations. However, the focus can be extended. Until mid-2017, the European Defence Agency (EDA) sponsored a project on the demonstration and evaluation of MIP solutions for Civil-Military Cooperation (CIMIC), exemplified by two capabilities, namely Convoy Protection and Transportation (‘C2 Information Interoperability Demonstration and EDA Contribution to MIP’, contract no 15.CAT.OP.096). Within the project, a MIP-based interoperability solution for these capabilities was defined and implemented. In this paper, we will describe the entire process of defining and implementing the solution, The process covers the solution from (a) the description of the capabilities, via (b) the definition of respective Information Exchange Requirements (IERs), (c) the definition of domain models for the capabilities, (d) the specification of message formats based on the models, (e) the selection and adaption of appropriate exchange mechanisms, to (f) the implementation of message formats and exchange mechanisms within two information systems – one for the civilian and one for the military side. We will show which extensions and adaptions of the MIP products are required to provide the solution for the CIMIC domain and conclude which further developments of the MIP products are considered necessary.

13:00-15:00 Session 19D: Track 8b - Methodology, Experimentation, Analysis, Assessment and Metrics
Chair:
Location: Room 3
13:00
Emergent Display Technologies: Developing Use-Case Prototypes for Military Command Teams in Virtual Environments
SPEAKER: James Karle

ABSTRACT. The complex operational environments that military command teams are faced with demand the filtration and consumption of significant amounts of information, the volume of which is predicted to only increase in coming years. A number of emergent display technologies (EDT), including augmented reality (AR) and virtual reality (VR) systems, provide novel methods of organizing and presenting this information. These new technologies can be difficult to assess for their potential value to command teams, as they are often at varying levels of technological readiness, the application environments they are intended for are inaccessible, and/or the devices are impracticable to modify at the physical or software levels. Simulations of EDTs in virtual environments (VEs) provide a moderate to high-fidelity, low-cost, and easily implemented solution to these limitations. As such, we developed a process for the rapid generation and assessment of EDT related use-case prototypes in a VE. In addition, using this process, we identified novel solutions for current and predicted near-future challenges encountered by Royal Canadian Navy command teams as associated with the management and exploitation of information. The capabilities and limitations of using VEs in this manner are described when leveraging them as tools for rapid-prototyping.

13:30
OODAWN in the MER: from distributed Situation Awareness to Decisions and Actions in operational C2 using weighted network theory
SPEAKER: unknown

ABSTRACT. We present a weighted network model for representing the distributed nature of activities across the spectrum of the Boyd OODA loop in the deployed operational environment. The model, denoted the OODA-Weighted-Networks (OODAWN), is a generalisation of the SAWN model presented previously at ICCRTS by some of the authors. Essentially, the approach develops a ‘social network’ – where some people-people interactions may be mediated by information technology artefacts – but weights links according to the function played by the information conveyed in the link according to an eight scale OODA model. After detailing the data collection approach, we apply to the model to data from headquarters across the Australian Defence Force Middle-East-Region (MER) of operations. Over and above showing network diagrams, we apply quantitative social network analysis to complement the visual insights the method provides.

14:00
Human Information Interaction in a Joint Electro-Magnetic Spectrum Coordination Cell
SPEAKER: unknown

ABSTRACT. In the summer and fall of 2016, the Canadian Forces Warfare Centre (CFWC) conducted experiments into the conduct of the joint targeting process with the use of non-munitions capabilities. Specifically, these first two experiments studied and trialed the adaptation of the process to electronic warfare (EW). To conduct this examination, the CFWC, with help from the Australian Defence Science and Technology Group, simulated the operation of a Joint Electro-Magnetic Spectrum Coordination Cell (JEMSCC) in an experiment called the Joint Non-Munitions Effects Experiment (JNEX). In JNEX, operators in the JEMSCC processed lines of bearing (generated from a constructive simulation) to produce tactical reports of the geo-location of emitters and network diagrams describing the relationship between the received signals. This information was then used to produce an electronic attack plan and a collateral effects estimate to develop a course of action for the joint targeting decision making process. The fidelity of the EW synthetic environment not only provided the operators with a realistic level of EW communication reports, but also simulated mission failure if mistakes were made. This paper discusses the quality and effectiveness of the information exchanges that took place, and the targeting products that were produced as part of the joint targeting process for EW during JNEX. Furthermore, the paper discusses the experiment design lessons learned and the challenges that will be faced in the next phase of the JNEX campaign, which will look at the effects of cyber warfare in the joint targeting process.

14:30
Mission-Relevant Criteria for Naval Task Group (NTG) Course of Action (COA) Analysis
SPEAKER: unknown

ABSTRACT. Defence Research & Development Canada – Atlantic Research Centre is developing a prototype planning and course of action (COA) testbed for naval command teams. As part of this effort, a study was conducted to understand how criteria for the assessment of Naval Task Group (NTG) courses of action are selected, defined, prioritized and applied. Fourteen naval officers with operational planning experience identified and defined a set of criteria for assessing COAs for four NTG missions. The responses of this first data collection activity were consolidated, categorized, and mapped to the Canadian Forces’ Principles of War, to create a short list of relevant criteria for each mission. In a follow-on data collection activity, naval officers were provided the refined criteria lists and, both individually and in groups, completed activities focused on defining the relative importance of the criteria to each mission. The results of the study are presented and framed from the perspective of their application to the COA testbed (COAT).