Multi-National Experiment 4 on Effects Based Approach to ...

Multi-National Experiment 4 on Effects Based Approach to Operations CFEC Analysis Report

Philip S. E. Farrell, Dave Allen, Paul Burrows, Paul Comeau, Steven Hughes, John Kachuik, Paul Labbé and Fred Lichacz

Defence R&D Canada --- Ottawa TECHNICAL REPORT

DRDC Ottawa TR 2006-230 December 2006

Multi-National Experiment 4 on Effects Based Approach to Operations CFEC Analysis Report

Philip S. E. Farrell Dave Allen Paul Burrows Paul Comeau Steven Hughes John Kachuik Paul Labbé Fred Lichacz

Defence R&D Canada – Ottawa Technical Report DRDC Ottawa TR 2006-230 December 2006

Primary Author Original signed by Philip S. E. Farrell

Philip S. E. Farrell, Ph.D.

Approved by Original signed by R. J. Chekan

Col R.J. Chekan

Commandant, CFEC

Approved for release by Original signed by C. A. Boulet

C.A. Boulet

Chair, Document Review and Library Committee

This document was produced by CFEC and published by DRDC. CFEC document reference number: CFEC TR 2006-001

© Her Majesty the Queen as represented by the Minister of National Defence, 2006

© Sa majesté la reine, représentée par le ministre de la Défense nationale, 2006

DRDC Ottawa TR 2006-230 i

Abstract

This report documents the Canadian analysis within Multinational Experiment 4 (MNE 4), and addresses the three Canadian Objectives that included the relevance of an Effects-Based Approach to Operations (EBAO), assessment of Multinational Interagency Group (MNIG) and Knowledge Support (KS) concepts for possible application to the Canadian Forces (CF) and other Canadian government departments. The report introduces the EBAO, MNIG, and KS concepts, provides an overview of the experimental design for MNE 4, and specifies the key Critical Operational Issues that were relevant to the Canadian Objectives. The results are summarized for each relevant COI. The discussion section addresses the Canadian Objectives in light of Canada’s International Policy Statement as well as MNE 4 assessments. The report concludes with a series of implications and recommendations for the CF.

The main recommendation is for Canada to continue being engaged in EBAO research, concept development, and experimentation. It is critical that Canada keep abreast of the latest ideas and thoughts on this concept as our major allies (US and NATO) move toward adopting EBAO ideas. Moreover, it is now time to think about “Canadianizing” the EBAO concept such that it addresses Canada’s unique challenges and exploits unique opportunities.

Résumé

Le présent rapport documente l’analyse réalisée par le Canada dans le cadre de l’Expérimentation multinationale 4 (EMN 4) et traite des trois objectifs canadiens en relation avec ce projet. Ces objectifs consistaient à évaluer la pertinence d’une approche des opérations fondées sur les effets (AOFE) ainsi que les concepts liés au Groupe multinational inter-agences (GMIA) et au Soutien des connaissances (SC) en vue de leur application possible au sein des Forces canadiennes (FC) et d’autres ministères du gouvernement canadien. Le rapport présente les concepts relatifs à l’AOFE, au GMIA et au SC, donne un aperçu du schéma expérimental qu’utilise l’EMN 4 et décrit les enjeux opérationnels critiques s’appliquant aux objectifs du Canada. Un résumé des résultats relatifs à chaque enjeu pertinent est fourni. La section sur la discussion traite des objectifs canadiens à la lumière de l’Énoncé de politique internationale du Canada ainsi que des évaluations de l’EMN 4. Le rapport se termine par une série de conséquences et de recommandations pour le bénéfice des FC.

La recommandation principale issue du rapport est que le Canada doit continuer de participer à la recherche sur l’AOFE ainsi qu’au développement du concept et à l’expérimentation. Il est essentiel que le Canada se tienne au courant des derniers développements entourant ce concept, au moment où ses principaux alliés (É.-U. et OTAN) adoptent les idées découlant de l’AOFE. De plus, il est maintenant temps de songer à « canadianiser » le concept de l’AOFE de sorte qu’il permette de relever les défis uniques du Canada et d’exploiter les possibilités particulières qui se présentent.

ii DRDC Ottawa 2006-230

This page intentionally left blank.

DRDC Ottawa TR 2006-230 iii

Executive summary

The Canadian Forces Experimentation Centre (CFEC) participated in the design, execution and analysis of Multi-National Experiment 4 (MNE 4) on the Effects Based Approach to Operations (EBAO) concept. The United States (US) Joint Forces Command (JFCOM) led MNE 4 and invited Australia (AU), Canada (CA), Finland (FI), France (FR), Germany (GE), Sweden (SE), the United Kingdom (UK) and NATO to participate in this instalment of the MNE series. MNE 4 was a concept refinement experiment and its primary objective was to assess and refine the EBAO Process, Organization and Technology (POT) developed to support an effects-based operation conducted by a Coalition Task Force (CTF) as well as a NATO Response Force (NRF) using the Combined Federated Battlelab Network (CFBLNet) as a secure means to communicate, share information, and carry out the EBAO process.

MNE 4 planning began shortly after the MNE 3 event in Feb 2004. Several planning conferences, workshops, and Limited Objective Experiments (LOE) took place over a two-year period culminating in the three-week event from 27 February to 17 March 2006. The military coalition and interagency-based EBAO was conducted by partner nations over the CFBLNet. The final phase of MNE 4 included preparations and synthesis for a Senior Leadership Seminar as well as the completion of the US JFCOM Proceedings and Final Reports that contain the EBAO assessments and makes recommendations for the refinement of the EBAO concept.

This report focuses on the Canadian analysis within MNE 4, the assessment of EBAO areas for which CFEC was responsible, and the implications of key EBAO areas for the Canadian Forces and other government departments in Canada. The reporting objectives were to:

1. Assess the EBAO concept and how it could be applied in a Canadian context, for example by Canada Command (Canada COM), Canadian Expeditionary Force Command (CEFCOM), Special Forces Command (SOFCOM), Standing Contingency Task Force (SCTF) and Canadian Operational Support Command (CANOSCOM);

2. Understand the Multinational Interagency Group (MNIG) concept and determine any common linkages to the emerging Canadian Defence, Diplomacy, and Development (3-D) strategy; and

3. Analyze Knowledge Base Development (KBD) and Knowledge Management (KM) and provide recommendations on the way ahead for Knowledge Support to operations in the Canadian Forces and other government departments.

In order to address these objectives, the CFEC analyst team produced the following analyses:

a. Technology (ANNEX B) b. Knowledge Base Development and System of Systems Analysis (ANNEX C) c. Knowledge Management (ANNEX D) d. Culture Analysis (ANNEX E) e. Common Intent (ANNEX F)

Likewise, the following assessments from the MNE 4 analyst team were reviewed in detail:

f. EBAO process and organization (JFCOM, 2006b; JFCOM, 2006c) g. Multinational Interagency Group (JFCOM, 2006b; JFCOM, 2006c)

iv DRDC Ottawa 2006-230

Addressing Canadian Objectives for MNE 4

In addition to the multinational experiment objectives stated in (JFCOM, 2006a), the CFEC MNE 4 national experimental directive (Gibbons, 2006) outlines three objectives for Canada’s involvement in MNE 4. Each objective is presented below followed by short summaries of insights and how that objective is addressed within the report.

1. Contribute to EBAO concept refinement and assessment, and analyse elements of the concept that Canada and the CF, especially Canada COM, CEFCOM, SOFCOM, SCTF, CANOSCOM as well as the Standing Contingency Force (SCF) Project could apply to achieve integrated strategic effects in operations.

o The MNE 4 EBAO Process assessments were reviewed (JFCOM, 2006c) and the robust areas of the concept of operations were conceptually applied within a Canadian context. The iteration of the EBAO planning, execution, and assessment functions until the Current State converges onto the desired End State seems to be a robust concept rooted in Control Theory. Within each of the EBAO functions there is room for further refinement and streamlining of the number of process tasks and review of who is responsible for the tasks. In general, the EBAO process played in MNE 4 requires further work but serves as a useful guide for ongoing efforts to develop a CEFCOM EBAO process.

o The MNE 4 EBAO organization adopted for the experiment’s Coalition Task Force Headquarters structure requires further development (JFCOM 2006c). Statements of the required competencies, authorities, and responsibilities were not developed for the event, and thus these roles and duties evolved over the course of the event (see ANNEXES E and F). Further development and validation of the EBAO staff structure and related staff duties based on the four EBAO functions is required before any aspect of the structure could be considered for incorporation into the staff structure of real-world operational level headquarters.

o The MNE 4 EBAO computer network was not robust due to bandwidth limitations coupled with software applications not being specifically designed and integrated for this network architecture and subsequent high usage during the experiment (JFCOM, 2006c; ANNEX B). Nevertheless, there were no indications that the technology could not be made robust given enough resources and time for a proper technical design and integration work. For example, EB TOPFAS, a planning tool developed by NATO, has since been deployed to support NATO operations in Afghanistan. To support Canadian Joint and Interagency operations in future, a common network outfitted with planning and collaboration applications similar to what was employed in MNE 4 would be essential. The software applications that support EBAO are maturing every year and Canada needs to monitor their development and consider adoption of appropriate tools as CF command and control information systems evolve.

2. Explore and establish the relationship between the emerging Canadian Defence, Diplomacy, and Development “3-D” or “whole of government” strategy, and determine whether EBAO represents a means of applying the 3-D approach.

DRDC Ottawa TR 2006-230 v

o The Multinational Interagency Group concept was developed and employed in MNE 4. MNIG’s basic function was to coordinate Other Government Departments (OGDs) and, where applicable, Non-Government Agencies, and then liaise with the military headquarters. The MNIG did not have the same level of resources and reach of the military headquarters (JFCOM, 2006c). This lack of resources became an impediment in supporting the achievement of desired actions, effects, and the desired End State. This is particularly true since the End State had a diplomatic and development elements. Nevertheless, the MNIG demonstrated great potential to enhance military-non-military coordination in a complex mission area.

o The concept of the whole of government coordination to achieve a common End State requires commitment from the highest levels of government. This is equally true in Canada. Stand-up of organizations such as the Foreign Affairs Canada Stabilization and Reconstruction Task Force (START) indicate that the Canadian government is taking active steps to move the “whole of government” concept closer to reality. At the same time, the lack of resources and the reach into various government organizations are a growing concern within Canada. The complexity of the 3-D whole of government approach and the MNIG concept increases when countries are involved in coalition operations. Even if there are international agreements in place to interact government to government, there may not be a high degree of interoperability (connectivity) within each government between the military and their civilian colleagues. Military and non-military interoperability within each country and between countries engaged in multinational operations would be critical for the successful implementation of the MNIG concept.

3. Develop and refine a Multinational Knowledge Management Plan for a Coalition Task Force Headquarters or NATO Response Force Headquarters that can be tailored and applied to meet national operational requirements.

o The assessment of KBD showed that this function did not produce a holistic understanding of the operational environment (JFCOM 2006b; ANNEX C). The assessment of the KM Plan showed that it failed to appropriately support the full range of knowledge activities required for effects-based operations (ANNEX D). Thus, the combined Knowledge Support (KBD and KM) concept needs to be revisited at the conceptual level. Some of the basic definitions underpinning the Knowledge Support concept still need to be resolved: for example, what are data, information, and knowledge? What is a knowledge base? Is it different from an information database? Where does Information Management fit relative to KM and KBD?

o The implication to Canada is that more concept development and research needs to be done on the notion of Knowledge Support (KS) to operations. Knowledge Support would include Information Technology management, Information Management, and more broadly Knowledge Management. Information Technology would manage the software and hardware components and tools required for communication and data connectivity. Information Management would include managing information applications, connecting information databases, information distribution, information search and request, and information display. Knowledge Management would assist in

vi DRDC Ottawa 2006-230

shaping organizational culture that would ultimately help each staff member develop their individual knowledge required for them to perform their job such as through information access, analysis, formalization, and categorization. System of Systems Analysis and Intelligence would have predominant roles in assisting with information analysis, formalization, and categorization. KM may take the form of specific training, business rules, library services, job aids, and liaison functions between subject matter experts and other organizations.

EBAO Concept Recommendations

Develop a Canadian EBAO concept and supporting process, organizations and technologies that is suited to Canada’s strengths and opportunities, and addresses Canada’s challenges while maintaining interoperability with the methodologies of allies and partners.

Continue collaboration with allies and partners of refinement of multinational EBAO process, organization and technology through opportunities such as the MNE series.

Track progress of allies, partners and industry in developing software applications in support of EBAO process and methodologies.

Adopt the concept of a planning, execution, and assessment cycle that is adaptable and responsive, and repeats until the Current State reaches the End State.

Conduct an experiment that specifically examines EBAO effectiveness as compared to current practice.

MNIG Recommendations

Conduct a series of events where Canada’s diplomatic, defence, and development communities can develop concepts and workable solutions on how they would work together in a domestic operation as well as part of a multinational coalition.

KS Recommendations

Develop a conceptual model of Knowledge Support that is theoretically defensible and testable.

Continue developing and testing KM concept and plan for Canadian operational use.

Experimentation Recommendations

Ensure experimental objectives are clearly articulated, testable and have a sponsor.

Maintain control over experimental hypothesis, design, execution, and reporting.

Farrell, P.S.E., D. Allen, P. Burrows P. Comeau, S. Hughes, J. Kachuik, P. Labbé, F. Lichacz. 2006. Multi-National Experiment 4 on Effects-Based Approach to Operations: CFEC Analysis Report. DRDC Ottawa TR 2006-230 Defence R&D Canada – Ottawa.

DRDC Ottawa TR 2006-230 vii

Sommaire

Le Centre d'expérimentation des Forces canadiennes (CEFC) a pris part à l'élaboration, à la mise en œuvre ainsi qu'à l’analyse de l’Expérimentation multinationale 4 (EMN 4) sur le concept d’approche des opérations fondées sur les effets (AOFE). Le U. S. Joint Forces Command (U.S. JFCOM), à la tête de l’EMN 4, a convié l’Australie (AU), le Canada (CA), la Finlande (FI), la France (FR), l’Allemagne (DE), la Suède (SE), le Royaume-Uni (UK) et l’OTAN à participer à cette partie de la série d’EMN. L’EMN 4 consistait en une expérimentation d'amélioration de concept, et son objectif principal consistait à évaluer et améliorer le processus, l'organisation et la technologie (POT) associés à l'AOFE mise en place pour aider au déroulement d'opérations fondées sur les effets menées par une force opérationnelle de coalition (FOC) ainsi que par une Force de réaction de l'OTAN, cela en ayant recours au réseau fédéré multinational de laboratoires de combat (le CFBLNet) pour pouvoir communiquer, échanger de l'information et mener à bien le processus lié à l’AOFE en toute sécurité.

La planification de l'EMN 4 a débuté peu de temps après l'EMN 3, en février 2004. Plusieurs réunions de planification, ateliers, et essais à objectif limité (EOL) se sont échelonnés sur une période de deux ans, aboutissant à l’EMN elle-même, étalée sur les trois semaines allant du 27 février au 17 mars 2006. La coalition militaire et l'AOFE inter-organisationnelle ont été dirigées par des nations partenaires dans le cadre du CFBLNet. La dernière phase de l'EMN 4 comprenait les préparatifs et une synthèse en vue d'un colloque pour les hauts dirigeants, ainsi que le parachèvement des rapports définitifs et des opérations du U.S. JFCOM, dans lesquels figurent les évaluations de l'AOFE et les recommandations formulées à propos de l'amélioration du concept de l'AOFE.

Ce rapport porte essentiellement sur l'analyse canadienne dans le cadre de l’EMN 4, sur l'évaluation des volets de l'AOFE relevant du CEFC, et sur les conséquences des principaux volets de l'AOFE pour les Forces canadiennes et d’autres ministères au Canada. Les objectifs en matière de préparation de rapports consistaient à :

1. Évaluer le concept d’AOFE et la manière dont il pourrait être appliqué au contexte canadien, par exemple, par le Commandement Canada (COM Canada), le Commandement de la Force expéditionnaire du Canada (COMFEC), le Commandement des Forces d’opérations spéciales du Canada (COMFOSCAN), la Force opérationnelle permanente de contingence (FOPC), et le Commandement du soutien opérationnel du Canada (COMSOCAN);

2. Comprendre le concept du Groupe multinational inter-agences et définir les liens communs avec la nouvelle approche préconisée par le Canada en matière de défense, de diplomatie et de développement (3-D);

3. Analyser le Perfectionnement des connaissances (PC) et la Gestion du savoir (GS), en plus de formuler des recommandations sur la voie à suivre sur le plan du Soutien des connaissances dans le cadre d'opérations menées par les Forces canadiennes et par d'autres ministères fédéraux.

Pour pouvoir atteindre ces objectifs, l'équipe d'analystes du CEFC a préparé les analyses suivantes :

viii DRDC Ottawa 2006-230

a. Technologies (Annexe B) b. Perfectionnement des connaissances et système d'analyse de systèmes (Annexe C) c. Gestion du savoir (Annexe D) d. Analyse de la culture (Annexe E) e. Intention commune (Annexe F)

De plus, les évaluations suivantes effectuées par l'équipe d'analystes de l’EMN 4 ont été examinées dans les moindres détails :

f. Processus et organisation de l'AOFE (JFCOM, 2006b; JFCOM, 2006c) g. Groupe multinational inter-agences

Répondre aux objectifs du Canada par rapport à l’EMN 4

En plus des objectifs d'expérimentation multinationale énoncés dans (JFCOM, 2006a), la directive expérimentale nationale du CEFC dans le cadre de l'EMN 4 (Gibbons, 2006) expose trois objectifs concernant la participation du Canada à l’EMN 4. Chaque objectif est présenté ci-après sous la forme de court descriptif des idées et de la manière dont l'objectif en question est traité dans le rapport.

1. Contribuer à l'amélioration et à l'évaluation du concept AOFE, et analyser les éléments du concept selon lesquels le Canada et les FC, et en particulier le COM Canada, le COMFEC, le COMFOSCAN, la FOPC, le COMSOCAN ainsi que le Projet de Force opérationnelle permanente de contingence (FOPC), pourraient s'appliquer pour parvenir à l'intégration des effets stratégiques dans les opérations.

o Les évaluations du processus lié à l’AOFE dans le cadre de l'EMN 4 ont été examinées (JFCOM, 2006c), et les volets robustes du concept des opérations ont été appliqués au contexte canadien d'un point de vue conceptuel. La répétition des activités de planification, de mise en œuvre et d'évaluation de l'AOFE jusqu'à la convergence de l'État actuel vers l’État final souhaité semble être un concept adéquat enraciné dans la théorie du contrôle. En ce qui concerne les fonctions de l'AOFE, il est possible d'améliorer et de rationaliser davantage le nombre de tâches du processus et de revoir l'attribution de la responsabilité des tâches en question. En règle générale, le processus lié à l’AOFE qui a servi dans le cadre de l'EMN 4 doit être retravaillé, mais s'avère utile pour orienter la poursuite des efforts de développement dans le cadre d'un processus lié à l’AOFE du COMFEC.

o L'organisation de l’AOFE retenue pour l’EMN 4 en ce qui a trait à la structure du quartier général de la force opérationnelle de coalition responsable de l'expérimentation doit être élaborée davantage (JFCOM, 2006c). Les énoncés des compétences requises, des pouvoirs, et des responsabilités n'ont pas été préparés pour l'occasion et, par conséquent, ces rôles et fonctions ont évolué au cours de l'événement (Annexes E et F). Il est nécessaire d'élaborer et de valider davantage la structure de personnel de l'AOFE et le service de l'état-major connexe en se basant sur les quatre fonctions de l'AOFE avant de pouvoir envisager d'intégrer le volet structure à la structure de personnel d'un état-major opérationnel dans le monde réel.

DRDC Ottawa TR 2006-230 ix

o Le réseau informatique utilisé pour l’AOFE dans l’EMN 4 n'était pas robuste à cause des limitations de la bande passante, conjuguées aux applications logicielles qui n'étaient pas vraiment conçues et intégrées pour ce genre d'architecture de réseau, et à la surutilisation qui en a découlé pendant toute l'expérimentation (JFCOM, 2006c; Annexe B). Néanmoins, rien n’indiquait qu’il ne serait pas possible de renforcer cette technologie si l’on disposait de suffisamment de ressources et de temps pour réaliser une conception technique et un travail d'intégration convenables. Par exemple, EB TOPFAS, un outil de planification conçu par l'OTAN, a depuis été déployé dans le cadre d'opérations de l'OTAN en Afghanistan. Pour aider à l'avenir les opérations interarmées et interinstitutions menées par le Canada, un réseau commun doté d'applications de planification et de collaboration semblable à celui utilisé par l'EMN 4 sera indispensable. Les applications logicielles servant à l'AOFE vieillissent au fil des ans, et le Canada se doit de surveiller leur évolution et d'envisager l'adoption d'outils adaptés à mesure qu'évoluent les systèmes d'exploitation et de contrôle de l'information des FC.

2. Étudier la nouvelle approche « 3-D » (défense, diplomatie et développement) ou « approche pangouvernementale » du Canada, déterminer les interrelations qui y sont associées, et décider si l'AOFE constitue ou non un moyen de mettre en application l'approche 3-D.

o Le concept de Groupe multinational inter-agences a été conçu et utilisé dans le cadre de l'EMN 4. Le rôle premier de ce Groupe consistait à coordonner les autres ministères du gouvernement et, au besoin, les organisations non gouvernementales, puis à assurer la liaison avec les quartiers généraux militaires. Le Groupe multinational inter-agences ne disposait pas du même niveau de ressources et d'accès que le quartiers généraux militaires (JFCOM, 2006c). Ce manque de ressources est devenu un obstacle au moment d'aider à la concrétisation des actions et des effets souhaités, et à l'aboutissement de l'État final recherché. Cela s'applique particulièrement étant donné que l'État final comportait des volets liés à la diplomatie et au développement. Le Groupe multinational inter-agences a tout de même fait preuve d'un grand potentiel pour ce qui est du renforcement de la coordination des intervenants militaires et non militaires dans une zone de mission difficile.

o Le concept de coordination pangouvernementale permettant d’atteindre un état final commun nécessite un engagement aux plus hauts échelons du gouvernement. Cela s’applique également au Canada. L’établissement d’organisations telles que le Groupe de travail sur la stabilisation et la reconstruction (GTSR) d’Affaires étrangères Canada indique que le gouvernement canadien prend des mesures concrètes afin de rapprocher le concept pangouvernemental de la réalité. Parallèlement, le manque de ressources et les répercussions sur les diverses organisations gouvernementales soulèvent de plus en plus d’inquiétudes au Canada. La complexité de l’approche pangouvernementale 3D et du concept de Groupe multinational inter-agences est amplifiée lorsque les pays participent à des opérations de coalition. Même si des accords internationaux ont été mis en place pour permettre une interaction entre les gouvernements, il se peut que le degré d’interopérabilité (connectivité) soit faible au sein de chacun des gouvernements entre les militaires et leurs collègues civils. L’interopérabilité militaire et non militaire dans chacun des pays et entre les pays

x DRDC Ottawa 2006-230

participant à des opérations multinationales serait primordiale pour la mise en œuvre réussie du concept de Groupe multinational inter-agences.

3. Élaborer et peaufiner un plan multinational de gestion des connaissances pour un quartier général opérationnel de la coalition ou un quartier général de la Force de réaction de l’OTAN pouvant être adapté et utilisé afin de répondre aux besoins opérationnels nationaux.

o L’évaluation du développement d’une base de connaissances a démontré que cette fonction n’a pas donné lieu à une compréhension globale du contexte opérationnel (JFCOME, 2006b; Annexe C). L’évaluation du plan de gestion des connaissances a démontré que celui-ci n’a pas permis d’apporter le soutien adéquat à toute la gamme d’activités liées aux connaissances nécessaires aux opérations fondées sur les effets (Annexe D). Par conséquent, le concept combiné portant sur le soutien des connaissances (développement d’une base de connaissances et gestion des connaissances) doit être réexaminé au niveau conceptuel. Certaines des définitions de base sur lesquelles est fondé le concept de soutien des connaissances doivent être établies. Par exemple, qu’est-ce qu’une donnée, une information et une connaissance? Qu’est-ce qu’une base de connaissances? Est-ce différent d’une base de données de renseignements? Où se situe la gestion de l’information par rapport à la gestion des connaissances et au développement d’une base de connaissances?

o Cela signifie pour le Canada qu’il devra effectuer plus de recherche et de développement conceptuel ayant trait à la notion de soutien des connaissances relativement aux opérations. Le soutien des connaissances inclurait la gestion des technologies de l’information, la gestion de l’information et, d’une façon plus générale, la gestion des connaissances. La technologie de l’information verrait à la gestion des composants logiciels et matériels et des outils nécessaires aux communications et à la transmission de données. La gestion de l’information inclurait la gestion des applications d’information, le raccordement des bases de données de renseignements, la diffusion de renseignements, la recherche et les demandes d’information et l’affichage de l’information. La gestion des connaissances aiderait à façonner la culture organisationnelle qui devrait permettre à chacun des employés de perfectionner les connaissances individuelles dont il a besoin pour exercer ses fonctions, notamment par l’accès à l’information et par l’analyse, la formalisation et la catégorisation de l’information. Les analyses et les données relatives au système des systèmes joueraient un rôle prédominant en vue d’aider à l’analyse, la formalisation et la catégorisation de l’information. La gestion des connaissances peut prendre la forme d’une formation particulière, de règles administratives, de services de bibliothèque, d’outils de travail et de liens entre les spécialistes et d’autres organisations.

Recommandations concernant le concept des opérations fondées sur les effets

Développer un concept canadien portant sur les opérations fondées sur les effets, de même qu’un processus, des organisations et des technologies de soutien adaptés aux

DRDC Ottawa TR 2006-230 xi

forces du Canada et aux possibilités qui s’offrent à lui, ce qui permettra de relever les défis tout en maintenant une interopérabilité avec les méthodes opérationnelles des alliés et des partenaires.

Poursuivre la collaboration avec les alliés et les partenaires en vue d’apporter une amélioration aux organisations, aux technologies et aux processus multinationaux portant sur les opérations fondées sur les effets par le biais d’occasions telles que la série d’Expérimentations multinationales.

Suivre les progrès des alliés, des partenaires et de l’industrie en ce qui a trait au développement des applications logicielles en appui du processus et des méthodes portant sur les opérations fondées sur les effets.

Adopter le concept d’un cycle de planification, d’opération et d’évaluation qui soit flexible et souple et qui se répète jusqu’à ce que la situation actuelle atteigne l’état final.

Procéder à une expérience permettant de vérifier expressément l’efficacité des opérations fondées sur les effets, par comparaison avec la pratique actuelle.

Recommandations concernant le Groupe multinational inter-agences

Organiser une série d’événements au cours desquels les communautés diplomatiques, du développement et de la défense du Canada pourront développer des concepts et des solutions viables sur la façon dont ils collaboreraient lors d’une opération nationale, de même qu’en tant que membre d’une coalition multinationale.

Recommandations concernant le soutien des connaissances

Mettre au point un modèle conceptuel de soutien des connaissances qui est théoriquement défendable et testable.

Poursuivre le développement et la mise à l’essai d’un concept et d’un plan portant sur la gestion des connaissances pour une utilisation opérationnelle canadienne.

Recommandations concernant des expériences

S’assurer que les objectifs expérimentaux sont clairement définis, peuvent être testés et sont supportés par un client.

S’assurer du contrôle du design de l’expérience, de la formulation de l’hypothèse expérimentelle, de l’exécution de l’expérience ainsi que de la rédaction des rapports expérimentaux.

Farrell, P.S.E., D. Allen, P. Burrows P. Comeau, S. Hughes, J. Kachuik, P. Labbé, F. Lichacz. 2006. Multi-National Experiment 4 on Effects-Based Approach to Operations: CFEC Analysis Report. DRDC Ottawa TR 2006-230 R & D pour la défense Canada - Ottawa

xii DRDC Ottawa 2006-230


DRDC Ottawa TR 2006-230 xiii

Table of contents

Abstract........................................................................................................................................ i

Résumé ........................................................................................................................................ i

Executive summary ................................................................................................................... iii

Sommaire.................................................................................................................................. vii

Table of contents ...................................................................................................................... xii

List of figures ........................................................................................................................... xv

List of tables ........................................................................................................................... xvii

Acknowledgements ............................................................................................................... xviii

Introduction ................................................................................................................................ 1

Concepts ..................................................................................................................................... 3 Effects Based Approaches to Operations Concepts....................................................... 3 Multinational Interagency Group Concept .................................................................... 7 Knowledge Support Concept......................................................................................... 8

Experiment Design ................................................................................................................... 11 Aim.............................................................................................................................. 11 Objectives .................................................................................................................... 12 Hypothesis ................................................................................................................... 12 Scenario ....................................................................................................................... 13 Battle Rhythm.............................................................................................................. 15 Analysis ....................................................................................................................... 16 COIs Relevant to the Canadian Objectives ................................................................. 17

Results Summary...................................................................................................................... 21 EBAO .......................................................................................................................... 21 MNIG .......................................................................................................................... 23

xiv DRDC Ottawa 2006-230

Knowledge Support ..................................................................................................... 24 Focus Areas ................................................................................................................. 26

Discussion of Canadian Objectives .......................................................................................... 28 Canada’s International Policy and the 3D Approach .................................................. 28

Conclusions .............................................................................................................................. 36 Canadian Objective Insights........................................................................................ 36 Recommendations ....................................................................................................... 37

References ................................................................................................................................ 39

ANNEX A: CFEC Experimental Directive and Setup ............................................................. 42

ANNEX B: Technology Results............................................................................................... 53

ANNEX C: Knowledge Base Development and Systems-of-Systems Analysis...................... 62

ANNEX D: Knowledge Management .................................................................................... 100

ANNEX E: Culture Analysis.................................................................................................. 129

ANNEX F: Common Intent.................................................................................................... 151

List of Symbols....................................................................................................................... 169

Glossary.................................................................................................................................. 173

DRDC Ottawa TR 2006-230 xv

List of figures

Figure 1. EBAO major processes depicted as a feedback control system................................. 5

Figure 2. A possible Control Theory configuration for EBAO CONOPS v0.9 ........................ 6

Figure 3. EBAO Organization Chart for MNE 4........................................................................ 6

Figure 4. Knowledge Management Organization as Proposed by the KM Plan ..................... 10

Figure 5. MNE 4 CTF / NRF Operational Areas..................................................................... 14

Figure 6. MNE 4 Experiment Design ...................................................................................... 16

Figure 7. Areas under investigation for MNE 4 ...................................................................... 16

Figure 8. Possible relationship of MNIG and Canadian START ............................................ 33

Figure A.1. Daily Battle Rhythm............................................................................................. 44

Figure A.2 Battlelab Layout ..................................................................................................... 45

Figure B.1 MNE 4 network/server diagram ............................................................................. 55

Figure B.2 MNE 4 application peak usage............................................................................... 56

Figure B.3 MNE 4 traffic proportion for the main applications............................................... 57

Figure B.4 Tool usefulness rated by MNE 4 participants. ....................................................... 58

Figure B.5 Tool trouble tickets reported .................................................................................. 59

Figure C.1 Distribution of correct responses for SA surveys, by organization ........................ 71

Figure C.2: Box Plot Distribution of SA scores by Group and Survey for the CTF HQ staff . 72

Figure C.3: Distribution of SA scores for KBD3 by group for the CTF HQ staff ................... 73

Figure C.4: Multi-dimensional scaling of KBD3 answers by Group ....................................... 74

Figure C.5: Distribution of Correct scores for KBD3 by experience level .............................. 75

Figure C.6: Distribution of KBD correct answers by user reported confidence level [27] ...... 76

Figure C.7 Proportion of staff using MindManager, by Group and organization .................... 79

Figure C.8 Use of MindManager by CTF Group and Survey date........................................... 80

xvi DRDC Ottawa 2006-230

Figure C.9: User ratings for Mind Manager for various questions, by CTF group .................. 80

Figure C.10: MindManager Layout for KB Access ................................................................. 82

Figure C.11: NRF User Knowledge Representation preferences (top 3) by Group [27] ......... 83

Figure C.12: Command's Approved Effects List (CAEL) and Dependencies of Effects......... 84

Figure C.13: Diagram of Influence in Humanitarian Assistance Operations ........................... 85

Figure D.1 KM Organization as Proposed by the KM Plan ................................................... 100

Figure D.2 Distribution of participants’ assessment on information access........................... 104

Figure D.5 Time evolution of the number of KRs that were issued, acknowledged, closed, managed, dropped, and answered by specific groups (KM, SOSA, Intel, MNIG, Control).118

Figure D.6 Quality of KR answers and timeliness of answers as perceived by participants.. 119

Figure D.7 Response time for the participants involved in processing the KRs .................... 119

Figure F.1. Two-person depiction of a multiple-human interaction model........................... 151

Figure F.2. Multiple responses to a single question OR a single respondent’s responses to multiple questions may form the measure vector OR a combination of multiple respondents and multiple questions can be compared to their corresponding reference vector............................................................................................................................... 157

Figure F.3. Action Consistency results as an indicator of Implicit Intent ............................ 165

DRDC Ottawa TR 2006-230 xvii

List of tables

Table 1. Correlating Implicit Intent with Mental Demand (MD) ............................................. 27

Table A.1 Experiment Schedule............................................................................................... 43

Table A.2 CFEC MNE 4 TEAM .............................................................................................. 47

Table D.1 Median for the number of right answer to the situational awareness survey among the CTF HQ staff............................................................................................................. 103

Table D.2 Situational Awareness Score for the Various Functional Groups ......................... 103

Table D.3 CTF HQ Staff Perception of the Quality of the Information................................. 106

Table E.1 Mean Ratings (MR) and Standard Deviations (Std) for Pre-Experiment Surveys........................................ 130

Table F.1 Means and Standard Deviation for Each Question................................................. 158

Table F.2 Correlations between Questions for KM and |R| > 0.5........................................... 159

Table F.3 Measured Vector Magnitude as a percent of the Reference Vector and the Angle between Measured and Reference Vectors ..................................................................... 160

Table F.4 Projection of Measured Vector onto Reference Vector (%) .................................. 160

Table F.5 Action Consistency with respect to Commander’s Intent ...................................... 161

Table F.6 Mean Values for Implicit Intent questions ............................................................. 163

Table F.7 Action Consistency Projection of Measured Vector onto Reference Vector (%) .. 165

Table F.8 Comparing EI and Roles and Responsibilities Results .......................................... 166

Table F.9 Comparing Implicit Intent and Decision-making Q1 and Q2 Results.................... 167

Table F.10 Correlating Implicit Intent results with Decision-making Q3 results................... 167

xviii DRDC Ottawa 2006-230

Acknowledgements

The authors would like to acknowledge the contributions of CFEC staff that supported and performed all administrative details for national workshops, international workshops and conferences, and the MNE 4 event itself. Also recognized are the MNE4 role players and additional data collectors who worked diligently to provide a rich dataset on which this analysis is based.

The authors would also like to acknowledge the MNE 4 analyst team, led by Mr. Mike Wahl, Lead Analyst, JFCOM, who worked collaboratively to design the assessments, collect data, and analyze the results.

DRDC Ottawa TR 2006-230 1

Introduction

The United States (US) Joint Forces Command (JFCOM)-led Multinational Experiment 4 (MNE 4) examined Effects Based Approach to Operations (EBAO) conducted by a coalition of nations within a distributed operational level headquarters.

“The effects-based approach focuses a combination of military and other activities on influencing the overall behavior of other actors – national and transnational, belligerent and benign - in an operational environment. Its application allows the planning, execution, and assessment of those operations to be based on a holistic and dynamic understanding of those and other actors in that environment. The resulting benefits are a set of actions that are explicitly linked to a set of strategic goals (expressed as an “end state”), coherently harmonized with those of other governmental organizations, and made truly adaptive within the course of their execution by effective assessment (JFCOM, 2005b).”

MNE 4 was the fourth major event in JFCOM’s continuing multinational experimentation campaign that included a series of workshops and Limited Objective Experiments (LOEs), which help refine the process, organizational, and technology concepts associated with EBAO. Australia (AU), Canada (CA), Finland (FI), France (FR), Germany (GE), Sweden (SE), and the United Kingdom (UK) were invited to participate in MNE 4, while the North Atlantic Treaty Organization (NATO) participated in parallel with the “Coalition”. A Coalition Task Force Headquarters (CTF HQ) and a NATO Response Force Headquarters (NRF HQ) formed the notional deployed operational level headquarters. The experiment scenario was based upon a present-day Afghanistan stabilization mission.

The specific focus of MNE 4 was to investigate the robustness of the four main hypothesized EBAO functions: Knowledge Base Development (KBD), Effects Based Planning (EBP), Effects Based Execution (EBE), and Effects Based Assessment (EBA). The CTF and NRF headquarters were developed based upon the emerging US Standing Joint Force Headquarters (SJFHQ) prototype developed in 2002 (JFCOM, 2003) and progressively implemented at cadre level across the US Regional Combatant Commands. A Collaborative Information Environment (CIE), which included a secure network and software applications including those for verbal and written communications, provided technological support to the two headquarters staff. In addition to these EBAO Process, Organization, and Technology (POT) concepts, the supporting concepts are listed below that were developed and examined during the experiment:

a. Strategic Context; b. Multinational Interagency Group (MNIG); c. Knowledge Management (KM); d. Information Operations; e. Intelligence; f. Logistics; and g. Medical.

Canada took the lead in coordinating the design, data collection, analysis and reporting for the Knowledge Management (KM) supporting concept as well as two focus areas – Culture and Common Intent – on behalf of the MNE 4 community.

2 DRDC Ottawa TR 2006-230

The Canadian Forces Experimentation Centre (CFEC) was responsible for planning and coordinating the Canadian participation in MNE 4. The Canadian MNE 4 team consisted of Experiment Management, Concept Developers, Technical Staff, Experiment Control and Grey1 cells, CTF HQ players, JFCOM Liaison Officer, and Analysts (see Annex A for complete details). This analyst report focuses on the three Canadian Objectives: 1) EBAO, 2) MNIG, and 3) Knowledge Support (KS: KM and KBD) in a Canadian context. This report uses the knowledge gained from not only the Canadian analyses of KM, Culture, and Common Intent but also other MNE 4 analyses including EBAO Process, Organization, and Technology as well as the MNIG assessments.

This report addresses the Canadian objectives and is organized under the following sections: Introduction, Concepts, Experiment Design, Results, Discussion and Conclusions. The ANNEXES include specific analyses on Technology, KBD and SOSA, KM, Culture, and Common Intent that were produced by CFEC analysts.

1 Grey cell consists of role players that work directly for the Control cell. As a general rule, analysts did not assess the Grey cell players.


Concepts

Effects Based Approaches to Operations Concepts

US JFCOM J9 (experimentation) has been exploring EBAO since 2001 with partner nations, including Canada’s Canadian Forces Experimentation Centre (CFEC). A complete description of the EBAO Concept of Operations (CONOPS) is provided in (JFCOM, 2005b). This section provides synopses of the concepts that impact the Canadian objectives.

Process Concept

The EBAO CONOPS describes four major functions: KBD, EBP, EBE, and EBA. Each function has multiple activities, each activity has multiple steps, and each step has multiple procedures: thus, the CONOPS describes a hierarchy of process tasks required to conduct EBAO.

KBD consists of four major activities:

1. Guidance and Requirements Analysis (GRA) 2. Development of a Multinational Knowledge Base (DKB) 3. Knowledge Processing (KP) 4. Continuous Systemic Analysis (CSA)

The KBD process is part of the Knowledge Support concept introduced below, and it describes a methodology to establish a virtual Multinational Knowledge Base (MN KB) spanning multiple databases all the while collecting new information and refining the existing information. In a multi-dimensional operation encompassing the gathering of information on Political, Military, Economic, Social, Information and Infrastructure (PMESII) systems of the operational environment, the MN KB would provide headquarters staff with information resources so that they could develop a holistic understanding of the environment. Collecting and interpreting information across such a spectrum would require experts from a wide variety of disciplines with cross-disciplinary and operational environment expertise.

KBD is described as a continuous function that runs in parallel with the three other EBAO functions. The MN KB would be updated continuously with new information and associated interpretations, and headquarters staff would draw from the MN KB for the latest information in order to do their work. In return, staff members would store and archive their deliverables in the MN KB for other staff members to access. This concept seems sound, however, there are technical, policy, security, language, and cultural issues related to Multinational Information Sharing (MNIS)2 that have not been resolved.

2 MNIS and Operational Net Assessment (ONA) were the concepts for Multinational Limited Objective Experiment 2 (J9, Joint Experimentation Analysis Division, 2003). This reference concluded that


The EBP function consists of six major activities:

1. End State Analysis (ESA) 2. Effects Development (ED) 3. Red and Green Teaming (RG) 4. Action Development and Resource Matching (ADRM) 5. Effects-Based Assessment Planning (EBAP) 6. Synchronization and Plan Refinement (SPR)

EBP is the most mature of the four functions and was fully explored in Multinational Experiment 3 (J9, Joint Experimentation Analysis Division, 2004). EBP requires the understanding of the End State in the context of the operational environment, developing desired effects and actions that are planned to achieve the End State, and providing some coordination in time and space for those effects and actions. The types of effects and actions being considered comprise of four categories representing the instruments government power: Diplomatic, Information, Military, and Economic (DIME). These actions are planned based on knowledge of the operational area represented by six interrelated PMESII systems; however, the relationships between DIME actions and the PMESII states of the system of systems are very complex. EBP attempts to develop synchronized DIME actions that produce a series of intermediate effects that in turn contribute to attainment of the desired End State.

The EBE function consists of three major activities:

1. Prepare Orders (PREPO) 2. Control and Coordinate Operations (C&C) 3. Identify Issues (IDI)

EBE, although the newest and least developed of the four functions, is fairly straightforward because this is what militaries do – they execute plans. Fundamentally, the EBE function involves converting the plan into Orders for the tactical level, controlling and coordinating the actions outlined in the Orders, identifying issues associated with the actions, and correcting the actions by generating new Orders, if necessary. This cycle may repeat itself until the desired actions are realized. However, a key nuance for EBE is that EBAO must consider all instruments of power (DIME) requiring (conceptually) four sets of orders and four interrelated cycles. Practically, EBE requires coordination not only amongst the coalition’s militaries, but also coordination amongst the military and non-military “tactical units” who are performing the actions.

The EBA function consists of five major activities:

1. Qualitative Campaign Evaluation (QCE) 2. Effects Analysis by use of Measures of Effectiveness (EAMOE) 3. Actions Analysis by use of Measures of Performance (AAMOP) 4. Measures of Effectiveness – Measures of Performance Analysis (MMA) 5. Campaign Assessment (CA)

MNIS is a critical requirement for ONA (EBAO’s grandfather concept). However, solutions are still under development (e.g., US JFCOM’s Cross-Domain Collaborative Information Environment).


Simply put, EBA provides feedback (or an assessment) for three primary variables: Current State (or QCE report), effects, and actions. If the Current State, Effects, and Actions are progressing towards the desired (or planned) End State, Effects, and Actions then the EBP and EBE staffs may continue to follow the plan and operations order. However, if these variables begin to significantly diverge from their desired states then this would prompt re-execution, re-planning, or even a re-evaluation of the strategic context and the desired End State. The challenge for EBA is to not only measure Current Actions, Effects and End State (especially when Effects may take years to develop to a state where they are perceptible) but also determine when a variable has (or is projected to) “significantly diverged” so that they can advise the operational level Commander, and the Commander can make a decision to continue with the current plan, or to re-plan. The four major EBAO functions described above may be schematically depicted as a simple feedback loop as shown in Figure 1.

Planning Execution

World

Assessment

DesiredEnd

State

CurrentState

Knowledge Base

+-

Figure 1. EBAO major processes depicted as a feedback control system

A Control Theory view of EBAO shows the relationship between EBP, EBE, and EBA as a feedback loop (sequential but iterative) with the world as part of the loop. At the same time, the MN KB contains information required for the planning, execution, and assessment phases. The key advantage of depicting EBAO as a control loop is that Classical and Modern Control Theory can be brought to bear on conceptualizing the system dynamics towards answering two fundamental questions: 1) does the system work (stability and robustness), and 2) does the system work well (optimization and effectiveness)? MNE 4 investigated EBAO robustness only.

Figure 2 is one possible configuration of the major phases for EBP, EBE, and EBA within a control theory framework derived directly from the EBAO CONOPS. This view shows three primary loops or three variables to be controlled: Current State, effects, and actions. Sampling Theory (Van de Vegte, 1990) would indicate that the inner loop (actions) should be, as a minimum, twice3 as fast as the middle loop (effects), which should be correspondingly as fast as the outer loop (Current State) as a minimum to achieving stability.

3 The rule of thumb is ten times depending on the accuracy requirements and energy costs.


ESA ED

RG

ADRM SPR PREPO

IDI

C&C

AAMOP

WModel World

AAMOE

QCE

MMACA

EBAP

DesiredEndState

CurrentEndState

DesiredEffects

CurrentEffects

DesiredActions

CurrentActions

states

Figure 2. A possible Control Theory configuration for EBAO CONOPS v0.9

Organization Concept

The EBAO organization concept employed by the CTF HQ and NRF HQ was subdivided into seven staffs as illustrated in Figure 3, and represents a departure from the traditional organizational structure based on the four major EBAO functions. The Continental staff J-Code structure was subsumed within the following seven composite staffs. The Command Group (CG) provides the overall instructions and guidance for the headquarters. The Multinational Interagency and Coalition Agency Groups (MNIG/CAG) were not part of the military command structure but were established to coordinate all non-military activities within the operation and advise the CG and the other staffs on diplomatic and development issues. The Knowledge Support (KS) staff is divided into two: KBD staff and KM staff. The KBD staff develops and maintains the MN KB, while the Knowledge Management (KM) staff governs and facilitates the knowledge activities throughout the headquarters. The EBP, EBE, and EBA staffs perform their respective functional activities.

Effects BasedPlanning

Effects BasedExecution

Effects BasedAssessment

Knowledge BaseDevelopment

KnowledgeManagement

KnowledgeSupport

CommandGroup

MultinationalInteragency Group

CoalitionAgency Group

Figure 3. EBAO Organization Chart for MNE 4

For MNE 4, there were a total of 124 billets for the CTF headquarters. However, the links between billets and tasks to be performed were tenuous and developed only as


the event unfolded. Clearly more conceptual work is needed to design the Competencies, Authorities, and Responsibilities (Pigeau and McCann, 2000) required for all the EBAO activities, steps, and procedures.

Technology

The EBAO technology concept requires a secure wide area network on which distributed but synchronous collaboration may take place, documents can be managed, and EBAO support tools can be used. The technologies include software tools for distributed collaboration, situation awareness, planning, modelling and simulation. During MNE 4, staff members were distributed across North America and Europe in seven facilities. Their primary means of communication was text and audio chat over the secure network. The headquarters shared documents and software tools, and accessed the KB via a Portal. The NRF HQ staff was co-located in a single facility in Istanbul Turkey and the staff used a similar software tools but also had the opportunity for face-to-face communications although they were discouraged from doing so.

Multinational Interagency Group Concept

Australia led the development of the Multinational Interagency Group (MNIG) concept (Australian Defence Force, 2005). The lead was passed to the US for analysis during and after the main event. The concept attempts to harmonize and align the multinational EBAO process with interagency planning and capabilities – a prerequisite given that the process calls for integrated DIME action under a common desired End State. The concept assumes that a coordinated coalition strategy is in place to provide commonly accepted goals and objectives, that the MNIG has been established, deployed, and is co-located with the CTF HQ, and that a collaborative environment is available and in use.

The goal of the MNIG is to help achieve the strategic objectives by:

Harmonizing the planning and actions of civilian agencies with those of the coalition task force;

Coordinating the provision of civilian capabilities, expertise, and perspectives; and

Enabling collaboration with partner agencies, international organizations, and nongovernmental organizations.

The concept employs a two-pillar model: one pillar being a military command system and the other pillar a civilian “influence” system. The Director of the MNIG is the head of the civilian system; the Commander of the CTF is the head of the military system. The Director of the MNIG has peer status with the Commander of the CTF. The lead pillar is determined by the emphasis of the campaign: the Commander of the CTF leads when military objectives and effects are paramount, and the Director of the MNIG leads when civilian objectives and effects are paramount.


Knowledge Support Concept

The Knowledge Support (KS) concept was originally conceived as an organizational structure that encompassed both KBD and Knowledge Management (KM) functions. KBD’s primary functions are to populate and maintain multiple databases pertaining to the operation. KM’s primary functions are to govern and facilitate knowledge activities within the headquarters. Thus, the KS concept is to develop and manage knowledge so that decision makers (at all levels within the headquarters) can make timely and effective decisions. Despite the conceptual delineation between KBD and KM, in practice KBD and KM overlap. For example, categorizing the information within the MN KB is currently the responsibility of the KBD staff. However, information categorization is a means of facilitating a person’s internal representation of that information – that is, facilitating their knowledge. The KS organizational structure was introduced, in part, to acknowledge that these two concepts are intertwined and so that further concept development could be conducted under the KS umbrella.

The KBD concept is discussed in the previous section, and so these next paragraphs focus on the KM concept. Contemporary operational commanders and their staff are often faced with information overload, yet they lack the means to synthesize the information required for timely and effective decision-making. Furthermore, the knowledge support challenge posed by EBAO will not lessen since EBAO involves the military as well as other government departments, non-government agencies, and international organizations. Achieving a common awareness of, understanding of, and knowledge about the operational environment amongst these culturally diverse groups is a formidable task, and yet this is the essence of the KS concept. KS includes information management but also goes beyond information management and can be rooted to effective communication, sense making and the nature of knowledge itself.

In addition to information management (providing the right information to the right person at the right time), there are several challenges that the knowledge support must face:

1. Establishing an effective information sharing culture 2. Facilitating a clear understanding of the meaning and reliability of the received

information. 3. Establishing team relationships 4. Developing trust in geographically dispersed and culturally diverse teams 5. Interacting with subject matter experts in non-military domains to seek their

opinion/guidance, and adjust military strategies accordingly.

By governing and facilitating knowledge activities and providing access to data and information repositories, KS hopes to positively impact these challenges, and therefore it would be a critical enabler for EBAO.

This description of the knowledge support issues endorses the view that knowledge is not simply information, but that the concept of interpretation and reliability are intrinsic to knowledge. In other words, knowledge results from the interpretation of information elements to which a degree of belief is associated. Using the concept of interpretation from predicate calculus (Crossley, 1972), knowledge with regard to a proposition P(S) about a subject “S” can be expressed mathematically by the following representation: K(P(S)) = P(S) μ (Allen,


Comeau, and Farrell, 2006). The accolades ( … ) are used to represent the interpretation of the proposition P. The degree of belief, , represents the belief of the individual regarding the validity of the proposition. Furthermore, since the interpretation and degree of belief of a proposition depends on an individual’s pre-existing knowledge, modelling an individual’s knowledge requires the introduction of lists (or n-tuples) of knowledge elements rather than the simple conjunction of knowledge elements; in other words, the order of the knowledge elements is important.

Knowledge activities can thus be classified in terms of an individual’s knowledge being modified:

1. New knowledge elements can be created either through the interpretation of new information or through a reasoning process based on existing knowledge.

2. The interpretation of knowledge elements can be modified. 3. The degree of belief in knowledge elements can be modified. 4. The order of the knowledge elements can be modified.

These various categories of knowledge activities are called knowledge creation, knowledge interpretation, knowledge formalization (verification and validation), and knowledge prioritization, respectively. Thus, KM is about the governance and facilitation of the creation, interpretation, formalization, and prioritization of knowledge. This view of KM may help in showing that KM and KBD are two distinct functions of KS. Whereas KBD deals with the development, compilation, storage and fusion of the information, KM deals with the human interaction with that information.

In preparation to MNE 4 main event, CFEC led a national and multinational investigation of the KM concept. US JFCOM supplied the KM Concept of Operations (JFCOM, 2004a) and Concept of Employment (JFCOM 2004b) for the Standing Joint Forces Headquarters as a starting point for the investigation. CFEC hosted one national workshop and three multinational workshops to identify the key KM principles and develop a KM Plan for MNE 4 that incorporates those principles. The KM principles are based on the following definition:

“KM is the governance and facilitation of the knowledge activities in order to achieve the objectives of the CTF HQ Commander” (Bernier, Farrell, and Elford, 2005).

The KM Plan version 0.9 (CFEC, 2005) was released on 20 October 2005 and it includes KM activities, KM organization including Tactics, Techniques, and Procedures (TTPs), and sets of KM business rules for the MNE 4 technologies. The KM organization proposed by the KM Plan is displayed in Figure 4.


Figure 4. Knowledge Management Organization as Proposed by the KM Plan

The roles and responsibilities of each KM officer are provided in (CFEC, 2005). Note that there is no systematic logic that links the theoretical knowledge activities to the KM organizational structure. Three sets of business rules were developed:

1. The collaborative meeting business rule, which provides rules for planning and executing virtual meetings using the Information Work Space (IWS) tool;

2. The KB interaction business rule, which specifies the way MNE 4 participants are to interact with the KB. In particular, it provides a file naming convention; and

3. The Knowledge Request (KR) process business rule, which sets guidelines on the way the knowledge requests should be processed. The KR function was designed to meet knowledge gaps and shortfalls of all HQ personnel.


Experiment Design

Multinational experimentation is a key series of events for US JFCOM’s joint concept development and prototype pathways. Each event in the series hopes to advance the cause of better coalition military cooperation that brings coalition members closer to recommending processes, organizational structures, and technologies to senior leaders based on experimental findings. The body of data collected during the multinational experimentation series guided the MNE 4 experimental design.

As the MNE 4 lead, US JFCOM wanted to refine and assess the EBAO Process, Organization and Technology (POT). A multinational analyst team was assembled to study EBAO POT. Each nation coordinated the design, analysis, and reporting of one or more supporting concepts and focus areas that are outlined in the MNE 4 Experimental Design Document (JFCOM, 2006). Canada volunteered to take the lead for one supporting concept and two focus areas, respectively:

a. Knowledge Management (KM) b. Culture analysis c. Common Intent

Additionally, the CFEC analyst team also focused on the following research areas, primarily in support of the Canadian MNE 4 objectives:

d. KBD, particularly System of Systems Analysis (SOSA) e. Technology

The five analyses performed by Canadian analysts are archived in the Annexes of this report. In addition, the following assessments from the wider MNE 4 analyst team were reviewed in support of Canadian MNE 4 objectives related to:

f. EBAO Process g. MNIG

Aim

The aim of MNE 4 experiment was to explore concepts and supporting tools for EBAO within a coalition environment involving stability operations with increasing levels of conflict to assist the development of future processes, organizations and technologies at the operational level of command.

The aim of this final report is to synthesise and distil the MNE 4 assessments to address the Canadian MNE 4 objectives as stated in the CFEC MNE 4 National Experiment Directive (Gibbons, 2006), and recommend a way ahead with respect to those objectives.


Objectives

In addition to the multinational objectives of assessing and refining EBAO POT, Canada identified specific issues pertaining to current national activities. Based on the identified issues, the Canadian objectives of MNE 4 were to (Gibbons, 2006):

1. Contribute to EBAO concept refinement and assessment, and analyze elements of the concept that Canada and the CF, especially Canada Command (Canada COM), Canadian Expeditionary Force Command (CEFCOM), Special Forces Command (SOFCOM), Standing Contingency Task Force (SCTF), and Canadian Operational Support Command (CANOSCOM) could apply to achieve integrated strategic efforts in operations;

2. Explore and establish the relationship between the emerging Canadian Diplomacy, Defence, and Development (3-D) strategy and determine whether EBAO represents a means of applying the 3-D approach; and

3. Develop and refine a Multinational Knowledge Management Plan for a Coalition or NATO Task Force Headquarters that can be tailored and applied to meet national operational requirements.

The corresponding reporting objectives are to:

1. Assess the EBAO concept and how it could be applied in a Canadian context, for example, Canada COM, CEFCOM, SOFCOM, SCTF and CANOSCOM;

2. Understand the MNIG concept and determine any common linkages to the emerging Canadian 3-D strategy; and,

3. Analyze KBD and KM and provide recommendations on the way ahead for Knowledge Support in the CF and other government departments.

Hypothesis

The Hypothesis of this experiment was: If an operational commander and staff employ effects based operations, then they will have the ability to:

• Create a holistic understanding and representation of the operating environment, the adversary, friendly, and neutral forces;

• Develop and refine an operational plan that establishes clear links between the commander’s desired end state and

– The effects that must be created in order to achieve those objectives,

– The actions and causality intended to create the effects,

– The joint, interagency, and multinational resources needed to conduct the actions; and

• Harmonize the full spectrum of diplomatic, information, military and economic (DIME) planning;


• Coordinate, direct, and monitor task force operations; and

• Assess actions and effects to identify operational deficiencies, recommend methods to improve force effectiveness, and assess progress towards achieving the commander’s end state.

The treatment (independent variable) was an operational level EBAO process, as described in the Concept section. The experimental units were two operational level commanders and their functionally organized headquarters staffs (i.e., CTF and NRF). The effect (dependent variable) was an assessment of whether the EBAO process as played provided the capabilities described in the hypothesis statement. Also, key human factors variables were measured (e.g., workload and Common Intent) that provided insight into distributed team performance.

Scenario

MNE 4 was staged in a 2006 Afghanistan setting. Regional influences were included to encourage a complete and realistic planning process that involved a PMESII description of Afghanistan and possible DIME multinational responses. The following paragraphs provide a summary of the geo-political scenario, for full details please see “Afghanistan’s Road to Crisis”, JFCOM (2005a).

The United Nations (UN)-mandated International Security Assistance Force (ISAF), headquartered in Kabul, was under NATO command. The original purpose of maintaining security in the national capital region was expanded, under UN Security Council Resolution 1510, to include areas outside of Kabul. ISAF established Provincial Reconstruction Teams (PRTs) in the northern and western provinces. The PRTs, consisting of small teams of civilian and military personnel, were to provide security for aid workers and help reconstruction work in Afghanistan’s provinces.

The newly elected Government of Afghanistan was experiencing problems establishing a new central government given the present security conditions within the country. The political leadership of Afghanistan was at stake as the fledgling government, endorsed by the UN, fought to ensure its internal security while validating a new constitution and legitimizing its place in the world. The reconstruction plans of Afghanistan were also at stake, as some Warlords considered the perimeter road around Afghanistan as a threat to their security. The resolve of the world was challenged as refugees, opium trafficking, and continuing human rights violations exacerbated conditions in Afghanistan.

The Afghanistan National Army (ANA) was partially operational, but not sufficiently effective to operate unilaterally.

The U.S. force levels were drawn down so that Combined Forces Command - Afghanistan (CFC-A) and Combined Joint Task Force-76 (CJTF-76) operated primarily in the eastern and northeastern parts of the country and along the border with Pakistan (see Figure 5). CJTF-76’s mission remained primarily to deny sanctuary to the Taliban, al Qaeda, and Hezb-e-Islami Gulbuddin (HIG).


Kandahar

Farah

HeratKabul

Shindand

MNE 4 CTF / NRF Operational Areas

Figure 5. MNE 4 CTF / NRF Operational Areas

The ousted Taliban regime was conducting military operations in Afghanistan, but on a reduced scale focused in eastern and southern Afghanistan. The Taliban were using the sympathetic Pakistani Pashtun population in Afghanistan and the Pakistani Inter-Services Intelligence to protect and assist them. Elements of the Taliban migrated into the safer, less active western provinces where familial ties provided security and support. Mullah Omar, a dark and elusive figure, remained the political and religious leader of the Taliban. He had one objective, to rule Afghanistan again.

Al Qaeda was believed to be in hiding, attempting to reconstitute whatever was left of their infrastructure. They provided technical and monetary assistance to the Taliban but primarily focussed their efforts in other regions of the world. However, Al Qaeda could not be discounted. Their presence in the northern region continued to fan the flames of resistance. Al Qaeda wanted foreigners out of Afghanistan in order to rebuild their bases, continue their mission, and realize their vision.

Six personalities seemed to dominate the political, economic and military landscape of Afghanistan – Mohammad Fahim Khan (former Vice President and Minister of Defense), Ismail Khan (former Governor of Herat), Gul Agha Shirzai (Governor of Kandahar), Rashid Dostum (head of the National Islamic Movement and former Deputy Defense Minister), Burhanuddin Rabinni (of the Jami'at-e Islami Party), and Hamid Karzai (current Afghan President).

Law and order beyond the borders of the major cities was marginal at best. Human rights violations were rampant, returning refugees were preyed upon, reconstruction efforts were blocked or used for individual gain, and political influences were bought and traded.


Economic prosperity was structured upon the illegal opium trade and licit and illicit materials smuggling.

Provincial Reconstruction Teams (PRTs) in the southern region had to limit the scope of their operations due to the drawdown of U.S. troops in 2005. A reduced contingent of U.S. personnel remained in each PRT providing continued support where possible, limited mainly by security concerns.

An extremely harsh winter has taken hold of Afghanistan. Record snowfalls and freezing temperatures have brought parts of the country to a standstill. Compounding the problem were the increased numbers of refugees that have made their way to UN camps across the country. Questionable security and minimal relief supplies have caused alarm within the camps and relief community.

Battle Rhythm

The experiment was conducted in JFCOM’s Distributed Continuous Experimentation Environment (DCEE) facility in Suffolk, Virginia, where the CTF HQ Command Group was located while other coalition players participating on a geographically-distributed basis from their national experimentation facilities in Canada, France, Germany, and the United Kingdom. The NRF HQ was collocated at the Ataturk Wargaming, Simulation and Cultural Center (WSCC), Istanbul, Turkey. A collaborative information environment supported the staff processes (see Annex B). All distributed sites and participants were connected via the Combined Federated Battle Laboratories Network (CFBLNet). A simulation federation was used to stimulate the Common Relevant Operational Picture (CROP) and support the effects-based planning and assessment processes.

Figure 6 depicts a high level view of the design of the main experiment. The idea was to test the robustness of the EBAO process by going through the process three times. The first time was “slow time”, where over a 5-month period, each step was deliberately performed by concept developers and control staff members so as to thoroughly examine the process and ensure all the necessary documentation and supports were in place for the main event. The second time occurred in week 1 of the event, where players reviewed EB Plan 1 and engaged in an update of that plan that was called EB Plan 1A. The third time through the EBAO process started in the middle of week 2, where there was a change in the End State from Regional Security to Humanitarian Assistance. If the EBAO processes were completed during these three instances, then EBAO would be said to be robust.


EB Plan 1 Planning EB Plan 1A Plng EB Plan 2 PlngEB Plan 1 Exe EB Plan 1 / EB Plan 1A Exe

EB Plan 1 Assess EB Plan 1 / 1A Assess

EB Plan 1 Development (Sep 05 – Jan 06)

ExecutionWeek 1

ExecutionWeeks 2-3

CTF

EB Plan N Planning EB Plan NA Plng EB Plan R PlngEB Plan N Exe EB Plan N / EB Plan NA Exe

EB Plan N Assess EB Plan N / NA Assess

NRF

Knowledge Base Development


EB Plan 1 Planning EB Plan 1A Plng EB Plan 2 PlngEB Plan 1 Exe EB Plan 1 / EB Plan 1A Exe

EB Plan 1 Assess EB Plan 1 / 1A Assess

EB Plan 1 Development (Sep 05 – Jan 06)

ExecutionWeek 1

ExecutionWeeks 2-3

CTF

EB Plan N Planning EB Plan NA Plng EB Plan R PlngEB Plan N Exe EB Plan N / EB Plan NA Exe

EB Plan N Assess EB Plan N / NA Assess

NRF



Figure 6. MNE 4 Experiment Design

Analysis The JFCOM analysis plan (JFCOM, 2006a), developed in collaboration with the multinational analyst team, provided the details for the experiment assessment and reporting. The analysis plan evolved over time due to the complex nature of this event. The analysis plan focused on the refinement and assessment of EBAO processes, organizations, and enabling technologies as listed in Figure 7.

Knowledge base development (GE) Effects-based planning (US) Effects-based execution (US) Effects-based assessment (US) Headquarters organization (UK) Technical environment (SE) Strategic context (FR) Multinational interagency group (AU/US) Knowledge management (CA) Information operations (GE) Intelligence (FI) Logistics (US) Medical (US) Culture (CA) Common Intent (CA)

Process

OrganizationTechnology

Supporting concepts and focus

areas

Hum

an Factors (US)

Figure 7. Areas under investigation for MNE 4

The MNE 4 priority was to examine the robustness of the EBAO:

1. process 2. organization 3. technology 4. supporting concepts

Culture, Common Intent, and Human Factors were focus areas that could be cited in order to explain some of the observations that were gathered in the primary and support concept


analyses. Each nation had clear analysis roles and reporting responsibilities, and Canada took the coordination lead for the design and reporting on KM, Culture, and Common Intent, and participated in the KBD and Technology analyses.

The analysis plan followed a dendritic structure that each one of the coordination leads were asked to follow for their area of analysis:

Aim o Objective

Hypothesis Critical operational issues

o Measures - Data

A Data Collection Matrix spreadsheet is available electronically upon request that lists all the questions that were designed and administered during MNE 4. It also provides key information to implement the survey in electronic form.

COIs Relevant to the Canadian Objectives

The following paragraphs describe the Critical Operational Issues (COIs) that are relevant to the Canadian MNE 4 objectives. For the full details, see (JFCOM, 2006a).

Effects-Based Planning

Does the MNE 4 implementation of effects-based planning facilitate the ability of the operational level of command to develop and refine an operational plan that establishes clear links between the commander’s end state and

o The effects that must be created in order to achieve those objectives,

o The actions and causality intended to create the effects,

o The joint, interagency, and multinational resources needed to conduct the actions, and

o Harmonizes the full spectrum of DIME planning?

Is the MNE 4 implementation of EBP anticipatory and adaptive non-linear/ iterative rather than a predictive and linear process?

What process implications occur due to parallel planning with operational-level Diplomat, Information, and Economic (DIE) mission partners?

What process implications occur due to the multinational nature of the operation?


Effects-Based Execution

Does the MNE 4 implementation of effects-based execution facilitate the operational level of command’s ability to coordinate, direct, and monitor task force operations?

What are the substantive, non-traditional aspects of EBE?

To what degree are the CTF and NRF able to monitor and coordinate activities with non-military activities?

Effects-Based Assessment

Does the MNE 4 implementation of effects-based assessment facilitate the operational level of command’s ability to assess actions and effects to identify operational deficiencies and recommend methods to improve force effectiveness?

Does the MNE 4 implementation of effects-based assessment facilitate the operational level of command’s ability to assess progress towards achieving the commander’s end state?

Does EBA adequately inform and enable adaptive EBP?

o Does the process equip the staff to recognize unexpected outcomes in a timely manner?

o Does the process equip the staff to recognize when to hastily adapt and when to be patient?

Multinational Interagency Group

Does the MNE 4 implementation of a MNIG facilitate the operational level of command’s ability to conduct effects-based operations?

What degree of coordination is achieved by the MNIG?

Are national interests and authorities represented by the MNIG?

How does MNIG membership evolve over time?

What member agency operations are required?

What are the required national or functional lines of organization?

Refine the MNIG size and scope of work

What decision making model can be employed by the MNIG?


Refine the leadership requirements of the MNIG

What is the relationship of the MNIG with NGOs?

Does the MNIG require collocation or can it function using distributed operations?

What are the conditions and triggers for organizational change within the MNIG?

What is the relationship of MNIG members to their parent organizations?

What is the relationship of the MNIG to the host nation?

Does the MNIG provide pro-active or reactive planning?

What reach and richness of information exchange is observed?

Knowledge Based Development

Does the MNE 4 implementation of knowledge base development provide a sound basis for the operational level of command’s ability to create a holistic understanding and representation of the operating environment, adversarial, friendly, and neutral forces?

Was the Knowledge Request Management sub-activity able to access and generate new information and knowledge, which do not currently exist in the KB or to task the knowledge KBD staff to develop answers based on the content of the KB?

Did the Knowledge Acquisition sub-activity ensure the matching of knowledge shortfalls with the connection of possible new sources?

Did the Knowledge Integration (KI) sub-activity ensure the integration of all input into the MN KB in a proper way?

Were the experimental conditions appropriate to allow the conduct of knowledge processing?

Knowledge Management

The KM COIs were designed such that the underlying measures provide an evaluation of the following three factors: 1) adherence to the KM Plan, 2) quality of the KM support provided during MNE 4, and 3) the need for KM support for the CTF HQ staff. With the aim to determine those factors, the following COIs were selected:

How well were the CTF HQ staff members able to perform the knowledge activities?

To what extent was the CTF HQ Knowledge Management Plan adhered to during MNE 4 main event?


How and to what degree was the KM support used by the CTF HQ staff?

To what extent was KM able to support the knowledge activities?


Results Summary

The following Critical Operating Issues (COIs) and findings are reproduced and summarized from the MNE 4 proceedings and final reports (JFCOM, 2006b; JFCOM 2006c). Occasionally, there was a missing link between the concept, the original COI in the design document, and findings found in the reports. What was anticipated in the design document was not actually played during the event, and so many COIs were modified or discontinued. Where applicable, the section headings below include a reference to the corresponding Annex for the concepts and focus areas that the Canadian analyst team were responsible for or participated in where applicable.

EBAO

Process

The following are short summaries of the findings for EBP, EBE, and EBA functions.

EBP

COI: Is the MNE 4 implementation of EBP anticipatory and adaptive non-linear/iterative rather than a predictive and linear process?

Both CTF and NRF analysts found EBP to be non-linear (non-sequential), adaptive (adjust to change) and anticipatory (plan ahead for change).

COI: Does the MNE 4 implementation of effects-based planning facilitate the operational level of command’s ability to develop and refine an operational plan that establishes clear links between the commander’s end state, effects, actions, and resources, and harmonizes the full spectrum of DIME planning?

The EBP process does link the effects, actions, and resources to the End State, however some notions, such as Centre of Gravity Analysis, seem to be missing from the concept. Harmonizing DIME planning was not addressed.

COI: What process implications occur due to parallel planning with operational-level DIE mission partners?

There was not sufficient parallel planning to adequately address the COI. However, it was concluded that non-military partners are needed in the EBP process even though it will create additional strain in order to synchronize inputs from MNIG and other external agencies.

COI: What process implications occur due to the multinational nature of the operation?


The CTF finding for this COI was that the EBP process with some streamlining and appropriate business rules could be effective in a coalition task force. However, the finding does not address the question directly.

EBE

COI: Does the MN EBO effects-based execution facilitate the operational level of command’s ability to coordinate, direct, and monitor CTF operations?

The findings implied that execution of the Operation Orders was not fully exercised during MNE 4, and so the question could not be addressed directly.

COI: To what degree does the CTF coordinate and monitor activities with relevant non-military activities.

There is an assumption that non-military organizations might have produced an equivalent DIE Operation Order to be acted upon by tactical level DIE units. However, this was not simulated during MNE 4, and so this COI was not answered.

EBA

COI: Does the MNE 4 implementation of effects-based assessment facilitate the operational level of command’s ability to assess actions and effects to identify operational deficiencies and recommend methods to improve force effectiveness?

The CTF found that EBA facilitated the assessment of actions, but the NRF found no evidence to suggest that the process helped identify operational deficiencies.

COI: Does the MNE 4 implementation of effects-based assessment facilitate the operational level of command’s ability to assess progress towards achieving the commander’s end state?

The CTF found that EBA facilitates the operational level of command’s ability to assess progress towards achieving the commander’s end state, while the NRF found the campaign assessment activity to be difficult.

COI: Does effects-based assessment adequately inform and enable effects-based planning?

There were positive indications that EBA could potentially inform and enable EBP.


Organization

Organization analysis was coordinated by the UK. They applied four analyses to look at the organization:

Social Network Analysis The UK Defence Technology Centre’s WESTT tool 4 Command Intent NITEworks’ Cross-Theme Dynamic Process Model

The focus of the analyses leaned toward the organizational structure, communication, and interaction. The organizational hypotheses suggested a comparison between CTF and NRF organizations, however, the analyses focused primarily on the CTF organization. The main finding was that the CTF HQ displayed evidence of a learning organization where the organization self-organised and where cross-functional working groups began to emerge. There was no evidence that the functional structure was better or worse than traditional military structures.

Technology (Annex B)

The technology used for MNE 4 was fragile (i.e., non-robust) – not necessarily because of the hardware and software itself, but primarily because the MNE 4 community made a conscious decision not to pursue a fully integrated software suite. Such a suite, with proper testing and evaluation, would have taken a lot of time and money to develop. It was a risk the community was willing to take. Unfortunately, some combination of the communication tools and the CFBLNet bandwidth limitations caused the system to work sub-optimally. Thus, the technology may have tainted any conclusions about the robustness of the EBAO process and/or organization.

MNIG

The MNIG analysis employed a process tracing technique5 where information produced by and solicited from the MNIG was traced through the EBP, EBE, and EBA functions.

The MNIG achieved uneven coordination with the CTF HQ and outside agencies, but this improved by the end of the experiment.

The source and derivation of authorities and responsibilities for the MNIG was not sufficiently defined and what was established was not well known or understood by the players.

4 Workload, Error, Situation awareness, Time and Teamwork (WESTT) 5 “Process Tracing in Case Study Research.” Alexander George and Andrew Bennett. Paper presented at the MacArthur Foundation Workshop on Case Study Methods, Oct 17-19, 1997. Cited at http://polisci.wisc.edu/~kritzer/teaching/ps816/ProcessTracing.htm, 26 November 2005.


The staffing of the MNIG fulfilled minimum requirements but appeared insufficient to optimize MNIG contributions.

Finally, distributed staff did not prove to be an asset for the MNIG and it was observed that technology was useful when used to enhance face-to-face working relationships but could not supplant them.

Knowledge Support

KBD and SoSA (ANNEX C)

COI: Does the MNE 4 implementation of knowledge base development facilitate the operational level of command’s ability to create a holistic understanding and representation of the operating environment, the adversary, friendly, and neutral forces?

The player audience did not gain a holistic understanding and based their answers as captured by tests during experiment execution on speculation rather than on information gained by use of the Knowledge Base (KB). Some groups were able to estimate their own performance well, while others were not.

COI: How intensive was the use of the KB in the different groups and phases of EBO?

Unsupervised usage of the KB was minimal throughout the play.

COI: Which way is preferred to access the Knowledge Base?

Consultation and advice remain essential. Direct use of prepared products was a secondary option for players. Systemic Analysis was mainly seen as being “SoSA business”.

COI: Which type of representation/level of detail is best suited to which part of EBP, EBE, and EBA?

Planners did not show clear preferences for a particular level of detail, while SoSA thought there were significant differences in the requested level of detail.

COI: Was the Knowledge Request (KR) sub-activity extensively used?

The use of the KR sub-activity was marginal at best.

COI: Was the Knowledge Request Management (KRM) sub-activity able to access and generate new information and knowledge, which does not currently exist in the KB, or does it task the KBD staff to develop answers based on the content of the KB?


MNE 4 execution did not allow for robust findings with respect to the KRM sub-activity as described in the EBO CONOPS.

COI: Did the Knowledge Acquisition (KA) sub-activity ensure the matching of knowledge shortfalls with the connection of possible new sources?

There is no proven way to identify knowledge needs. The experiment setting mainly hampered the connection with new information sources.

COI: Did the Knowledge Integration (KI) sub-activity ensure the integration of all input into the MN KB in a proper way?

Knowledge integration requires high familiarity with the contents of the KB, and with all activities within the EBAO process model. This knowledge process was inadequate during MNE 4.

COI: Are the Knowledge Process (KP) procedures feasible and was the environment appropriate to facilitate KBD?

KP is a supporting function, which requires a more detailed description than the one provided in the CONOPS.

Knowledge Management (ANNEX D)

KM failed to adequately govern and facilitate the knowledge activities for the CTF during MNE 4. Various issues were observed with regard to the ability of the CTF HQ staff to perform the knowledge activities. In particular, most staff reached a limited situational awareness during MNE 4. During the three weeks, the ability of the staff to associate an appropriate degree of belief to statements regarding the current situation improved significantly but the average situational awareness of the staff was merely 57%, which is only slightly larger than the expected score that would be obtained from a random guess (since every statement was true or false, a random guess is expected to lead to a score of 50%).

Factors influencing the ability of the staff to perform the knowledge activities included: past involvement with MNE events, previous coalition and EBP experience, years of military experience, and the involvement in the process steps. In particular, the amount of previous relevant experience was strongly correlated with the ability of the staff to attribute an appropriate degree of belief and adequately prioritize the information.

The main causes for hindrances of information sharing as identified by the CTF HQ staff, included: technical issues, procedural barriers, time constraint, and the lack of knowledge of who needs what information. The analysis supports the view that these factors hindered the sharing of information. For instance, tests made with the search engines available at the beginning of the event showed that only a very restricted search was performed independently of the specified search domain, supporting the view that technical difficulties hindered the access to information. Similarly, some


sites had a very slow connection to MindManager * during the first week and most participants were dissatisfied with the look and feel of the Portal, ONA, and MindManager tools.

The lack of adherence to the KM Plan business rules by the participants during MNE 4 was also a factor hindering the sharing of information. For example, the improper use of the Portal calendar can be linked with the difficulties some participants had in finding out the time and location of meetings. Similarly, the improper storage of the documents can be associated with the difficulties some participants experienced when looking for documents.

The Knowledge Request (KR) function that was in place to support the gathering of information not available in the KB did not provide adequate support to the knowledge creation activity. In particular, a majority of participants were dissatisfied with both the answers provided to the KRs and the timeliness of the answers. The main reasons for the inadequacy of the KR function include the large number of staff involved in processing the KRs, the lack of knowledge of who knew what information, the convolution of the KR process, the unsuitability of the KR tool to support standing KRs, and lack of involvement by some staff unaware of their role in the KR process.

The KM support to the formalization and prioritization of the knowledge was inadequate for MNE 4. Instances where contradicting information was exchanged between the participants were observed. No support was in place to facilitate the formalization of the knowledge in a consistent way. Finally, the folder subscription tool offered on the Portal was inadequate to support the prioritization of the knowledge. Many participants felt overloaded by emails and felt that the information provided was not concise enough.

Focus Areas

Culture (ANNEX E)

The culture and organizational data showed some trends toward differences in basic information processing and in perceptions of team related issues between the collocated (NRF) and distributed (CTF) HQs. Accordingly, these differences should be taken into account when attempting to explain the various performance and process measures collected in MNE 4. However, due to issues of unequal group sizes and less than optimal response rates, incorporating these findings into other aspects of the MNE 4 research effort should be made cautiously. Nevertheless, previous research has shown this type of data to be an important indicator of team performance and, therefore, these data should continue to be collected in these research environments where multinational teams are required to work together to achieve different goals.

* MindManager was the software application for one of the main databases used in MNE 4.


Common Intent (Annex F)

Common Intent (CI) is a relatively new concept developed by McCann, Pigeau, and English (2003). CI is the combination of explicit and implicit intent amongst two people, a team, or an organization such as coalition operational level headquarters. Common Intent, like situational awareness or decision-making, can be viewed as an intervening variable within cognitive structures that exists somewhere between perceivable information and observable behaviours. At the same time, CI can be thought of as a pre-condition that helps a team perform well. The term, “organizational health”, has been coined to describe what CI might measure. That is, a healthy organization or team should have a common understanding of the highest level objective because they have common personal expectations, training, tradition, ethos, and common cultural values that will ultimately lead to high team performance.

A Team Information Processing model was developed based on Perceptual Control Theory, and a relationship between TIP and Common Intent was postulated. The model suggested that Explicit Intent would stabilize team interaction by having clear roles and responsibilities, while Implicit Intent would optimize team interaction and this would be evident in the improvement of products and decisions.

An analysis was performed that investigated the relationship between Explicit Intent and Roles and Responsibilities (see Annex F), however, too few data points are available to draw a definitive conclusion.

Table 2 provides correlations between the Implicit Intent results and Mental Demand (which includes decision-making). Implicit Intent and Mental Demand are inversely related as predicted by the model. That is as high levels of Implicit Intent are obtained, decision-making becomes easier.

TABLE 1. CORRELATING IMPLICIT INTENT WITH MENTAL DEMAND (MD)

Implicit Intent wrt Cmd Intent and MD Implicit Intent wrt each other and MD

NRF (four dates) -0.92 -0.97

CTF (six dates) -0.42 -0.51

MNE 4 provided an excellent opportunity to measure Common Intent in a complex environment. Both Explicit Intent and Implicit Intent increased over the 3-week exercise indicating that it takes time to develop Common Intent. NRF Common Intent was consistently higher than CTF in all cases indicating that Common Intent may have a key role in the apparent success of EBAO conducted by the NRF headquarters. After measuring Common Intent in MNE 3 and MNE 4, the general impression is that a Common Intent value of 70-75% seems to be a minimum value required for a headquarters to move forward with some degree of success.


Discussion of Canadian Objectives

Based on the results presented in the previous section, it is very difficult to make any definitive statements about the EBAO process, organization, technology, and supporting concepts that would be supported by experimental data. Nevertheless, we have observed data trends indicating that it would be worthwhile further refining most of the concepts. The Canadian Objectives are discussed in light of these analysis observations and trends.

Canada’s International Policy and the 3D Approach

Canada’s International Policy Statement for Diplomacy, Defence, Development, and Commerce was published in 2005. Canada sees “Diplomacy [as] a central tool for achieving the priorities set out in this International Policy Statement (Canada, 2005).” The Diplomacy document refers to the New Diplomacy in the wake of the New Multilaterism and has identified a number of priorities including the establishment of “…a Stabilization and Reconstruction Task Force (START) to ensure that there is longer-term planning for early responses to international crises and that the required government skills and expertise are at the ready (DFAIT, 2005).”

Defence and Diplomacy go hand in hand. Diplomacy proceeds more smoothly when the situation is stable, the region is secure, and the people are protected, which are Defence responsibilities. The vision for Defence within Canada’s International Policy refers to this protection: “The Canadian Forces will protect Canadians at home, work closely with the United States in the defence of the continent, and deploy around the world with our friends and allies as part of a multilateral approach to international problems (DND, 2005).” In order to achieve this protection capability the CF is transforming into a force that is more effective, more relevant, and more responsive. “The transformation will require the Canadian Forces to:

adopt a fully integrated and unified approach to operations, evaluate their force structure on an ongoing basis improve coordination with other government departments and interoperability with allied

forces, update their command, control, communications, computers, intelligence, surveillance,

and place greater emphasis on experimentation (DND, 2005)” “Security and development are inextricably linked (CIDA, 2005).” For the most part, Development and Commerce occur once Defence and Diplomacy have secured the region and negotiated the agreements. The Development policy document dedicates an entire section to the Whole of Government approach to Development in order to tackle six focus areas: Promoting Good Governance Improving Health Outcomes Strengthening Basic Education Supporting Private Sector Development Advancing Environmental Sustainability


Ensuring Gender Equality

“In 2004, the Government established Canada Corps as a new vehicle to strengthen Canada’s contribution to human rights, democracy and good governance internationally. Canada Corps will develop collaborative partnerships across government, NGOs, the private sector, and with Canadian citizens to bring greater engagement, expertise, coherence and recognition to Canadian governance interventions abroad.

Canada Corps will:

mobilize Canadians of all ages and backgrounds, transform existing programming engage and communicate with Canadians expand the base of knowledge brand a common identity

In coordination with the Stabilization and Reconstruction Taskforce (“START” led by Foreign Affairs), Canada Corps will enhance its capacity to deploy governance expertise from within and outside of government (CIDA, 2005).”

There is a clear intention on the part of the Canadian Government to move forward with the 3D approach – Diplomacy, Defence, and Development. Also, Canada’s International Policy Statement hints to concepts found in EBAO. For example, EBP may be an option to fulfill START’s mandate to conduct longer-term planning for international crises. The Canadian Forces’ desire to improve interoperability with allied forces implies that they will need to continue to engage in EBAO research, concept development and experimentation since their closest ally (US) is committed to EBAO and this same concept forms the heart of NATO Transformation as well. Canada’s Whole of Government 3D approach to international affairs is akin to the notion of the military and MNIG working together to execute DIME actions. Even the Development policy statement explicitly talks about expanding the base of knowledge, which is directly relevant to the KS concept.

The following sections take the current MNE 4 assessments and discuss the implications if Canada were to adopt some form of EBAO, MNIG, or KS nationally or within a coalition. The three key questions to be addressed are:

1. Where do Effects-Based approaches and the 3D approach intersect? 2. Is there synergy between the 3D approach and the Multinational Interagency concept? 3. What would a Canada-wide Knowledge Support concept look like?

EBAO Implications for Canada

The EBAO Results Summary section indicated that the EBAO process exhibited a degree of robustness because the CTF and NRF staffs were able to execute the major phases of planning, execution, and assessment for various End States (Secure State and Humanitarian Assistance), and for various lengths of the operation (5 months and 3 weeks). This cycle resonates with Control Theory as shown in Figures 1 and 2. However, the analytical assessments did not indicate whether EBAO was effective (i.e., whether the desired End State was achieved in a timely fashion).


It was quickly realised that the Actions, Effects, and End State loops have three very different time characteristics. While actions could happen within minutes and feedback on those actions could happen within hours, a non-kinetic effect might take days, months, or years to begin to develop, and the End State might only occur after all the effects were realized. There was an attempt to implement time jumps using modelling and simulation and pre-determined injects, while at the same time maintaining the realism of the play. This timing issue made it difficult to fully explore the iterative nature of the EBAO process and to see the Current State converge onto the desired End State.

START, CF, or Canada Corps can take advantage of the basic notion of a planning, execution, and assessment cycle, and ensure that some aspect of each major phase is part of their operations cycle. However, an event (such as an experiment, workshop, or table top exercise) is required to show the effectiveness of the EBAO concept in order to show that the EBAO concept does what it claims it can do.

In the Canadian context, there may be a desire to integrate existing structures into EBAO. For example, the CF like its close allies and traditional coalition partners uses the Operational Planning Process (OPP). At this stage, inadequate attention has been paid to comparing EBP to the traditional OPP. At a higher level, EBAO has not been critically compared to the elements of the Operational Art. A study could be developed to understand how EBAO and Operational Art relate and differ and how the latter might fit into the former. The CF organizes an operation around the traditional continental staff system. A mapping between the MNE 4 functional structure and the traditional continental staff system can be developed with the key addition of the MNIG or some equivalent group. EBAO necessarily requires a broader involvement of the Whole of Government and the experiment has shown that Defence will need to work closely with Diplomacy and Development at the strategic, operational, and even tactical levels for both domestic and international operations.

Canada Command (Canada COM) Implications

The MNE 4 EBAO scenario focused on an international situation involving regional security and humanitarian assistance. For the most part, Canada COM would react to domestic crises such as natural disasters, civil unrest, and large-scale acts of violence. EBAO was primarily designed for proactive (as opposed to reactive) operations. Typically, proactive operations begin with deliberate pre-planning, while reactive operations begin with executing plans that have already been made, oftentimes through a highly compressed OPP. Canada COM might be able to incorporate elements of EBAO as they develop their pre-made plans that are based on possible scenarios. Whether proactive or reactive operations, it is the iterative nature of planning, execution, and assessment (or execution of a pre-plan, assessment, and re-planning in the case of reactive operations) at the Action, Effects, and Current State levels that help achieve the desired End State. The integrated DIME Action construct inherent to EBAO has definite application to the Canada Com mission, given the need to synchronize actions with federal, provincial and municipal departments and agencies.


Canadian Expeditionary Force Command (CEFCOM) Implications

The MNE 4 EBAO scenario and activities might closer resemble how CEFCOM would operate within a coalition that addresses an international incident on behalf of their governments. CEFCOM might offer their emerging expertise to fill staff roles in EBAO functions within operational level headquarters. Familiarity with EBAO would be paramount for CEFCOM under these circumstances, particularly if the NATO or US lead the operation. This is equally valid if Canada plays a sectoral or functional lead role within a multinational operation where the EBAO process is to be employed. Given that CEFCOM has interagency representatives on staff, the necessary expertise for integrated EBAO activities is available.

Special Forces Command (SOFCOM) and Standing Contingency Task Force (SCTF) Implications

In the EBAO construct, both SOFCOM and SCTF would be CF capabilities that Canada might offer to a coalition. They would likely operate at the tactical level, but possess unique capabilities capable of contributing to achievement of strategic effects. As EBAO progressively becomes an established methodology multinationally, SOFCOM and the SCF will need to develop knowledge and expertise concerning the concept.

Canadian Operational Support Command (CANOSCOM) Implications

Logistics and Medical activities were simulated to a limited degree during MNE 4. However, MNE 4 did not provide the necessary level of fidelity to assess whether the Joint Experimental Deployment and Support (JxDS) Concept is either an improvement on current US doctrine or viable to support EBAO. Logistics and Medical activities were also simulated in MNE 3, and the same conclusions were observed. The multinational community understands the criticality of Logistics in a coalition environment, however, the EBAO process, organization, and technology must be matured before other essential supporting elements, such as logistics and medical, are fully incorporated into EBAO. The challenge is to determine the specific requirements to adapt Logistics and Medical activities, including Joint, Combined and Interagency aspects of these functions, to enhance the ability to employ Logistics to achieve Effects.

START Implications

Since Diplomacy is Canada’s central tool for expressing its international policy and a Whole of Government is required to find solutions to the challenges of complex international missions, EBAO represents logical next step because its focus is broader than just a military operational environment as it encompasses political, economic, social, information and infrastructure


systems and the integrated employment of diplomatic, information, and economic instruments of power to achieve a common strategic End State. Developing a Knowledge Base, Planning, Executing, and Assessing operations across the DIME spectrum and not only at the military action level necessarily draws in the military and all applicable government departments. As the key division within Foreign Affairs for coordination of international security missions, START has been engaged to a degree in the MNE series and can play a decisive role in coordinating the engagement of other federal departments and agencies in examining the application to EBAO to an integrated Canadian approach to international missions. START has already been working with counterpart organizations in the US and UK on EBAO. A key requirement is to determine how other government departments can best coordinate their activities with the military EBAO process to achieve an enhanced unity of effort. One further question for investigation would be how would EBAO change if there was a diplomatic lead for the overall multinational mission instead of a military lead for the Coalition Task Force. Transition from military to civilian leadership is another key issue.

Organization and Technology Implications (ANNEXES B, E, and F)

The Focus Areas Results Summary showed that the geographically co-located staff seemed to exibit different information processing and different levels of Common Intent than the geographically dispersed staff. The design of the experiment precluded us in determining which organization or team performed better. Nevertheless, there are indications that a collocated team performs better than a distributed team, but if the team must be distributed (as likely the case for a Canadian whole of government headquarters) then a common culture and intent should significantly improve team performance.

The Technology Results Summary indicated that the tools and the network have a significant impact on team performance and mission completion. The focus in Canada for future operations would be whether the networks are in place so that the revelant government departments (as well as non-government agencies) can communicate and collaborate across vast distances, and whether a Canadian Task Force have access to computer support applications that are needed to assist in accomplishing the EBAO steps and procedures. MNE 4 has provided some suggested architectures and tools; however, Canada will need to ensure that the network infrastructure and tool suite are in place at home and abroad in order to perform missions when called upon.

MNIG Implications for Canada

The MNIG concept of harmonizing non-military planning and actions with those of the military, coordinating civilian capabilities, and enabling collaboration amongst international government and non-government agencies is a powerful idea. However, the MNIG Results Summary section suggests that many details need to be worked out


in order to realize the concept. For example, the MNIG requires clear roles and responsibilities. Moreover, the MNIG needs to have adequate resources to meet the demands for their expertise.

In the Canadian context, the MNIG concept is similar to START. The purpose of the START is “to knit together civilian and military strategies to support stabilization efforts” (Harder, 2006), such as in Afghanistan. The MNIG may provide a mechanism for coordinating Canadian efforts through the START with those of our coalition allies, as shown below in Figure 8.

Canadian Forces

Other Coalition Forces

Coalition Task Force<Military Effects>

Canadian START

Other Coalition Groups

Multinational Interagency Group<Civilian Effects>

Coalition Political & Strategic Level<Strategic Goals & Objectives>

Figure 8. Possible relationship of MNIG and Canadian START

The most senior level, the Coalition Political & Strategic Level, would formulate the strategic goals and objectives for the campaign. These goals and objectives provide guidance to the Commander of the CTF and the Director of the MNIG, who would be responsible respectively for realizing military and civilian effects. Canadian Forces units, which would be tasked by the CTF Commander, and the Canadian START, which would be part of the MNIG, coordinate to ensure Canadian goals and objectives are achieved.

There are still many questions to be resolved. How do Canada’s Diplomacy, Defence, and Development reach consensus, nationally, on the End State, desired effects, and desired (DIME or 3D) actions? Once in a coalition, how do the CF, START, and Canada Corps interact – through national lines of communication (if they exist), through the CTF and MNIG, or both? Do Diplomacy and Development have the resources to make this a reality? Although the MNIG concept is sound, there are key details at the national level that need more concept development.

Knowledge Support Implications for Canada (Annexes C and D)

The purpose of KS is to assist the access to information as well as govern and facilitate the knowledge activities. From this point of view, KS offers an environment where the information is easy to access and sufficiently concise, supporting timely decision making based on a holistic understanding of the environment. KS is clearly beneficial not only to CTF HQ but to any organization including the CF and other


Canadian government departments. In particular, various authors have argued about the importance of a collaborative environment and KS to support Logistic functions (see for example Simon 2001, Esper and Williams 2003, Grossmann 2004). It should be noted that KS is not a new function and some form of KS have been provided in the past through staff like librarians and clerks. However, the progress in the development of information technologies has brought new possibilities (e.g., higher data processing rate, distributed collaboration) and new challenges (e.g., information overload, distributed collaboration).

The KS Results Summary indicates that KBD did not produce a holistic knowledge of the operational environment within staff members, and KM did not fully govern and facilitate the knowledge activities amongst staff members. Nevertheless, the MNE 4 experience has clarified the full spectrum of KS (from physical to cognitive to organization support) that would/should include:

1. Physical network connectivity and hardware (IT)

2. Managing software tools and applications (IT/IM)

3. Information database population and hyperlinks to other databases (IT/IM, government wide Intelligence, Surveillance, and Reconnaissance assets)

4. Information distribution (IT/IM)

5. Information analysis, formalization, categorization (IM/CSA, SoSA and Intelligence)

6. Information search and request for information (IM/library services)

7. Information display (IM/KM/Human-Computer Interface design)

8. Education, Learning, and Training the EBAO processes, organizational roles and responsibilities, and technologies (KM facilitation)

9. Liaison and networking function within the staff as well as external Subject Matter Experts – academia, other government and non-government agencies, industry, etc. (KM facilitation)

10. Business rules and policy (KM governance)

It is likely that this list is not exhaustive, but clearly it coalesces into 3 main functions (Note the overlap does not need to be completely resolved, however, the staffs should be aware who is responsible for the specific sub-function):

Information Technology (IT) Management (1 thru 4)

Information Management (IM) (2 thru 7)

Knowledge Management (KM) (7 thru 10)


Thus, KS comprises of IT, IM, and KM functions. The result of the proper design of IT, IM, and KM, conceptually, would be the development of useful knowledge6, within the cognition of staff members, required to perform their tasks within EBP, EBE, and EBA.

The CF has Information Management plans and activities, and these plans might smoothly integrate into the KS concept. Most (if not all) of the KBD concept could be replaced by the well-understood IM practices (i.e., if the word “knowledge” were to be replaced by the word “information” within the GRA, CSA, KP, and DKB activities, then they would effectively become IM activities).

The CF (primarily through CFEC) needs to continue developing the KM concept. The concept of KM facilitating and governing knowledge activities was overshadowed by IT and IM activities in MNE 4 because these activities were readily understandable and could be implemented. The KM function needs to be responsible for the organizational and cultural changes that would set the stage for people to produce useful knowledge. These responsibilities might include the intentional design of a common language and taxonomy, common business rules, clear roles and responsibilities for all players, and clear intent, orders, and guidance.

The KM Plan, as developed for MNE 4, needs revision and refinement before being applied to the CF. It needs to be based on a logical and sound conceptual model. It cannot be considered in isolation but must grow into a KS plan that details all three functions – IT, IM, and KM – and clearly assigns roles and responsibilities to the three groups. Changes to the KM organizational structure and roles and responsibilities would be required to better reflect this KM concept. For example, IT has two positions in the KM organizational chart. KS could have a separate IT function reporting directly to the KS Chief. The KM organization could be re-organized to reflect Interface Requirements, Training and Education, Liaison, and Business Rules and Policy functions and positions.

6 Holistic knowledge, actionable knowledge, or knowledge advantage are intentionally not stated since these terms have conceptual difficulties from a cognitive perspective that require further study. Useful knowledge is knowledge needed to perform the required tasks. Therefore, knowledge is linked to task performance, which is an externally observable variable, even though knowledge is also related to internal cognitive variables such as situation awareness, intention, and decision-making.


Conclusions

This report documented the Canadian involvement in Multi-National Experiment (MNE) 4, and addressed the three Canadian Objectives using the analysts’ assessments. The report outlined the EBAO, MNIG, and KS concepts as documented in (JFCOM, 2005b). The report provided an overview of the experimental design for MNE 4, and listed the Critical Operational Issues that were relevant to the Canadian Objectives. The results were summarized for each relevant COI. The discussion section addresses the Canadian Objectives in light of Canada’s International Policy Statement as well as the MNE 4 assessments.

Canadian Objective Insights

The following concluding paragraphs address the Canadian objectives and summarize the key implications for Canada.

EBAO Implications

The first Canadian objective was to highlight those areas of the EBAO CONOPS that Canada and the CF could apply in operations. The notion of planning, execution, and assessment in an iterative fashion until the Current State reaches the End State is a robust concept. Canadian organizations, such as the CF, START and Canada Corps may want to adopt this high level, robust notion. However, the specific details of each activity, step, and procedure may vary from organization to organization, and may vary from the EBAO CONOPS itself.

MNIG Implications

The MNIG concept is a powerful idea, however the roles and responsibilities were not clearly articulated and there were insufficient resources to meet the demand of their expertise. Although EBAO and Canada’s 3-D approach have many aspects that are common, there are key organizational questions that remain unanswered from a national perspective such as whether START would talk directly to CF units or whether START would talk directly to the MNIG who in turn would relay information to the CTF Commander who in turn would communicate to the CF units, or some combination of both. The relationship between MNIG, EBAO, and the 3-D concepts requires further development and experimentation.

The MNIG construct developed for MNE 4 may itself need a review based on first principles to determine whether there are alternative approaches to military and non-military collaboration in the mission area.


KS Implications

The third Canadian Objective was to develop a Multinational Knowledge Management plan so that it could be tailored and applied to meet national operational requirements. However, in the assessment of the KM Plan as well as KBD concept for MNE 4, it is clear that the KS concept needs to be revisited. There still remains ambiguity between the KM and KBD concepts. Some of the basic definitions and concepts still have not been resolved (e.g., what are data, information, and knowledge? What is a knowledge base? Is it different from an information database? Where does Information Management fit relative to KM and KBD?). Follow-on KM concept development action is required.

Recommendations

The main recommendation is to continue to be engaged in focused EBAO research, concept development, and experimentation. This recommendation may seem self-serving. However, it is critical that Canada keeps abreast of the latest ideas and thoughts on this concept as our major allies (US and NATO) move toward adopting EBAO or some related variant. Moreover, it is now time to think about “Canadianizing” the EBAO concept such that it is effective in a Canadian context. Canada has a unique set of opportunities and challenges. One of the opportunities is Canada’s International Policy Statement that outlines the 3D and Whole of Government approach to International operations stating that Diplomacy, Defence, and Development will work together to achieve Canada’s International Policy. Effects based thinking may be the solution to realize any policy that takes Diplomacy, Defence, and Development into consideration. One of the challenges is to set up the mechanisms so that Diplomacy, Defence, and Development can work together, communicate, and interact towards a common aims and outcomes.

EBAO Recommendations

Develop a Canadian EBAO concept and supporting process, organizations and technologies that is suited to Canada’s strengths and opportunities, and addresses Canada’s challenges while maintaining interoperability with the methodologies of allies and partners.

Continue collaboration with allies and partners of refinement of multinational EBAO process, organization and technology through opportunities such as the MNE series.

Track progress of allies, partners and industry in developing software applications in support of EBAO process and methodologies.

Adopt the notion of a planning, execution, and assessment cycle that repeats until the Current State reaches the End State.

Conduct an experiment that looks at the effectiveness of EBAO.


MNIG Recommendations

Conduct a series of events where START, the CF, and Canada Corps (i.e., the 3D approach) can develop concepts and workable solutions on how they would work together both for domestic operations as well as part of a multinational coalition.

KS Recommendations

Develop a model of Knowledge Support that is theoretically defensible and testable.

Develop and test a KM Plan for Canadian operational use that is based on the KS concept presented in this report.

Re-evaluate the suite of tools for KS ensuring that the user has access to all reference documents.

Experimentation Recommendations

Ensure experimental objectives are clearly articulated, testable, and have a sponsor.

Maintain control over experimental hypothesis, design, execution, and reporting.


References

Alberts, David S. and Richard E. Hayes (2002). Code of Best Practice for Experimentation. CCRP, Washington, DC. http://dodccrp.org/html/pubs.html

Allen, D., P. Comeau, and P. S. E. Farrell (2006). Knowledge Management as a Supporting Concept to Effects Based Operations. 2006 International Command and Control Research and Technology Symposium Coalition Command and Control in the Networked Era, Cambridge, UK, September, 2006.

Australian Defence Force (2005). The Multi-National Inter-Agency Group, Concept of Operations (CONOPS) for MNE4, Version 1.1, 14 Dec 2005, Australian Defence Force, Australia.

Bernier, M., P. S. E. Farrell, and R. Elford (2005). Knowledge Management Workshop 1 Proceedings. DRDC-CORA/EX-ORT, Technical Note TN 2005-01, February 2005.

CFEC (2005). Coalition Task Force Headquarters Knowledge Management Plan for the Multinational Experiment IV. Version 0.88. CFEC, October 2005.

Crossley, J.N. (1972). What is Mathematical Logic? Oxford University Press, Oxford, England, 1972.

Canada (2005). A Role of Pride and Influence in the World – Overview: Canada’s International Policy Statement. Department of Foreign Affairs and International Trade, Ottawa, Canada, 2005.

CIDA (2005). A Role of Pride and Influence in the World – Development: Canada’s International Policy Statement. Canadian International Development Agency, Ottawa, Canada, 2005.

DFAIT (2005). A Role of Pride and Influence in the World – Diplomacy: Canada’s International Policy Statement. Department of Foreign Affairs and International Trade, Ottawa, Canada, 2005.

DND (2005). A Role of Pride and Influence in the World – Defence: Canada’s International Policy Statement. Department of National Defence, Ottawa, Canada, 2005.

Esper, T.L. and L. R. Williams, The Value of Collaborative Transportation Management (CTM):Its Relationship to CPFR and Information Technology. Transportation Journal, Summer 2003, pp.55-65.

Farrell, P. S. E. (2005). Calculating Effectiveness using Bi-Polar Scales and Vector Algebra. DRDC Toronto TR 2005-148, Toronto, Canada: Defence R&D Canada, 2005. 71 pp.


Gibbons, T. (2006). Multinational Experiment 4 (MNE 4) – National Experimental Directive. 3000-2 (MNE 4), Canadian Forces Experimentation Centre, February, 2006.

Grossman, M. The Role of Trust and Collaboration in the Internet-Enabled Supply Chain. The Journal of American Academy of Business, September 2004, pp. 391-396.

Harder, P. (2006). Address by the Deputy Minister of Foreign Affairs, Canada, to the Royal United Services Institute for Defence And Security Studies Conference on Canada, the UK and International Security in the 21st Century. (Online) Foreign Affairs Canada. http://www.fac-aec.gc.ca/department/deputy-minister-speeches-2006-04-25-en.asp (23-May-2006).

J9 Joint Experimentation Analysis Division (2003). Multinational Limited Objective Experiment II (MN LOE II) Final Report, US Joint Forces Command, prepared with contributions from Australia, Canada, Germany and the UK, December 2003.

J9 Joint Experimentation Analysis Division (2004). Multinational Experiment 3 (MNE 3) Final Report, US Joint Forces Command, prepared with contributions from Australia, Canada, France, Germany, United Kingdom, and NATO, April 2004.

JFCOM (2003). Standing Joint Forces Headquarters Prototype: Concept of Employment. Joint Forces Command. Suffolk, VA, 25 June 2003.

JFCOM (2004a). Knowledge Management in the Collaborative Information Environment Concept of Operations, Version 1.0, US JFCOM J9, 20 September 2004.

JFCOM (2004b). Knowledge Management Concept of Employment for the Collaborative Information Environment, Version 0.9, US JFCOM J9, 30 September 2004.

JFCOM (2005a). Afghanistan’s Road to Crisis (RTC) Multinational Experiment 4 (MNE 4) 1 Jan 2006, United States Joint Forces Command, Joint Futures Lab, September 5, 2005.

JFCOM (2005b). Effects-Based Approach to Multinational Operations, Concept of Operations (CONOPS) with Implementing Procedures Version 0.90, United States Joint Forces Command, Joint Experimentation Directorate, EBO Prototyping Team, December 19, 2005.

JFCOM (2006a). Multinational Experiment 4: Experiment Design Document Version 0.90, Joint Futures Lab; U.S. Joint Forces Command, dated 10 February 2006.

JFCOM (2006b). Multinational Experiment 4 Experiment Proceedings Report, US Joint Forces Command, prepared with contributions from Australia, Canada, Finland, France, Germany, Sweden, United Kingdom, and NATO, 31 May, 2006.

JFCOM (2006c). Multinational Experiment 4 Final Report, US Joint Forces Command, prepared with contributions from Australia, Canada, Finland, France, Germany, Sweden, United Kingdom, and NATO, in press.


McCann, C., R. Pigeau, and A. English (2003). Analysing command challenges using the command and control framework: Pilot study results, DRDC Toronto TR 2003-034, Defence R&D Canada Toronto, February 2003.

NATO (2002). NATO Code of Best Practice for C2 Assessment. CCRP, Washington, DC. http://dodccrp.org/html/pubs.html

Pigeau, R. and C. McCann (2000). The Human in Command: Exploring the Modern Military Experience, edited by McCann and Pigeau, Kluwer Academic/Plenum Publishers, New York, 2000. Chapter 12: Redefining Command and Control. pp.165 – 184.

Simon, S.J. The Art of Military Logistics – Moving to a Dynamic Supply Chain. Communication of the ACM 44, June 2001, pp. 62-66.

TTCP (2006). Guide for Understanding and Implementing Defense Experimentation (GUIDEx). The Technical Cooperation Program, Washington, DC. http://www.dtic.mil/ttcp/

Van de Vegte, J. (1990). Feedback Control Systems. Second Edition. Englewood Cliffs, New Jersey 07632: Prentice Hall.


ANNEX A: CFEC Experimental Directive and Setup

Command and Signal

LCol John Kachuik was the CFEC Experiment Director for Canadian participation in MNE 4. MNE 4 was held in the DRDC Battle Lab, Building 29.

Schedule

The following table outlines the schedule for MNE 4:

TABLE A.1 EXPERIMENT SCHEDULE

Week 0 22 Feb 06

23-24 Feb 06

1300- 2100 GMT

1100 – 2200 GMT

0800 – 1600 EST

0600 – 1500 EST

Week 1 -3 27 Feb – 17 Mar 06 1100 – 2200 GMT 0600 – 1700 EST

Daily Battle Rhythm

MNE 4 Experiment Daily Scedule

1630National Hot Wash

06000630

143015001530

1000

1100

National surveys

Experiment Control Hot Wash

Experiment preparation time

3.5-hourExperiment Window

Meal break

3.5-hourExperiment Window


Figure A.1. Daily Battle Rhythm

Experiment Control (reference Annex E of JFCOM, 2006a)

A control group ensured that the execution of the event proceeded along a path that was consistent with the design. The control group attempted to produce stimuli in the proper quantity, quality, domain area, and pace to allow detailed analysis of the COIs and testing of each facet of the experimental concepts.

Technical Support

The supporting technologies in MNE 4 were divided into three main areas of activities:

a. Collaborative planning tools, particularly InfoWorkspace (IWS);

b. Cross Domain Collaborative Information Environment (CD-CIE), the MNE 4 CD-CIE provided the synchronous (text chat, whiteboard, audio chat) and asynchronous (email, Portal, document sharing and management, document search and subscription) collaboration and planning tools necessary to support EBO; and

c. Numerous supporting applications such as knowledge maps and planning tools.

The CFEC battlelab layout is depicted in Figure A.2.


Exp Direction/Control US Liaison Grey Cell Player Analyst/Observer

Ms. Harding

LCol Kachuik

Spare

Mr. Comeau

Techs

Capt SwanMaj von Papineau

Dr. Farrell

Capt Campbell

Lt(N) Allan

Ms. White

Capt Porter

Maj Fuller

Capt Arcouette

Spare

CPO2 Knight

Mr. Hughes

Maj Horeczy

Maj Gundling

Lt(N) Givogue

Mr. Labbe

LCdr Gendron

Mr. Laustsen

Mr. Christensen

Dr. Allen

Mr. Lavoie

Mr. Burrows

Mr. Patrick

Ms. Sullivan-Kwantes Ha ton

Battle Lab 29 Seating Plan

Restricted Area

Dr . Lichacz

Figure A.2 Battlelab Layout

The MNE 4 enterprise network was operated at the Secret Releasable AUSCANUKUS and NATO (which includes FR & GE) classification level using a cryptographically separated enclave within the CFBL Network. All MNE 4 events were conducted in the DRDC Battle Lab, at distributed sites in the US, UK, GE, FR, and the NATO site at Istanbul, Turkey.

Finance

All expenses for MNE 4 for rations and transportation were charged to the International Program Coordination budget as follows:

Rations. Fund Centre 3929AA, Cost Centre 48223A, Fund L101, GL 7404. Vehicle Rental. Fund Centre 3929AA, Cost Centre 48223A, Fund L101, GL 5801.

For CFEC members, all claims were immediately submitted to CFEC MNE 4 Coordinator, WO Crosby for processing. For non-CFEC members who were on TD paid for by CFEC, copies of completed claims were sent to CFEC MNE 4 Coordinator for processing.


Working Hours

Participants were considered on exercise and unavailable to their parent Team or unit. The MNE 4 working hours were:

a. 22 Feb 06 from 0800 -1600 hrs EST daily; b. 22 -24 Feb 06 from 0600 – 1500 hrs EST daily; and, c. 17 Feb –17 Mar 06 from 0600 – 1500 hrs EST daily (Mon thru Fri).

Although these hours were dedicated to experiment conduct across the network, some members of the team had to arrive 1-2 hours prior to start and remained on station to finish business and prepare for the next day’s activities.

Military Personnel

Commandant CFEC exercised the option of considering requests for short leave after MNE 4 if supported by Experiment Director.

Civilian Employees

Overtime was authorized for experiment analysis team members to conduct analysis and commence report writing for input to the multinational report. This demand for overtime was associated with the support required for MNE 4 and not for other CFEC-related tasks. All overtime was accounted for in accordance with CFEC Standard Operating Procedures.

Canadian MNE 4 Team Organization

The MNE 4 organization was based the CTF headquarters with each of the eight nations filling a share of the HQ staff positions. While most of the CTF headquarters staff from the non-US nations participated on a distributed basis from their home country, a number deployed to US JFCOM to participate.

The Canadian MNE 4 team worked in the CFEC Battlelab facility in Bldg 29 and operated in a distributed mode, employing the CFBLNet to communicate with the other participating nations. The Canadian MNE 4 team was organized as follows:

a. Experiment Director b. Lead Controller/Deputy Controller c. Grey Cell d. CA Analyst Cell e. Technical Staff

Canadian contributions to the CTFHQ were as follows:

a. CTF HQ staff: b. COS (Double-hatted as CA Senior Concept Developer) c. KM Chief d. Medical Team Lead e. EBP, EBE and EBA Staff


f. EBA KM Liaison Staff g. Maritime Planner h. CTFHQ Command Staff Executive Assistant i. CTFHQ MNIG Executive Assistant j. CTFHQ Knowledge Support staff k. Control Staff for Maritime l. Control Staff for Logistics m. Control Staff for Transport n. Control Staff for DIE o. Control Staff for Intelligence p. Transport Operations/ National Logistics Rep to LCB

The CFEC MNE 4 team-manning chart is outlined in Table A.2, and the US JFCOM Experiment manning Document is available electronically upon request. MNE 4 players participated in a training session during 22-24 Feb 06 and then participated in the experiment during 27 Feb - 17 Mar 06. MNE 4 players from organizations external to CFEC reported to the CFEC facility at CRC Shirley’s Bay campus, Ottawa. Personnel were authorized TD travel and coordinated TAN, travel and accommodation arrangements with CFEC Administration staff. Dress was normal working dress.

TABLE A.2 CFEC MNE 4 TEAM

Cell Post Name Home Unit Experiment Management CA Experiment Director LCol Kachuik CFEC CA Experiment Coordinator WO Crosby CFEC Concept Development CA Concept Developer Mr Rob Grossman Vermaas DDA Experiment Analysts CA Lead Analyst Dr Farrell CFEC KM Analyst Dr Allen CFEC Culture Analyst Dr Lichacz CFEC KBD/SoSA Analyst Mr Comeau CFEC Data Collector Mr Burrows Contractor Data Collector Ms Hawton DRDC Toronto Data Collector Mr Hughes CFEC Data Collector Mr Labbe CFEC Data Collector Ms Sullivan-Kwantes DRDC Toronto Technical Staff Technical Lead Mrs Harding CFEC Network Mgt Specialist Mr Baziuk CFEC Experiment Control CA Lead Controller Mr Comeau CFEC CA Deputy Controller Capt Campbell CLS/LFDTS Grey Cell Grey Cell Capt Arcoutte J4 Log Grey Cell Capt Porter JSG HQ J4 Grey Cell Lt(N) Allan HMCS Winnipeg Grey Cell Ms Orick S&T HQ Grey Cell Maj Von Papineau J2 Cap Dev Grey Cell Lt(N) Swan 3 Int Coy Halifax


CTF HQ players SCD/COS Names are not given to KM Chief maintain participant KM Team confidentiality KM Team KM Team KM Team EBP Sustainment Officer EBP and Medical Lead EBP IO Planner EBP Green Planner EBE Maritime Planner EBE Land Planner EBE PIO EBA DIE Assessor CTF HQ Command Gp EA CTFHQ MNIG EA JFCOM Liaison US Liaison Officer Mr Lavoie US JFCOM

Public Affairs

The aim of the public affairs activities related to MNE 4 was to:

a. Improve understanding of the potential significance of EBO and related concepts to the conduct of future CF operations;

b. Promote awareness of a major multinational experimentation event that is central to initial JCD&E activity contained in Plan Pegasus;

c. Explain the benefits derived from CFEC’s cooperation with key allies and NATO on CD&E initiatives aimed at transformation of future military capabilities; and

d. Demonstrate CFEC’s capabilities associated with attaining IOC in Nov 03, including the operation of the CFBLnet/CFXnet.

Maj Sherwood and Ms Lawson coordinated Public Affairs activities during MNE 4 including a Vanguard, Maple Leaf and Defense News submission, as well a Media interviews with CBC Radio and CTV Newsnet. The following are brief descriptions of the themes communicated during the Public Affairs visits.

EBAO concepts being examined and refined during MNE 4 would contribute to the transformation of future coalition operations. EBAO responds to the increasing complexity and uncertainty of crises by integrating diplomatic, information, military and economic effects to achieve strategic aims and operational objectives.

The US and UK have clearly indicated their intention to employ EBAO during future operations while NATO is examining the concept for application to the new NATO Response Force. To maintain interoperability, Canada must develop the capability to participate in a coalition employing EBAO techniques.


EBAO is based upon an integrated approach employing all instruments of national power to the conduct of military operations. EBAO will enhance the conduct of joint, combined, and interdepartmental operations.

Participation in the MNE series permits Canada to assess the relevance of EBO to Canada and subsequently develop an appropriate effects based approach to CF operations.

Participation in MNE 4 permits Canada to collaborate with key allies and NATO on CD&E initiatives aimed at transformation of future military capabilities.

Visitor Programme

CFEC conducted a visitor programme for CF/DND and external personnel invited to CFEC for a MNE 4 briefing and tour. Maj Sherwood coordinated the Visitor Programme during MNE 4. The objectives were to:

a. Improve understanding across the CF/DND of Effects Based Operations (EBO) and related concepts. Create awareness of the potential applicability of an effects based approach to future CF Operations; and

b. Create a positive awareness of CFEC and the value of its JCD&E role to the CF/DND.

Most visits took place during 14-16 Mar 06, although some visitors were received on an exceptional basis at other times. Timings were arranged for each visiting group to avoid peak periods of experiment activity where possible. The visitors included DM, VCDS, two Senators, Chief of Transformation, DGOR, DG DREO, Comd Canada COM, CDI, DFAIT, JIIFC, and RCMP to name a few.

Visitors were met at the Commissionaire’s building at Shirley’s Bay and escorted to the Battlelab facility. An escort accompanied each group throughout the visit and returned the group to the Commissionaire building upon completion of the tour.

LCol Tom Gibbons, A/Cmdt conducted an initial overview briefing in Conference Room B within DRDC Ottawa. Visitors were escorted through the Battlelab to receive further briefings and see demonstrations of processes, organizations and technology associated with EBA. Briefings were provided at the following cells:

a. Experiment Direction/Control– LCol Kachuik, Mr Comeau; b. CTF HQ staff cell – Mr Patrick, Maj Horcezy c. Int Cell – Maj Von-Papineau d. Analyst cell – Dr Farrell e. Applicabilités – Maj Von-Paineau, Lt(N) Swan, Lt(N) Allan

Distributed MNE 4 Training

A comprehensive international developed training programme, mainly computer based training, was distributed to all MNE 4 participants one month prior to MNE 4. Each package contained individually study packages outlining relevant areas to be reviewed prior to arri ing at CFEC or deploying to JFCOM.


The four individuals reporting to NATO were provided with background reading material prior to their arrival in Istanbul for one week of formal training.

Senior Concept Developer (SCD) Ramp-Up Training

Organized by US JFCOM J9, this component provided concept and tool refresher sessions for National SCDs from each nation participating in MNE 4. MGen (ret’d) Alain Forand was Canadian SCD. Both he and LCol Kachuik, National Director, took part in this training conducted at Suffolk, VA in mid January.

MNE 4 Training Review Week

22-24 Feb 06 was set aside for MNE 4 training. The training was designed to provide all users an introductory level of understanding for the concepts and the applications that would be used during MNE 4. All participants successfully completed a threshold knowledge review of the basic concepts.

time Wednesday Loc’n Thursday Loc’n Friday Loc’n

0800-0815

0815-0900

Mind Manager Maj Dan McNamara

BL Info Ops Lt(N) Givogue

BL

0900-0945

EBO Process Overview LCol John Kachuik

BL MNIG Rick Patrick

BL

0945-1000

Break break

1000-1045

EB TOPFAS Karen Harding

BL detailed review & post assessment survey CPO2 Knight

BL

1045-1130

KM CPO2 Knight

BL Treasure Hunt CPO2 Knight Closing Remarks Experiment Director

BL

1130-1245

Dinner

1245-1330

Main Gate access - photos WO Crosby

Main Gate

IWS CPO2 Knight

BL

1330-1415

Welcome - Experiment Director Overview of the experiment process - Paul Comeau

Rm 211

CD CIE CPO2 Knight

BL


1415-1430

Battle Lab security brief. WebCOP accounts initiated Karen Harding

BL Break

1430-1515

Trg Overview, confirmation participants completed CBTs, distribute test study pkg CPO2 Knight

BL WebCOP Mike Collins

BL

1515-1600

Administration Review - WO Crosby

Rm 211

WebCOP Mike Collins

BL

CFEC Admin O coordinated temporary passes for all personnel augmenting CFEC in Shirley’s Bay. These passes provided access to the Campus gate, Building 4 and 29. A minimum of one CFEC member remained in the Battle Lab whenever MNE 4 personnel occupied the lab.

Note:

a. All personnel require access to Shirley’s Bay Campus, Buildings 4 and 5 and the Battlelab b. All augmentees are cleared to Level II - Secret c. All personnel required a Battlelab PIN (MCpl Welton)

Visits

The Experiment Director was the OPI for the authorization and coordination of all MNE 4 related visits. The Battle Lab was configured to minimize disruption caused by such events, and the Experimentation timings above permit visits both outside of the dedicated work, and within the normal unit work hours.

Telephones

Rental of cells phones for MNE 4 was not authorized. Personal cell phones were not allowed in the Battle Lab. CRC Emergency number is 998-0000.

Rations

Due to the continuous nature of MNE 4, participants spent the entire working period at their workstations. Two meals per day at public expense were authorized and took the form of box lunches. When box lunches were not available, the Experiment Director was delegated the authority to order a lunch meal for the staff. The maximum value per person did not exceed TD lunch or dinner rates as appropriate to the MNE 4 schedule. A MNE 4 member, designated by the Experiment Director, submitted on a CF-52 with a nominal role including a receipt for actual expenses. In addition, drinks and light snacks were authorized for MNE 4 participants. The MNE 4 Coordinator coordinated the procurement of these items. Personnel on TD did not claim meals when provided by CFEC. CFEC MNE 4 Coordinator coordinated MNE 4 rations


Transport

During MNE 4, there was a requirement to support CFEC personnel and augmentees for travel to/from work due to the non-standard hours required and the lack of public transport between 1200-0600 hours. The Experiment Director rented a full size vehicle to transport those personnel that would normally utilize public transport. Only drivers authorized on the rental contract to drive the vehicle did so. Taxis were not authorized. CFEC Admin O coordinated transport arrangements.

Postal

CFEC Orderly Room provided regular work hours postal support. Any mail for non-CFEC personnel was addressed to:

Canadian Forces Experimentation Centre National Defence Headquarters 101 Colonel By Drive OTTAWA ON K1A 0K2 Attention: MNE 4 – Individuals Name

Medical Evacuation and Hospitalization

There was no change to existing medical and dental procedures for military and civilian personnel. CRC Emergency number was 998-0000.


ANNEX B: Technology Results

The underlying technologies that support EBAO from the physical level up to the software application level were examined in order to establish their impact on the command and control aspects of MNE 4. Sweden took the lead for coordinating the technology assessment. For simplification this annex groups the underlying technologies under two categories:

1. Networks which include the Combined Federated Battle Lab Network (CFBLNet) and local-area networks (LANs) used at various locations, the application servers, and the computer setups provided to participants which include the required operating system.

2. Applications for collating, managing and sharing data, for decision aids, for geospatial and tactical information, and for collaboratively generating, managing and sharing plans.

The applications are listed in the main documents of the experiment. Data were collected from the following applications: ALLIANCE, DMS CUPOL, EXCHANGE, JOANA, ONA, CD-CIE Portal, EBTOPFAS, IWS, JSAF, and the WEBCOP. There seem to be no assessments for other applications such as Oracle, the MS Office Suite (Outlook) and Adobe Acrobat.

To capture the network side of the underlying architecture Figure B.1, prepared by JFCOM and DISA, provides the labels (including IP & Port pairing) of the components, the capacity of each leg of the CFBL network and the servers at play.

The peak value in Megabytes per second (Mbps) of channel utilization for the aggregate of target applications (06 March – 10 March, MNE 4 3rd week) shown in Figure B.2 exceeds the capacity reported in Figure B.1. Beside the fact that the data of Figure B.2 are incomplete due to data that were not collected at some of the servers, it shows peak usage incompatible with the following utilization statistics in bits-per-second (bps) for each of the MNE 4 CFBL legs. These statistics indicate that no over utilizations of the channels available had occurred while, Figure B.2 shows peaks that could have caused the loss of services observed by the users.

1. JPO/NOC to CAN (11Mbps Link Data Rate)

a. Peak utilization in/out for period wk3 - 1.64Mbps / wk1 - 0.13Mbps

b. Average utilization in/out for period wk1 - 0.15Mbps, wk2 – 0.15Mbps /wk1 - 0.00012Mbps


Figure B.1 MNE 4 network/server diagram


Target Application Peak Usage

0

5

10

15

20

25

30

35

40

Targe

t Agg

IWS

ONA

EXCHANG

E

WEBCOP

CD-CIE

EBTO

PFAS

DMS C

UPOL

JSAF/J

OANA/ALL

IANCE

Applications

Mbp

sWeek 1 Week 2 Week 3Target Application Peak Usage

0

5

10

15

20

25

30

35

40

Targe

t Agg

IWS

ONA

EXCHANG

E

WEBCOP

CD-CIE

EBTO

PFAS

DMS C

UPOL

JSAF/J

OANA/ALL

IANCE

Applications

Mbp

sWeek 1 Week 2 Week 3

Figure B.2 MNE 4 application peak usage


2. J9 to JPO (20Mbps Link Data Rate)

a. Peak utilisation in/out for period wk2 – 1.46M / wk2 – 5.19M

b. Average utilisation in/out for period wk2 – 0.14M / wk2 – 0.64M

3. J9 to NC3A (20Mbps Link Data Rate*)

a. Peak utilisation in/out for period wk1 - 4.49M / wk2 – 5.52M

b. Average utilisation in/out for period wk2 – 0.57M/2.21M

4. J9 to UK (4Mbps Link Data Rate*)

a. Peak utilisation in/out for period wk3 - 2.08M / wk2 – 3.44M

b. Average utilisation in/out for period wk1 - 0.27M/0.69M

Another useful summary is the proportion of the total traffic generated by each monitored application. Figure 3.B shows that the total traffic was predominantly due to IWS and secondly due to CD-CIE. One can qualify these two applications as heavy and the others as light in terms of traffic load on a network.

Target Applications Comparison - Final Summary

CD-CIE13%

EBTOPFAS6%

DMS CUPOL0%

IWS71%

WEBCOP4%

EXCHANGE4%

ONA2%

Figure B.3 MNE 4 traffic proportion for the main applications

Peak delay and average delay statistics are essential in systems analyses. These delay statistics should be collected in LOEs in preparation of MNE5 in order to better understand what happens and predict what is needed from the applications, the middleware and the underlying networks and servers.


MNE 4 Technical Environment Facilitates Effects-Based Operations

COI 1: Does the MNE 4 technical environment facilitate the operational level of command’s ability to conduct effects-based operations (EBO)?

Finding: Yes it does but in a limited way.

Discussion/Evidence: The CFBL network successfully supported distributed EBO activities for the majority of MNE 4. There were only a few isolated, short-term instances of localized outages or reduced throughput. Canada had a hardware failure that was repaired within about 3 hours. With the exception of IWS and WebCOP, the CFBL net and MNE 4 tools suite were available to the participants >99% of the experiment day. Six primary tools were available for all MNE 4 participants to use: IWS, CDCIE Portal, MindManager, ONA, EBTOPFAS, and WebCOP. MNE 4 participants rated the usefulness for four of the six tools (IWS, CDCIE Portal, MindManager, and ONA) as neutral or better. They rated EBTOPFAS usefulness as slightly below neutral, while WEBCOP was significantly below neutral. The statistics supporting these statements are presented below in Figure B.4 where “1” on the y-axis means “not at all useful” while a “7” means “completely useful”.

27423085139142N =

MNE4 Tool Usefulness Ratings

WebCOPEBTOPFAS

ONAMindManager

CDCIE PortalIWS

Parti

cpan

t Mea

n R

espo

nse

6

5

4

3

2

Figure B.4 Tool usefulness rated by MNE 4 participants.

Comments and recommendations show that the participants were generally able to perform their duties, but the tools support was neither user friendly nor well adapted to the processes.

Of all technical trouble tickets, 51% were submitted on the six primary tools used by the participants. As shown in Figure B.5, the highest contributors to the tools related trouble tickets were IWS and the CDCIE Portal at 25% and 17%, respectively, of all trouble tickets submitted.


MNE4 Tools Trouble Tickets

0

50

100

150

IWS

Portal

Web

COP

MindMgr

ONA

EBTOPFAS

Application

Num

ber o

f Tro

uble

Tic

kets

Week 1Week 2Week 3

Figure B.5 Tool trouble tickets reported

Note that Figure B.5 indicates that fewer tickets, for IWS and in general for all tools, were filed the second week and even fewer the third week. With such data we have to be careful in concluding that fewer problems were observed as we were progressing with the experiment because alternative causes can be at play. For example, discussions with the players revealed that they cared less about reporting problems since they had lost confidence in getting them fixed, or in the case of IWS tickets, guidance from “control” encouraged participants to not report problems with IWS.

More information on this COI can be found in the JFCOM Final report (JFCOM, 2006c). The following recommendations and issues have been excerpted from that report:

Recommendations: The tools suite is not ready to be fielded. The tools need further development and integration with the processes to effectively support EBAO.

Sub COI 1.1: Do the MNE 4 tools facilitate communications and collaboration?

Finding: Even with the operational limitations imposed on Info Work Space (IWS), the primary collaboration and communications tool facilitated the required collaboration between the CTF staff elements and associated organizations to a significant degree.

Discussion/Evidence: Two tools surveys were administered, one at the beginning of experiment week 2 and one at the end of experiment week 3. The experiment audience rated both IWS ‘usefulness’ and ‘ease of use’ positively and improving over the course of MNE 4. IWS ‘usefulness’ improved from 5.16/5 (mean/median) to 5.44/6 and ‘ease of use’ improved from 5.18/5 to 5.47/6.

Finding: The availability of a good Common Operating Picture (COP) is essential. WebCOP was not able to fill that need.


Discussion/Evidence: The ratings for the WebCOP usefulness and ease of use were around 3 out of 7 and became lower during the experiment. Comments show that the participants wanted better maps and better interfaces to interact with the displayed information. The lack of availability of WebCOP hampered the use, as it could not always be used when needed. When the participants could get into the system it was slow and cumbersome to work with.

Recommendations: Conduct a study to identify tools that could provide the staff with a COP with improved functionality and availability (or improve an existing tool in order to provide the desired fidelity, user friendliness, and timeliness).

Sub COI 1.3: Do the MNE 4 tools enable information sharing?

Finding: The underlying tools philosophy for the experiment was “portal pull”, and this philosophy was sub-optimal during MNE 4.

Discussion/Evidence: The comments from the participant audience showed frustration with how and where to find information and how to use the search engine to find information. The ONA tool was also too complex to let the non-expert user easily use functions in the tool. MindManager was rated as useful, easy to use, and stable.

Recommendations: Further studies are needed to find a more useful information design and tailored information management tool for EBAO. In an environment with complex, dynamic processes like EBAO, information sharing requires supporting tools, processes and rules that provide more information “smart push” and less reliance on individual users having to “pull” all required information. An agent-based approach might be preferable in order for users to define an agent that will seek the needed information and its updates.

Finding: Extensive use of Outlook for information sharing hampered knowledge development.

Discussion/Evidence: During the first week, the instability of the tools forced the participants to use Outlook for information sharing which denied effective development of the knowledge base. This behaviour did not end when the tools stability became better

Recommendations: Implement effective backup procedures for information sharing that allow knowledge development.

MNE 4 Wargaming Tools/Decision Aids In EBO

COI 2: Do MNE 4 wargaming tools/decision aids have utility for planners at key points in the planning process?

Finding: Wargaming tools have potential utility for planners in the ADRM and SPR functions. The tools, however, must be well integrated into the process. The users must understand the capabilities and limitations of the tools as well as how to use the tools effectively. Have a common “look and feel” for all tool interfaces. Too many tools discourage untrained users.


Experimental Process Lessons

Based on the NATO code of best practice (COBP) for command and control assessment (NATO, 2002), the CCRP COBP for experimentation (Alberts and Hayes, 2002), and the TTCP GUIDEx (TTCP, 2006), more LOEs should be conducted to avoid repeating the following problems that obscured (buried in experimental noise) the facts and variables that need to be observed in support of the hypotheses:

1. poor technology integration,

2. lack of structure and training,

3. improper contingency plans in case a technology or application fails to work,

4. incomplete data collection and analysis of emails.

The underlying architecture of an experiment or the prototype used always interferes with the desired observations, but preparation reduces such inferences. So showing the service delays as function of time for all applications provides a good means for identifying such interference. For example, the decrease in IWS tickets from users from week one to week three can easily be correlated with the fact that players did not mind reporting due to lack of perceived fixes and due to the fact that control indicated that players did not have to report anymore since control was to take care of this problem. The decrease in tickets can also be correlated with other factors so further studies and experiments are required.


ANNEX C: Knowledge Base Development and Systems-of-Systems Analysis

Introduction

The Knowledge Base (KB) can be viewed, in part7, as a model of the operational environment (see main text Figure 1). Developing and maintaining an evolving KB to support an effects-based approach to operations is a challenging task. The Knowledge Base Development (KBD) process hopes to ultimately help staff begin to understand the complex system dynamics, and give them some insights in order to move forward with planning, execution, and assessment of effects-based operations. A key component of the KBD process is a System-of-Systems Analysis that is supported and conducted by subject matter experts called System of Systems Analysts (SoSA). SoSA are the key individuals that, in essence, perform systems analysis and help interpret the system dynamics. In general, all staff can achieve some level of systemic thinking in developing situational understanding.

It is recognized that traditional linear models and reductionist techniques do not sufficiently describe complex systems [1]. However, nonlinear modelling techniques have been successfully applied to certain aspects of complex systems[2]. Operational environments in the presence of conflict often exhibit adaptive, emergent8 and chaotic behaviours, some of which may not be possible to predict (unobservable states) or can not be influenced (uncontrollable states: see [3] for a full discussion on system observability and controllability). These socio-technical-economic-political systems consist of a large variety of subsystems9 with hierarchical structures that exhibit nonlinear and complex interactions and that learn and adapt over time. In open systems, there are exchanges of energy, information and materials between the system (as defined by the operational environment under consideration) and its external environment. System-of-System Analysis attempts to identify aspects of the system and define boundaries in order to begin to understand the system dynamics.

While it is generally appreciated that EBAO are knowledge intensive (as certainly demonstrated in MNE4), the traditional levels of command (strategic, operational, and tactical) tend to become blurred. Events and actions at the tactical level can quickly generate effects of strategic significance and may require changes at the strategic or operational levels. Ways to represent and model effects at varying resolution scales and scope need to be 7 In addition to explicit knowledge captured within the knowledge base as knowledge representations, there is un-coded and implicit knowledge as well as “know-how” that exists with staff interacting within the CIE. Knowledge base development is enabled by KM for knowledge elicitation, discovery, formalization and evolution. 8 Emergence can be defined as a process where the assembly, breakdown or restructuring of a system results in properties that are present in a macrostate but not present in a microstate, as defined by resolution and scope [31]. 9 Subsystems in the domains of Political, Military, Economic, Social, Information and Infrastructure (PMESII) have been recognized. Elements are often part of multiple domains so this taxonomy is not exclusive.


developed to support adaptive planning. The contemporary security environment is better characterized as a Complex Adaptive System where there are no perceivable cause and effect relationships or they are only coherent in retrospect and do not repeat. In such cases, a discovery approach of probe-sense-respond [4] to influence may be more appropriate.

There has also been previous recognition that Effects Based Approaches to Operations (EBAO) are knowledge intensive [5]. Despite inherent complexities, it is possible to develop some understanding and knowledge about the system, and this knowledge can be used to develop strategies to influence the system towards a desired end state. The limits of knowledge about complex systems have recently received greater attention by several authors ([6], [7]). It has been argued that knowledge is not absolute; rather it is a “reflexive” process, highly dependent on the perspective taken [8].

Although Effects-Based Planning, Execution, and Assessment (EBP, EBE, and EBA) staff members developed their own knowledge in order to perform their jobs, KBD was a formal and intentional process for developing knowledge representations and archiving those representations in physical databases. During the MNE 4 workshops, it was not clear whether KBD management would be limited to physical databases or would include managing the knowledge that reside within subject matter experts’ cognition, and whether KBD or Knowledge Management (KM) would manage the physical and/or cognitive knowledge sources. Regardless of who was to be responsible, the management of both knowledge sources needed to be done, and so KBD managed the physical databases while KM was responsible for liaising and soliciting knowledge from SME’s inside and outside of the organization. Thus within MNE 4, KBD involved requirements capture for knowledge and its representations, the development of a multinational KB that contain knowledge representations, processing of knowledge representations, and continuous systemic analysis of information and knowledge representations.

Concepts Related to KBD

As part of MNE 4, KBD and KM concepts were developed with prototypes implemented in terms of process, organization and supporting technologies ([9], [10]). These concepts were enabled by a Knowledge Support (KS) organization with a simulated Coalition Task Force (CTF) headquarters, and NATO Response Force (NRF) headquarters.

Knowledge Definition

In order to conduct this analysis and to discuss knowledge in the context of KBD, it is important to establish a clear notion for the idea of ‘knowledge’. Knowledge is a concept used with too much ease [6]. There continues to be a very active debate on the definitions of data/ information/knowledge (DIK), and defining knowledge brings one quickly in the domain of epistemology (a branch of philosophy that studies the origin, nature, and scope of knowledge).

Starting with the early philosophers, Plato’s definition of “justified true belief” provides a basis to examine knowledge. However, in the 1960’s Gettier demonstrated that although these conditions were necessary, they were not sufficient [11]. It was in fact possible to have justified true belief but for the wrong reasons. What this means for EBAO is that claims and


propositions must be carefully worded since our ability to link cause and effect may be under question (i.e., the model of the real world may be a poor imitation of the true complex dynamics, at best).

In general, knowledge can be considered as consisting of 1) concepts and 2) propositions. The identification of ideas, objects, and abstractions is certainly part of knowledge. In effect, such concepts are a way to view, perceive, or model the world. Although a given view may not be the best method, specific concepts allow people to communicate about ideas. Ideas such as heat, randomness, cohesion, morale and political influence are not so much absolute truths but rather constructs to look at the world [12].

The second area for knowledge consists of propositions that build upon elementary concepts. For the purposes of this analysis, we will use the following definitions:

Data are any signal upon which pattern generation and recognition occurs [13];

Information is the result of patterns of data that reduce uncertainty ([13]; note that the literature defines information and data recursively);

Knowledge is “interpretation” of information in context , represented as one or more propositions P on a subject S accepted as true or probable with belief , and encoded for use in reasoning by an individual (building on [14]).

Stated symbolically, knowledge can be represented as: ( , ) ( )K P P S μℵ = (this equation

appears in [15] except that context is not made explicit but is implied in the text). When multiple propositions are being considered, there is a conjunction-type operation that a person must perform and resolve to integrate the full set.

The “interpretation” of a proposition P on a subject S is key to knowledge and is represented by accolades . . This interpretation generates knowledge for an individual. According to this definition, information can be coded and stored in physical or computer systems while knowledge is the result of a cognitive process. We thus make a clear distinction between Information Management (IM) (mostly about the creation, distribution, categorization, formalization, application, and evolution of information) and knowledge management (KM) (mostly about people and their perceptions, communication, creativity, experiences, context, culture, and beliefs).

As in most situations, including a complex security environment, it is not possible to achieve the same interpretation and 100% belief about all propositions (i.e., complete knowledge that is referred to in MNE 4 as holistic understanding or entire Situation Awareness/Situation Understanding: SA/SU), but people are willing to accept claims or propositions as possible and useful for reasoning. The degree of belief is denoted , and is context specific. This degree of belief can be viewed as a Bayesian model and can be represented by a number between 0 and 1, or, where accurate specification is not possible, represented through subjective classes such as improbable, likely, or almost certain.


In the definition of knowledge, the importance of context cannot be overstated. Context can be defined as the circumstances surrounding a statement or proposition and it provides a set of underlying conditions used to interpret a proposition. It is the context that allows the formation of meaning. The context of an event (i.e., other information garnered from all five senses as well as memory), denoted as , includes the circumstances and conditions that surround the proposition.

The notion of “information within context” is consistent with the Concept of Operations for KM in the CIE [16], which states that knowledge is “the result of analyzing information and evaluating its meaning by placing information in proper context and applying cognition. The user begins to build an accurate picture of the situation through integrating and interpreting various pieces of processed data and information to enable meaning and understanding.”

Three fundamental types of knowledge are presented in [17]: descriptive, procedural, and reasoning. Descriptive knowledge characterizes the state of the world, real or projected. Descriptions of objects and concepts as described above fall in this category.

Different than mirroring the state of the world (or facts), there is also knowledge of how to do something. This knowledge can consist of procedures for handling a task. Examples of procedural knowledge include algorithms10, analysis strategies, and general methods. Like descriptive knowledge, procedural knowledge can be acquired as information, selected, put into context, given a belief value ( ), generated, coded into information, and shared as information. One prime example would be the EBO CONOPS itself on how to use an effects-based approach. In our case, the proposition becomes the “procedure” and the interpretation becomes its correct use. The proposition can be seen as “If you employ this procedure, then you will derive a useful result or solution” which is believed to be reliable to some degree, in a given context.

The third type of knowledge is reasoning. In this case the proposition becomes “if this situation occurs, then it is possible to derive a conclusion which is consistent or valid”. The basis for the proposition could be logic, correlation, analogy or “known” causality. This last type of knowledge is less often converted into a knowledge representation, except in areas such as philosophy and concept development. Experience is a form of knowledge which is not easily coded.

In all cases and types of knowledge, the user deals with sets of propositions in some context, and each requires adequate interpretation in order to assign a belief to the proposition. This analysis focuses primarily on descriptive knowledge, since that is what the KB attempted to capture.

Knowledge Representations

If knowledge is information interpreted in context that a human believes it to be true to some degree then knowledge is an internal, cognitive object – not perceptible by an external 10 There is a whole field dedicated to procedural information called “algorithmic complexity.” It uses metrics such as Kolmogorov complexity to measure the amount of information embedded in a program. It is notionally measured by the smallest set of instructions required to replicate the program.


observer. On the other hand, the raw information and its interpretation, context, and degree of belief may be all represented in the form of information that can be perceived. We refer to this group of information – the raw information, its interpretation, context, and degree of belief – as a knowledge representation.

Knowledge representations can be instantiated within a computer database, with paper and pen, as audio/visual recordings, or any other physical media that is perceivable by the five senses. It is the knowledge representations that are created, captured, managed, analyzed, evolved, distributed, applied and stored. Practically speaking, KBD is about the capture, management, and evolution of knowledge representations within a set of databases.

KBD Concept

KBD consists of four major activities [9]:

1. Guidance and Requirements Analysis (GRA) 2. Development of a Multinational Knowledge Base (DKB) 3. Knowledge Processing (KP) 4. Continuous Systemic Analysis (CSA)

GRA sets the context for the information. DKB attempts to make logical links between multiple databases. KP involves Knowledge Request Management, Knowledge Acquisition, and Knowledge Integration (strictly speaking, it is knowledge representations that are managed, acquired, and integrated into a physical database). It is still unclear whether the responsibility for KP lies with KBD or KM. CSA involves the interpretation of the raw information and generating knowledge representations called inference diagrams that show relationships between actors and objects within the environment. Influence diagrams establish (arbitrary) bounds of the system of systems under consideration and propose a linked set of dependencies (positive, negative) and attributes (delayed response, info flows etc). Degree of belief becomes important in reference to influence diagrams as per the knowledge definition.

Knowledge base development is intricately tied to sense making and organizational learning To achieve true learning, strategies must be in place to both ask the right questions, and to manage what can be gained (justified true belief – or knowledge). Knowledge development results from ‘belief revision’ as a process of developing new beliefs and changing beliefs to take into account new pieces of information. There is a large body of research on belief revision in the fields of philosophy, artificial intelligence, information systems theory and also propositional calculus in mathematical logic. Although the KBD concept did not rely explicitly on such formalism to evolve the KB, the concepts of inference in complex systems is a topic of possible future CD&E.

System of Systems Analysis Concept

The CSA concept and term replaced the Systems of Systems Analysis concept and term that was used in MNE 3. For all intensive purposes, System of Systems Analysis and Continuous Systemic Analysis yield the same product – a model of the operational environment. System-of-Systems Analysts (SoSA) are subject matter experts that stepped through the CSA process,


interpreted the information, and generated the inference diagrams. The concept calls for SoSA who understand fundamental aspects of all domains, with a specialization in one particular area because of the complexity of the system. Also, the concept calls for SoSA teams to be well established ahead of a crisis, although ad hoc teaming will undoubtedly occur in real situations. It makes it even more important to have a common methodology and language for analysis support.

Knowledge Base (KB) Concept

Ideally, complete knowledge representations would be found in a Knowledge Base (KB). More often than not, the interpretation, context, and degree of belief are not entered explicitly in the database, and so the knowledge representation collapses to the raw information and the interpreter must assume the context and assign their own belief to the proposition. However, the process activities of KBD aim to have not only the raw information in the KB but also some interpretation of that information in its proper (PMESII) context.

Strictly speaking, a KB in support of effects-based operations can be viewed as a set of propositions (with associated belief values) about the operating environment, opposing forces, neutrals and ourselves as well as how we should reason about these things. The KB is formed through prior knowledge representations contained in various databases or knowledge within people. The use and effectiveness of various knowledge representations are discussed later in the Annex.

Aim of the Experiment and KBD Issues

According to the Experiment Design Document (EDD), the stated aim of MNE4 was to examine and assess the ‘robustness’ of proposed EBO process, organization, technology and supporting concepts [18]. For this experiment, robustness was defined as “the reliability of the process over phases of the operation and the two staffs under experiment conditions.” This definition addresses the viability of the process during the experiment, and not necessarily its effectiveness in real or simulated conditions.

In this case, robustness is the ability of a system to continue to function across a wide range of operational conditions, and to fail gracefully outside that range. Examples of challenging conditions may include loss of staff personnel (due to re-tasking or absence), loss of connectivity within distributed networks, introduction of erroneous inputs or data, and increased workload or significant time pressure. The concept of scalability can be tied to robustness, in the sense that a robust architecture can be adapted in size and capacity to address some intended function.

The common design strategy that attempts to achieve robustness by predicting the conditions in which the system will operate, and then to optimize it for those (and only those) conditions, results in systems that are prone to fragility [19]. Only a few extraneous conditions were tested systematically (or by design) within MNE 4. The IWS network outages were reported in Annex B and there were isolated cases of staff being unavailable.


MNE 4 focused on the viability of the process, as per EBAO CONOPS ver0.90, used by a CTF staff of approximately 125 personnel working within a collaborative information environment. The results must therefore be interpreted as such.

KBD Hypothesis

KBD for MNE4 was one of the core components and key enablers to EBO. In fact, elements of KBD can be found in each of the main hypothesis statement ([18]; see also pg 27 and 28 of this report). However, the primary outcome of effective KBD is postulated to be the development of a holistic understanding and representation of the operational environment. Holistic understanding and representation will be analyzed in detail.

The KBD process contributes to establishing links between effects and desired end state through systemic analysis, creating coherent sets of actions that generate desirable effects while minimizing or managing unwanted effects, and performing holistic assessments of effects through the campaign. That is, KBD is a key element and contributor of EBP, EBE, and EBA.

Experimental Setup

In this section, we briefly describe the experimental setup related to the KB and System of Systems Analysts (SoSA).

The KB essentially consisted of all elements accessible within the Collaborative Information Environment (CIE) that includes the CDCIE Portal, Xythos document management system, the MindMaps, IWS cabinets, ONA database, or email data files). Of interest is the relatively large number of influence diagrams that were produced for the experiment.

A multinational SoSA team was established with 2 analysts responsible for each of the PMESII domains. It consisted of a core team of 12 individuals, with two analysts assigned to each of the 6 PMESII domains. There was in addition the possibility of national reachback. In addition to the SoSA team assigned to the CTF and NRF HQs, certain nations maintained permanent SoSA teams such as with Germany, the US and NATO. These teams were employed in direct support and provided substantive contributions during the experiment due to timeliness demands placed on SoSA.

Canada provided an Assistant Economics SoSA analyst as part of the CTF. Also, Canada provided analysis and executive summaries of issues for the Social and Economics domains as contributions to the initial multinational knowledge base. This research work was conducted under contract by the CIFP Group at Carleton University ([20]and [21]).

The core SoSA team met for the first time during training week on IWS. Although some individuals had worked together before, through pre-event workshops such as EBPlan1 development, many of SoSA had never met in person. A period of team forming is to be expected and this may have affected experiment results. The concept of operations for this extended team was to work through the KBD activities, but given time and resource constraints, the process in the KBD CONOPS was followed loosely at best.


Analysis Methodology

Findings documented in this report are based on user surveys, structured observations and data captured from C4I systems. Questionnaires were administered using the JDCAT survey administration system, while observations were captured by a multi-national team of data collectors in real time using the multi-user web-based JOT (Joint Observation Tool) application.

Non-parametric methods were used for ordinal data analysis. Many of the surveys captured data in the form of ordinal ratings. These data were generally based on a 7-point Likert scale (variants of 1-low to 7-high, with 4-neutral). In the cases where the data were ordinal, non-parametric methods were employed11 to analyze data, making no assumptions about underlying distribution (such as normally distributed responses), or uniform rating scales. Raw data distributions are reported directly with medians and percentiles of responses (typically 25th, 50th, 75th) using standard statistical techniques such as Box Plots [22].

Non-parametric tests for differences among groups included the Kruscal-Wallis (K-W) rank tests as a generalization of Wilcoxon test for more than 2 groups. For instance, the K-W test is a non-parametric version of one-way analysis of variance. The test is based on an analysis of variance of ranks of the data values, not the data values themselves. When there are only 2 groups, the K-W test reduces to the commonly known Mann-Whitney test.

When required, data was segmented by major staff Group – EBP, EBE, EBA, MNIG, Command Group (CG), and KS – within the CTF and NRF headquarters, or further categorized by sub-group (e.g., KS-KM, and KS-KBD, KS-Intel). Demographic data was also used when needed such as site, nationality, experience level, and organization (NRF versus CTF).

Overview of Critical Operational Issues

This section of the report addresses the experiment’s identified critical operational issues (COIs) related to KBD. COIs are questions related to the hypothesis and concepts being examined. These issues help guide the investigation, and are especially useful for data analysis and data collection planning.

COIs related to KBD and SoSA address:

Development of a holistic understanding of the operating environment;

Use and access of the knowledge base, over phases of EBO;

Knowledge representations;

Knowledge acquisition;

Knowledge integration; and 11 With data on an ordinal scale, it does not make sense to compute arithmetic means or averages.


System of Systems Analysis support and methods.

The following sections report on the Analysis of the Critical Operational Issues in turn. SoSA support has been integrated in related COIs.

HOLISTIC UNDERSTANDING AND REPRESENTATION OF THE OPERATING ENVIRONMENT

COI 1: Does the MNE 4 implementation of knowledge base development facilitate the operational level of command’s ability to create a holistic understanding and representation of the operating environment, the adversary, friendly, and neutral forces?

Overall Finding – The player audience did not gain a holistic understanding of the operating environment.

Finding 1 – The level of staff SA/SU initially increased after the first week but then stabilized to a level that was less than perfect.

Method

To quantify and estimate situational awareness and understanding (SA/SU), the experiment employed a SAGAT type survey [23] (see list of SA questions in Appendix 1). A total of 4 surveys called KBD1, 2, 3, 4 were administered over 3 weeks containing statements to be classified as True/False. Each question was qualified with a degree of belief or confidence (level 1-Guess, up to level 5-Certain). All statements were extracted from the KB, and were available to HQ staff through the CIE within the MindManager and Xythos Portal applications. Although the questions did not reflect particular incidents or control injects, they represented available reference knowledge about the scenario and the operating environment. In the case of KBD4, statements were specifically related to the CTF’s own effects-based plan, and the information could be found directly in briefings, plans or other products created by the staff itself.

Observations and results

Based on available data, it is possible to compare overall results between the CTF and NRF staff. KBD1 was given during week 1, KBD2 and KBD3 during week 3 and KBD4 in the third and final week of execution. In the case of the NRF, only KBD1 and KBD2 were administered.


KBD1 KBD2 KBD3 KBD4SURVEY

01234567 89

101112

Tota

l Cor

rect

CTF NATO

ORGANIZATION

Figure C.1 Distribution of correct responses for SA surveys, by organization

The average score for CTF staff across all SA surveys was 57%. The K-W statistic (115.7, p < 0.001) shows that the difference between KBD1 and KBD2 is statistically significant. However, there were no significant differences between KBD2, KBD3, and KBD4. Furthermore, the variability in distribution of scores did not decrease significantly as can be seen in the box plot distribution above.

Although we can conclude that SA increased initially, there remains the question of whether SA and depth of understanding was sufficient to conduct EBAO. Unfortunately, the MNE 4 design cannot provide answers to this question.

Finding 2 - SA/SU was not significantly different across CTF groups and individuals.


SA was not significantly different across sub-groups except for KBD2 (K-W= 22.7, p=0.007) where evidence is indicative of variation, as depicted in Figure C.2. There was wide variability of answers across sub-groups. In this case, MNIG and Command Group obtained median scores marginally higher than the other sub-groups. For the case of KBD3, differences in medians across groups were not significant as shown in Figure C.3.


CGEBA

EBEEBP KS

MNIG/CAG

GROUP$

0123456789

101112

TOTC

OR

RE

CT

KBD1KBD2KBD3KBD4

SURVEY$

Figure C.2: Box Plot Distribution of SA scores by Group and Survey for the CTF HQ staff

When the analysis is conducted by “Group” for multiple periods, EBA (with K-W statistics of 6.2, p=0.045) and EBE showed a tendency towards significant differences in SA over time (K-W= 5.9, p=0.058).

Looking at the highest performing individuals (not shown here for purposes of confidentiality), each group had something to contribute in terms of “knowledge” from their realized level of SA. This claim includes MNIG members who could very much complement knowledge of the military staff. As shown in the figure above, most of the individuals within EBA had achieved reasonably good SA/SU by the end of the experiment.

It is not possible to directly attribute high SA to solely KBD although the descriptive and procedural knowledge representations were found in the KB. Increases may have been due to thinking about the problem, staff interaction within the CIE, experience and individual learning. A different experiment design would have been required to establish this causal link, such as with a control group.

Many of the staff members reported various levels of confidence in their SA responses. If they had seen the information in the KB from a reputable source, they presumably would have indicated the right answer with high confidence.

SURVEY

GROUP

Tota

l Cor

rect


CGEBA

EBEEBP KS

MNIG/CAG

GROUP

0

2

4

6

8

10

12

TOTC

OR

RE

CT

Figure C.3: Distribution of SA scores for KBD3 by group for the CTF HQ staff

The answers to 12 SA questions generates up to 212 = 4,096 possible answers. Only one such combination is correct. It is possible to view how close various responses were to each other to detect clustering using Multi-Dimensional Scaling [25]. MDS takes points in a multi-dimensional space and maps them to a 2 dimensional picture that retains as much of the distance information among pairs as possible. A so-called ‘stress’ or error function is minimized for the mapped point distances. This technique has been used before to identify schools of thought in decision conferences [26].

Figure C.4 shows a 2-dimensional MDS projection for survey KBD3. The reference point, KBD3, is shown together with group labeled points, and points closer to KBD3 are also close in the original space. The result is that individuals from all Groups are close to the reference point. As the figure indicates, several MNIG members (non-military) had achieved high levels of SA as measured by the instrument. There is also a cluster of individuals who reported the same incorrect answers at the centre right of the figure (overlapping labels).

Tota

l Cor

rect


-8 -6 -4 -2 0 2 4-6

-4

-2

0

2

4

6

KBD3

CG

CG

CG

CGCG

EBA

EBA

EBA

EBA

EBAEBA

EBA

EBAEBA

EBA

EBA

EBA

EBA

EBE

EBE

EBE

EBE

EBE

EBE

EBE

EBE

EBE

EBE

EBE

EBE

EBE

EBEEBE

EBE

EBE

EBE

EBE

EBE

EBE

EBEEBP

EBP

EBP

EBP

EBP

EBPEBP

EBP

EBP

EBP

EBP

EBP

EBP

EBP

EBP

EBP

EBP

EBP

EBP

EBP

EBPEBP

EBP

EBP

EBP

KS

KS

KSKS

KS

KSKS

KS

KS KSKS

KSKS

KS

KS

KS

KS

KS

KS

KS

KSKS

KS

KS

KS

KS

KS

KS

KSKS

KS

KS

KS

KS

KS

KS

KS

MNIG/CAG

MNIG/CAG

MNIG/CAG

MNIG/CAGMNIG/CAG

MNIG/CAG

MNIG/CAG

MNIG/CAG

MNIG/CAG

MNIG/CAG

MNIG/CAG

MNIG/CAG

MNIG/CAG

MDS1

MD

S2

Multi-dimensional scaling of KBD3 SA Survey Answers BY Group

Figure C.4: Multi-dimensional scaling of KBD3 answers by Group

An attempt was made to identify possible contributors or drivers of high SA/SU. SA/SU as measured in the experiment was not significantly related to experience levels (as shown below, p=0.1 to 0.6 for a K-W test), but also level of education, nationality, or site within the CIE. In fact, the median score for individuals with 1-5 years of military experience exceeds the median score for 16-20 years experience (not statistically significant). None of the available demographic variables proved as discriminators for SA. It is possible that domain knowledge (or those who have had prior direct/indirect exposure to the Afghanistan theatre of Operations) may be a better indicator of high SA. There were no data to examine this claim.

Again, based on high individual scores indicated in Figure C.5, staff across a broad range of reported experience levels contributed to group SA. The challenge of course is that in a real world situation, one does not know who knows more. However, by considering and leveraging staff knowledge to the greatest extent, there may be a possibility of improving overall team performance.

Reference Point (KBD 3)

Reported same incorrect answers


1 to 5 years

11 to 15 yea

16 to 20 yea

21 to 25 yea

26 to 30 yea

6 to 10 year

More than 30

MILEXP$

0123456789

101112

TOTC

OR

RE

CT

Figure C.5: Distribution of Correct scores for KBD3 by experience level

Finding 3 - A majority of staff overestimated their knowledge about the operating environment and its inherent dynamics.


An individual would normally report valid propositions with high degree of confidence while low confidence answers to propositions would constitute guesses. Figure 6 shows the level of calibration achieved overall. This chart demonstrates that staff overestimated their level of knowledge.

Tota

l Cor

rect

Military Experience


467 291 496541

571

482 270 492395

296

0%

20%

40%

60%

80%

100%

1 2 3 4 5confidence

correct incorrect

Guess

CorrectApproximate

Expected Calibration

467 291 496541

571

482 270 492395

296

0%

20%

40%

60%

80%

100%

1 2 3 4 5confidence

correct incorrect

Guess

CorrectApproximate

Expected Calibration

Figure C.6: Distribution of KBD correct answers by user reported confidence level [27]

Additionally, there were no significant correlations between centrality measures obtained from the Social Network Analysis conducted by the UK analysis team for the entire organization and the score of the SAGAT-like surveys (KBD3 and 4) produced by the German analysis team. Maximum correlations obtained were on the order of 0.1. This may indicate that one’s knowledge of the operational environment was gained and internalized by different means or over a long period as opposed to through immediate direct interaction with other staff within the CIE.

Finding 4 – Some source documents were not exploited as part of the knowledge base.


As previously discussed, context can be critically important to understand information about a subject. To better understand the challenge of the effects-based mission facing the CTF and NRF, the US SoSA team produced an extensive background brief about the country, its history, internal dynamics, cultural factors, power struggles and popular attitudes towards the West [28]. A portal link to this brief was provided during training week but not observed to be actively referenced or accessed in the KB during execution. Informal query of staff indicated that very few individuals had read it. This knowledge product might have contributed to a better holistic understanding of the operating environment and the challenges to influence key players and the general population towards coalition goals.

In conjunction with the Command Group and HQ leadership, Knowledge Support staff need to consider what information is available and what knowledge is required for successful execution of the operation, and proactively make products easily accessible to staff. Assistance in prioritizing key references is an important function of KS staff.


Recommendations

1. Given the wide variability of knowledge and SA across staff, the EBO organization should strive to capitalize on available staff knowledge, recognize what is not yet known with a high degree of belief, and to prioritize areas for knowledge development.

To help achieve higher SA/SU, EBAO organizations should examine the use of decision support system (DSS) capabilities including the design and use of collaborative systems, search engines, as well as the optimal employment of liaison officers, and subject matter experts. DSS systems allow for direct contributions to be captured in parallel fashion for consideration by the group. Effective DSS use must be facilitated however in order to achieve tangible results. Leadership needs to ensure it recognizes the contributions of all its members including civilian members of MNIG.

2. Even in the presence of imperfect knowledge and situational awareness, the HQ staff requires common intent and level of understanding of each other’s roles and responsibilities (see Annex F). It appears that the SA/SU briefings resulted in achieving some degree of common awareness. However, the absence of a continuously maintained influence diagrams (the influence diagrams provided only a snapshot in time of the complex operating environment) may have contributed to variability in SA. Controlled experimentation will be required to examine this effect. An EBO headquarters should implement a capability to build and maintain a relevant recognized picture accessible to staff.

KM staff should take a proactive role to ensure that information deemed critical to the success of the mission is disseminated in an accessible and timely manner. Much time was expended each day attending the Commander’s update in order to gain SA/SU. Further experimentation is required to examine the best mix of means to attain sufficient levels of SA to contribute to the organization’s objectives.

3. Relevant SA/SU will vary over time and by function. It is important to recognize the importance of orientation to the problem and thus leadership must allocate sufficient time and resources to attaining adequate knowledge and situational awareness to contribute to the mission.

Conclusion

The development of SA/SU takes a significant amount of effort, both initially and continuously throughout a campaign. It may be as important to recognize what is unknown than what is known with a high degree of certainty and accuracy.

The above results provide a benchmark for the level of SA that was achieved in a simulated headquarters. The implied assumption is that higher SA leads to better planning execution and assessment. However, the KBD surveys were not designed to discriminate SA in terms of knowledge required by various staff elements and individual roles.


USAGE OF THE KNOWLEDGE BASE

COI 2: How intensive was the use of the Knowledge Base (KB) in different groups and phases of EBO?

Finding 5 - Various elements of the knowledge base were accessed and used through phases of EBO by most groups, although extensive use of detailed products was limited, both in extent and time.


Although it could be argued the use of the KB consists of access to the entire Collaborative Information Environment (CIE) composed of people and information/data sets, according to the CONOPS, the primary consolidated KB was to be resident within the MindManager application. Given the tempo of activity within the HQ, there were times when acquired or internally generated documents posted to the CDCIE portal could not be re-integrated within MindManager very quickly. In fact some documents were stored within the portal but not in the so-called consolidated KB.

Despite the above caveats, it is interesting to measure the use of the MindManager segment of the KB by the staff in the HQ. Figure C.7 shows the overall proportion of staff from the CTF and NATO who used the MindManager application to obtain data and information. Other than the Command Group and the Components (notionally participating as an experiment Control element), usage of MindManager was relatively high.

Results are similar between CTF and NRF, except for the Effects-based Assessment group (EBA). Assessment data within the NRF segment was not stored in MindManager rather it was kept in separate spreadsheets, and thus use by that group would be expected to be lower. This result does not explain however the use of the KB to gain SA to understand the context of the mission and surrounding facts and analyses to perform a holistic assessment.


CG

ComponentEBA

EBEEBP KS

MNIG/CAG

GROUP

0.0

0.2

0.4

0.6

0.8

1.0

Pro

porti

on o

f Use

rs

NATOCTF

ORG

Figure C.7 Proportion of staff using MindManager, by Group and organization

We then examine use of MindManager over time. Figure C.8 shows data for CTF only, by survey date. For all groups (except EBA), reported use of Mind Manager was maintained or increased slightly between 6 and 16 March.

CG

ComponentEBA

EBEEBP KS

MNIG/CAG

GROUP

0.0

0.2

0.4

0.6

0.8

1.0

Pro

porti

on

6-316-3

DAY


Figure C.8 Use of MindManager by CTF Group and Survey date

It is also possible to consider usability of the Mind Manager application. For those individuals indicating they had used Mind Manager, a set of 5 user-rating questions was administered. The results of these ratings for the CTF, over both survey dates are shown in Figure 9. Median ratings across groups ranged mostly from 4 to 6 (4 being neutral, 7 being agree, 1 indicating disagree). Overall the ratings from the Command Group exceeded those of other groups (non-parametric KW test with p<0.033 with 6 df) , although their reported use was lower, as shown previously. We note that the EBA and MNIG group ratings tend to be lower than those of other groups but this result was not statistically significant. These rating may have been confounded with the actual content of Mind Maps as opposed to the KB environment itself.

CG

ComponentEBA

EBEEBP KS

MNIG/CAG

GROUP

0

1

2

3

4

5

6

7

8

Use

r Rat

ing

MindMgr was StableMindMgr was RobustMindMgr Look and FeelMindMgr Ease of UseMindMgr Overall Usefulness

QUESTION

Figure C.9: User ratings for Mind Manager for various questions, by CTF group

Similar ratings have been derived for the CDCIE but are documented in Annex B.

The above results show promise for KB environments such as Mind Manager. Early concept development efforts by the German KBD concept development team focused on more flexible ontologies to organize information in various ways (possibly dynamic and user specified). Software tools such as Protege that provide this flexibility were examined, however, developing a working prototype and deploying it in time for MNE 4 was not possible. This is a field that is quite active in the research domain and could be a topic for future investigation in terms of user-defined ways to logically organize information.


Recommendation

Future Knowledge Base Development efforts should be carried out with EBAO staff members in concert with Subject Matter Experts (SMEs) who can guide those users to the right sources and to assist in interpretation. A distributed and scalable concept for Knowledge Base Development should be explored based on indexing trusted sources. Software applications such as Autonomy could be trialed.

Conclusion

The effort dedicated to developing a KB for the MNE 4 scenario and during execution was substantive (several person years). Much of the effort was provided by a single nation within the experiment. In future operations, some form of robust distributed model involving coordination of KBD activities among multi-national partners will be required.

PREFERRED LEVEL OF DETAIL FOR EBAO

COI: Which type of knowledge representation is best suited to which part of EBP, EBE, EBA?

Finding 6 - Influence diagrams were not the preferred knowledge representation about the operating environment or EBAO, especially as compared to slide presentations and short summaries.


The consolidated KB contained several elements. In addition to executive summaries assembled for each of the PMESII domains pre-experiment, it contained the following elements as shown in Figure C.10:

Primary actors (organized by class12 – e.g. Powerbrokers, International organizations, non-government organizations, etc) and a synopsis for each,

PMESII Domain Overview maps,

Special topics,

Influence Diagrams related to special topics, actors, effects etc,

Planning Products (noted as EBX), and,

Cross Reference Indices

12 MindMaps provided links to subordinate Mind Maps for more detailed information.


One of the problems observed and noted frustration from users was that certain elements, such as actors, could be found in multiple MindMaps at varying levels. The information store was a veritable web of topics and information that could be classified and organized in multiple ways. Although some users found the KB informative to navigate, explored its content, and educated themselves about elements of the system, searching for useful information using general keywords was problematic. For example, the Mind Manager search capability could not access the actual content of descriptions, just the label associated with various branch titles.

Figure C.10: MindManager Layout for KB Access

The staff used only a small fraction of influence diagrams. Some of these proved useful during the experiment. However, very few (if any) were observed to have been developed collaboratively across the MN community of SoSA experts.

User preference for knowledge representations is shown in Figure C.11. In general, users prefer presentations, short summaries and text documents. The ONA database was primarily used to document individual nodes or entities that were also accessible through Mindmaps, and so usage was marginal. Influence diagrams representing dynamic aspects of the environment were not a preferred knowledge representation of most staff. There are many plausible reasons for this finding. First, staff may not have had the necessary training to


interpret influence diagrams. Second, the influence diagrams were not validated to a sufficient degree. Nonetheless, these were often used by the KBD staff to brief the Command Group about aspects of the evolving situation in the area of interest. Influence diagrams were seen as a SoSA tool and product that needed simple explanation in context.

0%

20%

40%

60%

80%

100%

CG EBA EBE EBP MNIG All

group

Other Text documents Short SummariesPresentations ONA MindmapsMaps Influence diagrams

Figure C.11: NRF User Knowledge Representation preferences (top 3) by Group [27]

Recommendation

Influence diagrams are a higher order representation of systems, containing select entities/phenomenon, and relationships. There should be acknowledgment that these representations require careful interpretation and may not be for everyone. The influence diagrams are primarily SoSA analysis tools and not necessarily for all EBAO staff members. Therefore the EBAO designers should identify and systematically match tools to people and visa-versa. Confusion, error or serious misinterpretation could result otherwise. The applicability and validity of identifications are important knowledge management issues that require further attention.

Finding 7 - Some influence diagrams were used extensively by CTF KBD staff to frame the campaign problem although the influence diagrams were not necessarily validated.


At the first situation awareness briefing to the Command Group and staff on 1 Mar, a general update to the security situation was given for the AOR together a list of incidents in the previous reporting period. To achieve the desired End State, the Commander’s Approved Effects List (CAEL) was shown in Figure C.12. The proposition made was that it was not


possible to deal sequentially with “effects” and that they were in fact related with the identified feedback loops. The statement made was that “a delay of 15 days in achieving the first effect would lead to a delay of 15 days in all other effects. Actions on one effect, always influences all other effects.” Given what was known about the system Afghanistan, this was an interesting hypothesis, although not necessarily validated by direct evidence.

E001 - Security and Freedomof Movement for Allied

Infrastructure andReconstruction Efforts

E002 - Security forHumanitarian Assistance

Operations

E003 - Basic Package ofHealth Services to Afghan

Population E004 - CitizensAcquainted with Legal

System and Gov't

E005 - Schools HavingSufficient Water andSanitation Facilities

E006 - Anti-GovernmentForces Rendered Ineffective

E007 - Gov't Security ForcesMaintain a Stable, Secureand Lawful Environment

+

+

+

+

++

+

+

++

+

+

Figure C.12: Command's Approved Effects List (CAEL) and Dependencies of Effects

The relationship of linked effects appears plausible at first sight, but the point to make here is that this particular representation and resulting “belief” was not questioned or investigated further. The initial CAEL diagram was maintained in its original form in the KB throughout the experiment. This diagram may have been the most relied upon and important “mental model” of the campaign.

Some planning staff initially proposed that only the first few effects should be considered in order to be able to go through the EBO process as written in the CONOPS. It was proposed at one point that effects E005 thru E007 would not be considered so as to speed up the process , this despite one of the goals of EBO to gain a holistic understanding. These effects were eventually recognized as contributing to root causes of the operational problem and became part of planning considerations. Effect E008 on Effective Delivery of Humanitarian was added later in the campaign as the scenario evolved.

In planning actions to generate desired effects, there were no explicitly observed inquiries as to the possible root causes for the effects not being achieved. Lists of actions predominantly based on available resources were developed that potentially could contribute to the effect. On the other hand, the EBA staff did use the KR process in order to find whether the effects were attained. However, less than 15% of KRs were answered adequately. They ultimately relied on intelligence reports, but they did not contain any non-military information until week 3.

The next example of influence diagram that was used to support EBO is related to Humanitarian Assistance Operations and effect E008. Figure C.13 is a slightly modified ID presented on 14 Mar describing the inferred main influences of humanitarian assistance operations. The original diagram did not identify the improving conditions in the IDP camps


as a major factor driving international media coverage. Media coverage was assumed to be driven directly by local Gov’t support and credibility. Another modification to this diagram were annotations indicating the presence of feedback loops. A reinforcing loop has been identified where humanitarian aid directly improves situation the situation in IDP camps, resulting in media coverage, increased foreign funding, support by IOs and NGO’s, and additional aid delivered. A balancing loop is also present, whereby improved conditions in camps and Government credibility is met with increased anti-government force activities and possibly threatening lines of communication and supportive media coverage.

What this representation is indicating is that opposing forces are “expected” to react to progress from the coalition regarding humanitarian assistance and improved credibility of the local government. Although this is a very simple example, there could be much more complex feedback loops, specially since opposing forces are known to be segregated into factions and do not necessarily behave in a coordinated fashion as power brokers struggle for control in various parts of the country.

Military Activities ofCTF and National

Forces

Credibility ofGov't

Social andEconomic Activities

of Gov't

MilitaryActivities of Red Safe and Secure

Lines ofCommunication

Humanitarian Aid from Gov't and Coalition

ForeignFunding

Support by IOs,NGOs and Civil

Contractors

Support to Gov'tby Media

-

+

++

+

+

+

Credibility ofRed

-

+

+

Improved Situationin IDP Camps

+

Figure C.13: Diagram of Influence in Humanitarian Assistance Operations

Within the MindMap database, there were a total of 11 variants of this diagram13 with overlapping elements from 3 distinct SoSA authors (with the above diagram briefed by the 13 MindMap Afghanistan > Diagrams > Topics > HA Operations


KBD Chief but not integrated or incorporated in the final Mind Map.) It is unclear if SoSA members collaborated in deriving this systemic understanding. If the KB is to contain the most up-to-date knowledge about the operating environment, some selection process is required that may involve associating a degree of belief with the diagram.

Influence diagrams are developed for a purpose in a given context at a certain point in time. For each ‘knowledge representation’ (or piece of data or information), there must be an explanation (i.e., context) to assist possible users in its interpretation. Some of the description summaries were created but were not necessarily attached with the diagram itself. In some but not all cases, the MindMap database provided links to diagrams and associated explanations. There is of course the risk of staff misinterpreting a diagram or attempting to use knowledge embedded in a representation that is inappropriate for another use. Thus, SoSA are crucial to assist staff with knowledge interpretation.

Conclusion

Effects-based operations are conducted in complex and continuously adapting systems. Yet, there were no observed discussions on the representation of this complexity within the KB. Influence Diagrams provide dynamic representations of system element interactions, but it is not clear how these may change and evolve over time. Not only is the operating environment a complex system but also it continuously adapts. In addition, the concept of emergence – properties of the systems that exist at a given scope (such as rallying popular support) as a result of interactions was not considered. A study that examines concepts of complex adaptive systems (socio-economic and political) would benefit future EBO concept development.

Finding 8 - The content and scope of the consolidated KB did not change significantly over the course of the experiment.


The difference between the KB at the beginning and the end of the experiment amounts primarily to the inclusion of interim products of the planning process, staff briefings, control scenario injects and synthesis of information already resident. This is partially due to the security enclave and the difficulty in bringing new sources of information within the information store. The depth of information on specific effects or conditions remained relatively static.

Recommendations

1. Study the strengths and limitations of influence diagrams for abstraction of complex adaptive systems including the limits of verification and validation. Consider how to represent possible system adaptation and emergence.

2. Provide accessibility to open sources of information for staff conducting effects based operations. Technical issues will need to be resolved such as one-way data flows and security.


A separate network and set of information links may be required for relevant authoritative open sources of information. MNIG members reported this as a major shortfall.

3. Given the level of effort to collect, validate and structure large amounts of information related to a particular effects-based operation, it is recommended that KBD work start early and that some areas may benefit from continuous updating of a net assessment type of knowledge base.

Finding 9 - Visualization as a means to convey knowledge and meaning was underexploited.


A significant effort was undertaken by a sub-team of the Knowledge Management concept development team to visualize EBAO process, organization, and technology – that is, EBAO procedural knowledge. The final InfoWorkSpace and CDCIE Portal interfaces used a few of the ideas that were developed (see the Annexes of [10] for details).

On the other hand, there was little forethought on how to visualize EBAO descriptive knowledge – that is knowledge about the world states and how they evolved. During the later part of the experiment, consideration of a humanitarian assistance effect drew significant attention, as driven by scenario injects indicating a deteriorating situation. The daily SA/SU brief to the CTF Command Group and staff on Mar 10 attempted to synthesize in humanitarian crisis. A series of eight slides with maps and text were utilized to depict the situation. These cannot be replicated here due to classified labels applied. However, the slides were not integrated into a single (or compressed) comprehensive knowledge representation. In addition, a distance matrix between cities was presented when a map could have shown relative distances and travel times. The status of each camp was reported individually making it difficult to develop an overall understanding. There are certainly examples of visualizations that could have been produced in MNE 4, based on raw data (such as geo-plots and time series). One example of such work is on spatial analysis for criminal event prediction [29].

Since EBAO deals with complex systems, there may be multiple alternative views on the system. By clearly displaying, in context, what is observed about the system, and noting implied interpretation, the operator may be able to understand aspects of system dynamics [30].

Recommendation

Future experimentation should examine means of visualization data to support effects-based operations. The visualization of data and information should be cognitively compatible with the user’s mental model at many levels of abstraction and multiple time intervals. For example, a Component Commander may require local weather reports at minute intervals, while the Operational Commander may require a Farmer’s Almanac to determine the best time of year to provide Humanitarian Assistance, or to determine how natural weather patterns may affect Security Operations.


Displays that show physical objects moving on a map (i.e., Common Operating Picture) are fairly mature in terms of showing the right level of detail (and being able to turn the detail on and off). However, more thinking is required concerning how to integrate other non-physical layers onto such a map display (such as a political layer, an economic layer, a social layer, and an information layer). Moreover, the interactions within and between these layers would need to be made visible to the user. The user would need to have access to not only the raw data for these layers, but also see the data and information at various stages of integration and levels of detail [30].

Conclusion

The required level of detail for EBO is probably not the right question. Rather different levels of abstraction, levels of command, scales (time and space), shape, size, colour, spatial and location on the display may all required to understand and guide EBAO.

USAGE OF KNOWLEDGE REQUESTS AND GENERATION OF NEW INFORMATION

COI: Was the Knowledge Request (KR) function extensively used? Were the results adequate?

Finding 10 - Demands for knowledge was extensive, with significant variability in response times and quality of product.

General observations and results for this COI have been incorporated in Annex D, including specific SoSA and MNIG results.

Recommendation

Prioritize knowledge requirements based on information needs analysis. In addition to priorities, KRs should also be assigned an expected level of effort and suspense time for analysis products to be of use.

CONNECTION OF POSSIBLE NEW SOURCES TO MANAGE KNOWLEDGE SHORTFALLS

COI: Did the Knowledge Acquisition (KA) sub-activity ensure the matching of knowledge shortfalls with the connection of possible new sources?

Finding 11 - Information and knowledge requirements were not sufficiently articulated by the MN staff prior to execution.



Prior to developing the initial multi-national KB, the EBO CONOPS calls for Guidance and Requirements Analysis (GRA) activity to scope requirements for operational support. Although some of this work was conducted at the national level by Germany as the lead nation for the KBD concept (exact extent unknown), there was no derivation and approval of Commander’s critical information requirements (i.e., there was intentionally no formal mechanism in the CONOPS concerning these information requirements). These requirements were discovered, evolved and adjusted as the experiment progressed.

One key symptom of this outcome was the initial proliferation of KRs that were simply unanswered. There did not appear to be any prioritization mechanism established although it is mentioned in the CONOPS.

The KB for Afghanistan contained some 15,000 individual documents or files. The KBD effort must focus on information required to understand the situation (declarative knowledge) and to do the job (procedural knowledge). The Command team may also wish to allocate a small contingent and a fraction of resources to explore data and information unconstrained by assumed priorities.

Recommendations

Following End State Analysis, formally derive knowledge representation requirements and obtain approval by Command Group on initial priorities. Further explore the idea of a ‘Standing KR” as a means to provide updates to ongoing information needs. These reports could be based on regular time intervals or when events occur. The standing KRs would have visibility from all staff as well as the latest information updates to these KRs. Develop a database of indicators for tracking data about the operational environment. This database should indicate source, and other relevant meta-data.

Significantly improve the search capability within the CIE, to include both the portal and all its information assets in a unified search capability, and to people resources within and outside the CTF HQ.

Ensure SoSA team possesses baseline knowledge to support staff. Given the complexity of the operating environment, SoSA staff should be selected early and allowed to work as a team for extended period. Building contacts and networks of subject matter experts takes time and effort. If a SoSA type capability is provided within a Joint HQ conducting EBO, that resources be devoted to anticipatory type analysis in contrast to direct support to planning staff. Teaming and establishing a network of subject matter experts should occur before a crisis.

The establishment of analysts permanently assigned to certain portfolios should be considered within DND. Such individuals could be drawn from strategic analysts, intelligence analysts, social analysts, or other subject matter experts. In addition, generalist modellers should be assigned to a SoSA team to support the others in synthesising data and findings into more holistic models.


Finding 12 - Data acquired for the purpose of MOE and MOP development was not integrated in the knowledge base and made accessible. Several MOEs required for EBA could not be derived with available data.


While EBA staff collected much data to estimate Measures Of Effectiveness (MOEs) and Measures Of Performance (MOPs), much of this data was kept locally and not directly accessible through the KB. EBA kept data files used to assess effects and the progress of actions the logic of which was difficult to trace. If assessments were based on unbiased observation, these would be expected to become part of the knowledge base as propositions on aspects of the (complex) system. Observers noted that this data was not properly tagged as to source. The time, spatial and attribute characteristics of events could have been noted for later exploitation. The capability to manage data and access data series for various purposes was not evident. Staff reported difficulties to trace back EBA assessments to initial data sources.

As part of daily assessments from the EBA group, the realization of effects and progress of actions was reported. Several individual assessment items were observed to lack data (assessment briefs were tagged as classified so samples cannot be shown here). Although the situation improved with time (partially due to feedback from experimental control), EBO can involve phenomenon not directly observable, or for which change is slow to respond. This creates a challenge for effects-based planning, as feedback on the effectiveness of operations may not be readily apparent. This is a classical Control Theory problem, and there are solutions developed for this class of problems. For example, [3] is a Control Theory textbook that provides basic definitions of control and provides solutions for analyzing and controlling systems, and [2] provides standard solutions to nonlinear systems using linear and nonlinear control techniques.

Recommendation

When working an effects-based operation, it is important to be able to track evolution of the system of systems. A robust data management system, with ready access to time series and spatial data should be developed as a source of assessment and SA/SU on the progress of the effects-based operation.

Finding 13 - MNIG links to external knowledge sources were not available thus under-exploited.


In comments provided in surveys and through observation, members of external agencies expressed frustration they could not access or easily incorporate their own sources of information. In the words of an MNIG player: "To my mind, in this experimentation, the KB concept has proved an utter failure. It is heavy, it does not evolve easily and is in fact, very authoritarian in its assumption that there is one and only source of knowledge. But most of all,


it seems to me completely unrealistic. As [staff member] has aptly pointed out, in real-life, it would immediately clash with the need to consider outside information sources.”

Recommendation

Provide a capability for HQ staff to work both within classified domains as well as open networks for access to open source information and external subject matter experts.

Conclusion

In an attempt to find “authoritative” sources of knowledge in the KB for OP Secure Freedom, staff increasingly relied on people connections as sources of knowledge. They appeared to value “know-who” as opposed to “know-what” and where, as articulated in the UK analysis of the MNE 4 organization [24]. Interaction with knowledgeable staff is highly desirable but should not be the only means to gain knowledge. A search capability for internal and external information sources is required.

INTEGRATION OF NEW KNOWLEDGE INPUTS INTO THE KB

COI: Did the Knowledge Integration (KI) sub-activity ensure an integration of all inputs into the MN KB in a proper way?

Finding 14 – Knowledge integration was incomplete, and subject to central staff availability.


The model for knowledge integration adopted in MNE 4 was centrally controlled. The KBD staffs (from Germany) were tasked to integrate information elements into the KB. Given the effort required and delays, multiple observations show that staff often preferred to ask someone else rather than rely solely on the KB and its limited search function. There are instances where knowledge products were briefed to staff but could not later be located in the consolidated KB (due either to inadequate search capability or unavailability).

Recommendation

It is recommended that a distributed model of information management be developed and tested in future EBO experiments, and compared to a centralized model in order to determine the strengths and weaknesses of each model.


GENERAL CONCLUSIONS

Sense-making and organizational learning were not taken into consideration during the KBD concept development stage and therefore, they were not analysed. However, there was anecdotal evidence that these concepts are important to developing knowledge.

The issue of knowledge integration (KI) requires more attention. In MNE4, KI was performed mostly by co-located German staff. Even in this construct, coordination and responsiveness was a challenge. More should be done about examining multi-national information sharing and trust issues when data and information are distributed among many different sites and sources. Direct search and access to timely and authoritative sources of knowledge is generally what people seek to develop their own knowledge and add value to organizational goals.

MNE 4 tested the viability of the EBO process, not explicitly its effectiveness. This is unfortunate since so much effort and resources went into the design, execution and assessment of MNE 4. A different design would have been required to derive useful conclusions related to effectiveness. This would have required a much more realistic and dynamic synthetic environment. Consideration of EBO effectiveness may have to rely partially on what is experienced in the field in real world operations.

It is not possible to accept or refute the KBD hypothesis since the KBD process did not follow the CONOPS as stated, there were missing elements in the experimental design, and not enough data were available to show statistical significance in most cases. Despite these facts, much has been learned about KBD in support of EBO. From the above, we propose a series of recommendations.

Overall KBD Concept Recommendations

For future EBO experimentation, the following recommendations are brought forward:

1. Examine the robustness and effectiveness of EBO in relation to KBD, information distortion, time pressure, and other related factors;

2. Explore how concepts of complex adaptive systems can be applied to EBO and KBD;

3. Develop analytical techniques to integrate EBA and EBE products into the KB;

4. Consider means for effective information sharing across nations and agencies so as to ensure the visualization of propositions, the interpretation of propositions, and the assignment of appropriate levels of belief in the propositions are achieved;

5. Test procedures and technologies for information requests and knowledge development in distributed teams of varying sizes and composition.

6. Refine the concept of KBD in a multi-national and inter-agency setting;


7. Look at this process with a smaller core staff of say 30-50 individuals and under increasing levels of time pressure, staff turnover, with varying quality information inputs as might be expected in a live operational setting.

8. Consider aspects of the concept that can be used today such as strategic context and effects-development.

REFERENCES

[1] Axelrod, R., M.D. Cohen (2000). Harnessing Complexity: Organizational Implications of a Scientific Frontier, Basic Books, 2000.

[2] Slotine, J. E. and W. Li (1991). Applied Nonlinear Control. Prentice-Hall, Inc. A Division of Simon & Schuster, Englewood Cliffs, New Jersey 07632.

[3] Van de Vegte, J. (1990). Feedback Control Systems. Second Edition. Englewood Cliffs, New Jersey 07632: Prentice Hall.

[4] Kurtz, C.F., Snowden D.J. (2003). The new dynamics of strategy: Sense-making in a complex and complicated world. IBM Systems Journal, 42(3): 462-482

[5] Farrell, P. S. E., P. Comeau, R. Grossman-Vermaas, L. Lichacz, R. Patrick, S. Villeneuve and K. Wheaton (2005). CFEC Analyst Final Report for Multi-National Experiment 3. DRDC Toronto TM 2005-149, Toronto, Canada: Defence R&D Canada, June 2005.

[6] Cilliers, P. Knowledge, limits and boundaries. Futures 37 (2005) pp. 605-613.

[7] Richardson, K. The hedgemony of the Physical sciences: an exploration in complexity thinking. Futures 37 (2005) pp. 615-653.

[8] Richardson, K. et al. The Theory and Practice of Complexity Science: Epistemological Considerations for Military Operational Analysis, pp. 25-67.

[9] Effects-based Approach to Multinational Operations: Concept of Operations (CONOPS) with Implementing Procedures, version 0.90, US Joint Forces Command J9 Joint Experimentation Directorate, EBO Prototyping Team, 19 Dec 2005.

[10] Canadian Forces Experimentation Centre. The Coalition Task Force Headquarters Knowledge Management Plan. Version 0.9, October 2005.

[11] Chrucky A. (1997). Is Justified True Belief Knowledge? Edmund L. Gettier from Analysis 23 (1963): 121-123, transcribed into hypertext by Andrew Chrucky, Sept. 13, 1997. Retrieved 11 Sepember 2006 from http://www.ditext.com/gettier/gettier.html

[12] Kelly, G.A. (1955). The Psychology of Personal Constructs. New York, Norton.


[13] Shannon, C. E. (1949). A Mathematical Theory of Communication, from The Mathematical Theory of Communication, 1949 by the Board of Trustees of the University of Illinois. Used with permission of University of Illinois Press.

[14] Pigeau, R. (2005). The Human Dimension of Cyberspace. Presented at the CDS Initiative – DRDC Workshop. NDHQ, Ottawa, Canada, May 10 – 11, 2005.

[15] Allen, D., P. Comeau, and P. S. E. Farrell (2006). Knowledge Management as a Supporting Concept to Effects Based Operations. 2006 International Command and Control Research and Technology Symposium Coalition Command and Control in the Networked Era, Cambridge, UK, September, 2006.

[16] Knowledge Management in the Collaborative Information Environment – Concept of Operations, version 1.0, USJFCOM J9, dated 20 Sept 2004.

[17] Holsapple, C.W. Chapter 9 - Knowledge and Its Attributes, Handbook of Knowledge Management Vol 1: Knowledge Matters, Springer-Verlag, 165-188, 2002.

[18] US JFCOM, Multinational Experiment 4 Experiment Design Document (EDD) ver 0.90, with contributions from MNE4 partners, dated 10 Feb 2006.

[19] Gribble, S.D. Robustness in Complex Systems, 8th Workshop on Hot Topics in Operating Systems, May 2001, pp21-26.

[20] AFG Economic Indicators Executive Summary, prepared for MNE4 by CA SoSA Team Crisis Indicators for Foreign Policy (CIFP) Carleton University Ottawa, Canada, 8 pp, 21 Feb 2006.

[21] AFG Social Indicators Executive Summary, prepared for MNE4 by CA SoSA Team Crisis Indicators for Foreign Policy (CIFP) Carleton University Ottawa, Canada, 9 pp, 21 Feb 2006.

[22] SYSTAT 11 Statistics, User Manual, Systat Corporation Inc. 2004.

[23] Endsley, M. R. (2000) Direct measurement of situation awareness: Validity and use of SAGAT. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement. Mahwah, NJ: LEA

[24] JFCOM (2006). Multinational Experiment 4 Final Report (KBD Annex), US Joint Forces Command, prepared with contributions from Australia, Canada, Finland, France, Germany, Sweden, United Kingdom, and NATO, June, 2006.

[25] Van Deun K. and L. Delbeke (n.d). Multidimensional Scaling. Retrieved 11 September, 2006, from Open and Distance Learning Mathematical Psychology Web site: http://www.mathpsyc.uni-bonn.de/doc/delbeke/delbeke.htm

[26] Offiong, J. and P. Comeau, Fundamental Investigation of Defence Objectives – Decision Support to the Strategic Capability Planning Process, ORD Report PR 2000-20, Nov 2000.


[27] Bärtl, M., (2006). Multinational Experiment 4 Knowledge Base Development (KBD) Analysis Inputs for Reports. A draft document for insertion into JFCOM MNE 4 Final Report.

[28] US SoSA Team. Afghanistan Background Briefing (SECRET), ppt slides, Feb 2006.

[29] Xue, Y. Brown, D., Spatial analysis with preference specification of latent decision makers for criminal event prediction, Decision Support Systems 41 (2006) 560-573.

[30] Vicente K. J. (1999). Cognitive Work Analysis Toward Safe, Productive, and Healthy Computer-Based Work. Mahwah, New Jersey: Lawrence Erlbaum Associates.

[31] Ryan, A.J. Emergence is coupled with scope, not level, Nonlinear Sciences online, (draft), Sep 2006.


Appendix 1 to Annex C: Situational Awareness Probes

A series of questions were administered as part of a participant survey to gauge the level of individual situational awareness achieved as a result of the experiment. This technique is similar to the Situational Awareness General Assessment Technique (SAGAT) [23]. The technique assumes a relevant set of questions for the task at hand. The questions below were developed with extensive consultations with experts with experience in the theatre of operations. The questions also address three similar levels of SA: recognition of elements (e.g., raw information), recognition of interrelations (information in context), and the ability to project an element’s status in the future (prediction using knowledge representations or the model of the operational environment).

In contrast to the first three tests (KBD1-3), the last test KBD4 was developed during the experiment and addresses specific aspects of the CTF Effects-based plan under development. This last test was not administered to NATO NRF staff.

KBD1 Test

1. If international reconstruction aid continues in line with that up to 2001, then Afghanistan’s reconstruction needs will be sufficiently supported.

2. The number of soldiers trained annually for the ANA must be increased to meet the 3 year target strength of the ANA.

3. The life quality status of the Afghan population is much more sensitive to a change to the level of donations made directly to the Afghan government than the same change in donations made directly to the NGOs.

4. As of 2005, most Afghan provinces still had no Provincial Department to assist returnees.

5. NGO's are unable to operate in southern, south eastern and eastern regions because they have been specifically targeted by armed elements.

6. It is anticipated that a literacy campaign in rural areas would cause a significant backlash against education by a majority of Afghan rural population.

7. The PRTS benefit from their common organization and staffing.

8. Municipalities can be directly involved in the construction of public health facilities.

9. Securing ASF training facilities will contradict CTF’s planned actions.

10. Transferring government employees away from their home areas has partially reduced the power of local commanders.

11. The main source of telecommunications in Afghanistan is landline telephone systems.


12. Less than 40 percent of the Afghan population view the United Nations as "favorable."

KBD2 Test

1. Less than a 1/2 of the target number of health facilities was achieved in 2004.

2. The longer the CTF’s presence in Afghanistan lasts, the more it will be viewed as an occupational force by the Afghan population.

3. CTF will conduct a public affairs campaign with aggressive, forward-leaning public affair efforts to inform about western views.

4. ANP’s increased professionalism has resulted in a gain of the Afghan population’s trust in them.

5. CTF coordination with ANA and Combined Forces Command Afghanistan has solely to be done via the Afghan department of Defence.

6. The Afghan financial sector is almost entirely operated by the unofficial economy.

7. Introducing standardized documentation into Afghanistan’s main value transfer system will increase its viability.

8. Provinces are not authorized to independently borrow money.

9. In 2006, opium trade will be a minor source for funding Taliban.

10. Even after learning alternative skills, the majority of IDPs are unwilling to abandon their traditional nomadic livelihood.

11. To facilitate reconstruction the coalition should supervise Afghan people how to rebuild their homes in accordance with earthquake-mitigation techniques immediately.

12. In the last 3 years the Afghan cabinet passed a law barring NGOs from bidding for government contracts because some NGOs were responsible for widespread corruption and misuse of public funds.

KBD3 Test

1. The new university system in Afghanistan will help strengthen the democratic civil society in the country.

2. Extensive experience with teacher-training programs is UNDP’s key strength.

3. One of the current core functions of the Afghan NGOs’ Coordination Bureau is advocacy & policy.


4. Up to now there has been found no way to create strong unity within the ANA because of the mixture of different ethnicities.

5. In the mid to long term cultural traditions will make ANA soldiers continue to stay away from their units for long periods of time.

6. CTF assumes that Adversary Forces will very likely exploit international boundaries.

7. Assistance from the Pakistan Government combined with appropriate boarder agreements will reduce the number of illegal ACM border crossings.

8. It is very likely that next Taliban escalation will come in the form of "widespread urban terrorism" in the south, including bombings and assassination attempts.

9. CTF is authorized to communicate directly with the U.S. Ambassador and his country team for coordination in order to expedite CTF operations.

10. In winter there is no usable road link between Kabul and the western and north-western provinces for humanitarian operations.

11. In Afghanistan, doctors may skip their hours in the public facility if they run a private practice.

12. Food or tents should not be air dropped without having a reliable local distribution mechanism in place.

KBD4 Test

1. The foreign forces in AFG were accused by Afghan Clerics of abusing Children and poisoning the Faithful Muslims.

2. In recent attacks at a FOB and a PRT near TARIN KOWT insurgents were able to seize several US STRYKER Armored Vehicles.

3. Over the recent weeks the MOP-MOE plot indicated the stagnation of EBPLAN 01.

4. Effect E005 states: Hospitals and NGO installations throughout the CJOA-SA have sufficient water and sanitation facilities to ensure operation during a period of D+6-15 month.

5. Weather conditions in the CJOA pose a serious threat to CJTF operations and achievement of effects.

6. The disease recently observed in the region of Kandahar is contagious.

7. Effect E001 (Security and Freedom of Movement) has to be established until D+30.

8. According to EB Plan 1, increasing the number of escorted Convoys will affect the MOE for Effect 006 (ACM Ineffective) positively.


9. Effect E006 (ACM Ineffective) has assigned activities in each branch of the DIME spectrum.

10. CFMCC is supporting CFACC by flying AIR MISSONS.

11. The Commanders Intent is to carry out operations aggressively.


ANNEX D: Knowledge Management

INTRODUCTION

Knowledge management (KM) is an important enabler of EBAO. KM’s raison d’être is to govern and facilitate the knowledge activities. The governance includes the setting, regulating, monitoring and coordination of policies and procedures for knowledge activities and information products. Facilitating implies aiding and assisting various knowledge activities throughout the CTF HQ and its interaction with non-military organizations.

In preparation for the MNE 4 event, the Canadian Forces Experimentation Centre led a national and multinational investigation of the KM concept. US JFCOM supplied the KM Concept of Operations (CONOPS) and Concept of Employment (CONEMP) for the Standing Joint Forces Headquarters (SJFHQ) as a starting point for the investigation. CFEC hosted one national workshop and three multinational workshops to identify the key KM principles and develop a KM Plan for MNE 4 that incorporates those principles.

Figure D.1 KM Organization as Proposed by the KM Plan

The KM Plan14 version 0.9 was released on 20 October 2005; it includes KM activities, KM organization including Tactics, Techniques, and Procedures (TTPs), and sets of KM business rules for the MNE 4 technologies. The KM organization proposed by the KM Plan is displayed in Figure D.1. This organization is divided in two groups: on the left, the staff that focus on maintaining the network and ensuring effective working conditions of the information technology; on the right, the staff that focus on governing and facilitating knowledge activities among the CTFHQ staff. Both groups report to the KM Chief. This construct recognizes the need for both a technical and a team environment to achieve

14 CFEC (2005). Coalition Task Force Headquarters Knowledge Management Plan for the Multinational Experiment IV. Version 0.9, October 2005


organizational objectives. The detail of the roles and responsibilities of each KM staff is provided in the KM Plan.

The list of business rules provided by the KM Plan is divided in three groups: the Knowledge Request (KR) business rules, the Knowledge Base (KB) interaction business rules, and the Collaborative Meeting (CM) business rules. The KR business rules specify the procedure to be followed for processing the Knowledge Requests. The KR function was designed to meet knowledge gaps and shortfalls of all HQ personnel. The KB business rules stipulate some rules to regulate the CTF staff interaction with the KB. In particular, they provide a file naming convention. Finally, the CM business rules regulate the planning and the practice of collaborative meetings. In particular, they mention that a calendar should be maintained specifying the time, virtual location and agenda of all meetings.

KM Experiment Aim and Objective

The aim of the experiment was to assess the value of the KM Plan and evaluate the ability of the KM support function to govern and facilitate the knowledge activities. The hypothesis was:

If the KM Plan is applied and adhered to during MNE 4, then:

1. The governance and facilitation of knowledge activities will be robust (reliable, stable, scalable, not constraining) across the EBAO process; and,

2. The governance and facilitation of knowledge activities will contribute positively to the situational awareness, common intent, and decision-making processes.

The experimental aim was achieved by measuring: The ability of the CTF HQ staff to perform the knowledge activities; The extent to which the KM Plan was followed; The use by the CTF HQ staff of the KM resources; and, The extent to which the KM functions support the knowledge activities.

Due to the lack of a control group in the experiment design, a direct comparison of the quality of the knowledge activities and knowledge management support was not possible. The method was thus to perform a relative evaluation of the knowledge activities and knowledge management support. For instance, it would have been possible to compare the use of the KM resources between two groups and their ability to perform the knowledge activities. If the KM functions were to provide a good support to the knowledge activities then a larger use of the KM resources would lead to a better ability to perform the knowledge activities.

Methodology

Many factors other than KM influence the staff ability to perform knowledge activities. Among possible factors, the staff knowledge of the tools available and involvement in the development of the EBAO Concept of Operations (CONOPS) will certainly influence the


ability to perform the requested knowledge activities. Based on Ajzen’s behavioural theory15, a list of factors can be developed. According to this theory, three categories of factor influence an individual’s behaviour: The attitude toward the behaviour (Are the probable consequences of my deed compatible with my convictions?); external expectations or social norms (Do others, whose opinions matter to me, expect me to behave like this?); and the perceived behavioural control (Can I see anything that might help or hinder my carrying out this action?). These three factors can be summarized as the individual’s desire, the perception of external expectations, and their capability. From a CTFHQ point of view, these three categories include factors such as:

1. Desire: Organizational culture 2. Perceived Expectations: Understanding of own and others roles and responsibilities,

understanding of commander’s intent, past experiences 3. Capability: Language, training, access and knowledge of available technological tools,

time constraints

All these factors impacting the knowledge activities have been considered for assessing KM. More specifically, the correlations among 55 factors potentially impacting on the knowledge activities were considered. The list of factors and the correlation among the factors are displayed in Appendix 1 of this annex.

COI 1.1 – How well were the knowledge activities performed?

The quality of knowledge activities was determined by evaluating the ability of the CTF HQ staff to perform the various knowledge activities: create new knowledge elements, adequately interpret elements of information, associate an appropriate degree of belief to elements of information, and prioritize adequately the knowledge elements.

Overall Assessment: Technology issues and the lack of task specification limited the knowledge activities, in particular the interpretation and formalization of information.

a. Knowledge Creation

As aforementioned, there are two types of process leading to knowledge creation: by accessing new information elements or through a decision-making process. The ability of the CTF staff to perform both processes was evaluated.

The access to information depended on the staff ability to access products in the Knowledge Base (KB) and to share information among them.

Finding 1 – The CTF HQ staff had a limited knowledge of the KB content.

Observations

The CTF HQ staff had an average 57% score to the Situational Awareness (SA) survey.

15 Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179-211.


A significant improvement of the CTF HQ staff SA is observed in the second and third weeks (see Table D.1).

There were no significant differences among the situational awareness of the various functional groups (see Table D.2). The score and confidence intervals shown in Table D.2 are based on the overall SA score obtained by each functional group’s players.

TABLE D.1 MEDIAN FOR THE NUMBER OF RIGHT ANSWER TO THE SITUATIONAL AWARENESS SURVEY AMONG THE CTF HQ STAFF.

Population Median 95% CI of Median 1-Mar - Ratio of rights 112 0.417 0.333 to 0.417 6-Mar - Ratio of rights 115 0.600 0.600 to 0.700 8-Mar - Ratio of rights 118 0.583 0.583 to 0.667

13-Mar - Ratio of rights 113 0.600 0.600 to 0.700

TABLE D.2 SITUATIONAL AWARENESS SCORE FOR THE VARIOUS FUNCTIONAL GROUPS

Functional Groups Population Median 95% CI of Median CG 6 0.565 0.500 to 0.650

EBA 14 0.610 0.508 to 0.642 EBE 22 0.596 0.525 to 0.633 EBP 26 0.579 0.504 to 0.638 KBD 25 0.550 0.517 to 0.613

KM 12 0.563 0.517 to 0.625 MNIG 13 0.604 0.533 to 0.650

Finding 2 – The ability to access KB information was an important factor influencing CTF HQ staff’s SA.

Observations

From a total of 111 respondents, 55 CTF HQ staff answered that they had to seek most of the information themselves, 11 answered that most of the information was provided to them, and 45 answered an equal amount of information sought and provided (see Figure D.2).

The CTF HQ staff that expressed having to seek most of the information had generally a lower SA score (Kendall b correlation between ‘SA’ and ‘Having to Seek Info’ is -0.25).


0

10

20

30

40

50

60

Provided byothers

Both equally much Seek it out myself

Num

ber o

f Res

pond

ents

Figure D.2 Distribution of participants’ assessment on information access.

Finding 3 – Information accessibility was limited due to tools inadequacy.

A majority of the CTF HQ staff were dissatisfied with the look and feel of the Portal, ONA and MindManager tools. They also found these tools were not user friendly.

During the first week, some sites had limited access to the Mindmaps. Significant delays were required to load a map. This issue was resolved by replicating Mindmap data on servers at various sites.

Both the Portal and MindManager had inadequate search engine capabilities. In particular, MindManager’s search engine was limited to the terms appearing directly in the Mindmaps. It did not search through the documents or the text associated with these terms.

The ability to search the Portal could have improved in the third week when a link to Xythos search engine was provided. However, the number of requests for document locations that were observed in the text chat did not decrease when compared with the number of similar requests observed during the second week (see Figure D.3). The decrease observed after the first week seems to be due to the CTF HQ staff having learned the usual location of documents. Note that the increase from the 28th February to the 2nd March is correlated with the workload increase and may be attributed to the need for more information.


Number of Requests for Documents Location

0

5

10

15

20

25

2/27/2

006

2/28/2

006

3/1/20

06

3/2/20

06

3/3/20

06

3/4/20

06

3/5/20

06

3/6/20

06

3/7/20

06

3/8/20

06

3/9/20

06

3/10/2

006

3/11/2

006

3/12/2

006

3/13/2

006

3/14/2

006

3/15/2

006

Days

Num

ber

of R

eque

sts

Figure D.3 Number of text requests on IWS Chat for location of documents among the CTF participants.

Finding 4 – Information sharing was primarily constrained by technical and procedural issues as well as time constraints and the lack of knowledge of who needs what information.

Observations

The CTF HQ staff rated the importance of ten factors (listed below) in hindering the sharing of information. Although there was limited agreement among the players in evaluating the importance of each factor, a good agreement about which factors are more important was obtained. Using a consensus-ranking model (FIDO16), the following order of factors was obtained (in decreasing order of importance):

1. Technical difficulties,

2. Procedural barriers,

3. Time constraints,

4. Lack of knowledge of who needs what information,

5. Priorities,

6. Organizational culture,

7. Influence and power,

8. Culture differences,

16 D. MacLean. Fundamental Investigation of Defence Objectives User Guide for FIDO Version 1.0. DRDC-CORA, Technical Report, PR 2000/25, December 2000.


9. Approachability of the Commander,

10. Language.

Observations on the Knowledge Request (KR) process and the interaction between the CTFHQ staff and the MNIG confirm that the lack of knowledge of the staff roles and responsibilities had a negative influence on the information sharing. Similarly, there is a positive correlation ( b = 0.38)17 between staff workload and the amount of information shared with superiors and peers, indicating that time constraint would have limited the staff highly involved in sharing information.

Finding 5 – A majority of the CTF HQ staff were concerned with the quality of the information shared.

Observations

When asked about the quality of the information available, a large number of participants disagreed with the statements: “The information I received was complete for my tasks”, and “The information received was concise in the format it was presented” (see Table D.3). The number of participants disagreeing with the completeness of the information was particularly high within the EBA and MNIG.

Time delays to obtain an answer to an issued KR also support the view that there was an issue with the information timeliness.

The staff that felt the need to seek the required information by themselves were three times more likely to be dissatisfied with the timeliness of the information ( b = 0.462) and twice more likely to be dissatisfied with the information sufficiency ( b = 0.364).

The years of military services was also positively correlated with the perceived quality of the information (the Kendall- b correlation between the amount of military years and the sufficiency of the information is 0.358 and between the amount of military years and the conciseness of the information is 0.325). This result means that senior military officers either: 1) have lower expectations, 2) received better quality information, or 3) were more apt at understanding the provided elements of information.

TABLE D.3 CTF HQ STAFF PERCEPTION OF THE QUALITY OF THE INFORMATION

Agree Remains Undecided Disagree

Accurate 50 38 12

Appropriate 43 39 18

17 b is the Kendall-tau correlation. It is a non-parametric approach to compute correlations. The correlation between two sets of data is done by selecting a pair of numbers from each set and computing the number of concordant and discordant correlations over all possible pairs. For more information see D. Wilkie, Pictorial Representation of Kendall’s Rank Correlation Coefficient, Teaching Statistics 2, 1980.


Accessible 53 21 26

Relevant 51 27 22

Timely 45 29 26

Complete 35 33 32

Sufficient 43 31 26

Concise 36 30 34

Interpretable 52 26 22

Understandable 54 28 18

Finding 6 – There appears to be a competition between vertical and horizontal exchange of information.

Observations

The CTF HQ staff that indicated sharing a lot of information with their superior were more likely to be less connected with other staff members (correlations between exchange of information with superior and SNA degree is b = -0.287 and with betweeness b = -0.277). (SNA metric results for week 3 were provided by UK analysts.)

The CTF HQ staff that indicated sharing a lot of information with their superior were more likely to have a good knowledge of commander’s intent ( b = 0.333).

Finding 7 – Specific teams evaluated their decision-making ability as insufficient.

Observations

The CTF HQ staff evaluated their own team decision-making capability. Two teams rated the quality of the decisions made by their team as poor: the KM team and the SOSA team.

Decisions process observed within the KM team were done quickly without generating and evaluating many options. A good decision process should encompass: a clear definition of the problem issue, a brainstorming where a list of options are generated, an evaluation of the pros and cons of all options, the development of a common representation among the group, the support for the expression of concerns and doubts, and the facilitation of the development of an agreement. The lack of a good decision-making process might explain why some issues re-surfaced at different KM meetings.

In a few recorded instances, the KM Operations Officer (KMOO) and KS Intelligence Supervisor expressed concern about the decision process, stating that the decision was not substantiated by any analysis. Their concerns were not considered.


Finding 8 – Past experience and prior involvement in the EBO process impacted on the feeling toward the decision-making process.

Observations

The CTF HQ staff with concerns about the appropriateness and interpretation of the information were twice as likely to show concerns toward the decision making process ( b = 0.325)

The CTF HQ staff with a higher workload were similarly more likely to show concern towards the decision-making process ( b = -0.426).

The CTF HQ staff that expressed sharing less information with their superiors and peers were also more likely to show concerns toward the decision-making process ( b = -0.416 and b = -0.541 respectively).

The CTF HQ staff that mentioned having effects-based planning experience were also more likely to show concerns toward the decision-making process ( b = -0.397).

b. Knowledge Interpretation

Finding 9 – Information interpretation was observed as an issue in select instances.

Observations

Most participants did not consider the interpretation of the information as being problematic. However, it was evident that some participants had not adequately interpreted some information regarding the UN Humanitarian operation in Afghanistan modified involvement with the humanitarian aid. Similarly, the importance of Tarin Kwot was not clearly understood.

Finding 10 – Culture, language and experiences are factors impacting on the ability to interpret the information.

Observations

The CTF HQ staff who interacted with people of different nationality were more likely to be less satisfied with the intelligibility of the received information ( b = -0.312).

The CTF HQ staff having participated in the MNE4 process rehearsal were more likely to be more satisfied with their ability to interpret and understand received information ( b = -0.42).

c. Knowledge Formalization

Finding 11 – Information formalization was observed as a problem in select instances.


Observations

Conflicting information was shared among the CTF HQ staff about the withdrawal of UNAMA from the refugees’ camps, about incursions through the Afghanistan-Pakistan border, and about the imprisonment of a Taliban spokesperson.

No or very few formal clarifications of information elements were observed during briefs involving the whole CTH HQ.

Some conflicting elements of information were propagated for days (for example, the imprisonment or not of the Taliban spokesperson was only clarified in the 3rd week although the question came up on 2 March).

Finding 12 – Information formalization did not address the needs of a majority of participants.

Observations

Various observers identified a lack of knowledge of the participants regarding the veracity of the information received: Is the information based on corroborated facts or is it based on opinion? In agreement with this observation, a majority of participants expressed a lack of knowledge on the certainty and sources of the Knowledge Request responses.

Finding 13 – Training, past experience and involvement within the CTF HQ were important factors impacting on the ability to select an appropriate degree of belief.

Observations

The CTF HQ staff that expressed having to seek most of the information by themselves was more likely to wonder about the accuracy of the received information ( b = -0.394). This is expected since when someone provides the information, authority is more easily associated with the information.

The CTF HQ staff that expressed having a lower workload was more likely to wonder about the accuracy of the information ( b = 0.28).

The CTF HQ staff that expressed not having received enough training was more likely to wonder about the accuracy of the information ( b = 0.325).

The CTF HQ staff not having previous coalition HQ experience was more likely to wonder about the accuracy of the information ( b = -0.363).

The CTF HQ staff not having participated in the MNE 3 experiment or the MNE 4 process rehearsal was more likely to wonder about the accuracy of the information ( b = -0.469, b = -0.313).

Finding 14 – The ability of the CTF HQ staff to attribute a correct degree of belief improved with time.


Observations

The CTF HQ staff were asked to evaluate the validity of a list of statements four times during the 3 weeks (survey: KBD1 on 1 March, KBD2 on 6 March, KBD3 on 8 March and KBD4 on 13 March). The staff were also asked to estimate their confidence in their answer. Although the correlation between being right and the confidence level depends largely on the question, an improvement in the correlation can be seen over time, with the worst correlation observed on each day being: -0.44 on 1 March, -0.365 on 6 March, -0.16 on 8 March, -0.09 on 13 March.

d. Knowledge Prioritization

Finding 15 – Information prioritization did not address the needs of a majority of participants.

Observations

Many CTF HQ staff expressed a lack of prioritization of the information. Most staff received a large amount of email messages and information every day, with significant amount of information that was not considered relevant to them.

Similarly, CTF HQ staff expressed a lack of prioritization of the information presented during the Commander’s daily update brief. These briefs lasted from 35 to 95 minutes, approximately.

Finding 16 – The staff experience was a major factor that impacted on their ability to determine the relevance of the information.

Observations

The staff not having participated in MNE 3 or MNE 4 process rehearsals was more likely to wonder about the relevance of received information ( b = 0.506, b = 0.43).

COI 1.2 – To what extent was the KM Plan adhered to?

Overall Assessment: The lack of adherence to established business rules hindered effective sharing of information.

a. Business Rules

Finding 17 – Many infringements to the business rules were observed.

Observations

The CTF participants identified many business rules that were not followed (by their own admission) including: the IWS shared-view interruption, the storage of documents on the Portal rather than on IWS file cabinet, the setting of face-to-face meetings rather than virtual meetings, the file naming convention, the file storage, the email of files, and transmission of information by email (Figure D.4).


Non-adherence to the identified business rules was observed on many instances. Moreover, in addition to the non-followed business rules identified by participants, the inadequate use of the Portal’s calendar, which was clearly observed for many groups, has been an issue.

0123456789

10

F2F

Mee

tings

IWS

File

Cab

inet

sV

iew

sha

ring

"inte

rrupt

ions

"Li

mit

to 2

IWS

room

s

Not

e ta

king

Ear

ly s

how

up

for m

eetin

gV

oice

Pro

cedu

re

Rol

l Cal

l

Rev

iew

of

Age

nda

Pre

sent

atio

n:C

olor

sP

rese

ntat

ion:

Tim

e

File

Nam

ing

File

Sto

rage

Business Rules

Num

ber

of R

espo

nden

ts

Figure D.4 Number of participants who identified specific business rules that were not followed

Finding 18 – The lack of adherence to the business rules hindered the sharing of information.

Observations

The improper use of the Portal calendar can be linked with the difficulties some participants had in finding the time and location of meetings. Participants reported missing meetings, possibly resulting in lower SA.

The improper storage of documents can be associated with the difficulties some players experienced when looking for documents (the documents would simply not be available on the Portal and could only be found in an IWS file cabinet).

Many participants did not understand the exact meaning of the IWS shared-view interruption rule. The non-adherence to this rule could have been responsible for some network stability issues experienced during MNE4.

b. Adherence to KMLO Standard Operating Procedures (SOPs)

Finding 19 – A poor adherence to the SOPs was observed during MNE4.

Observations


The Portal’s calendar was not updated for the following day by 15:00 Zulu as prescribed by the SOPs.

The KM Liaison Officers (KMLO) did not develop a subject matter expert list as prescribed by the SOPs.

The KMLOs did not assist the KMOO in the development of a process flow plan to keep up with the Battle Rhythm; neither did the KMOO develop such a flow plan.

The KMLOs did not provide a daily brief to the KM chief (the daily brief was to contain feedback on the Knowledge Request process, on technical issue, an update on the EBO step process and a list of expected issues). Furthermore, the KMLOs’ interactions with the KM team decreased during the 2nd and 3rd week, with most of them missing most KM meetings.

No EBP step process checklist was produced by the KMLOs.

c. Adherence to the KM Roles and Responsibilities

Finding 20 – The KM team performed a limited number of their roles and responsibilities.

Observations

During the first few weeks of the experiment, various documents were not posted directly onto the Portal, in particular documents produced by the CG and EBE functional groups.

In many teams, the Portal’s calendar was not kept up-to-date. Some meetings occurred that were not on any calendar, and others that had been planned did not occur.

Monitoring of information flows was limited to feedback from the KMLOs. In particular, the KMOO stated that additional tools should have been provided to better support the monitoring of information flows.

No criteria to prioritize the KRs were documented.

The KMLOs were limited by the Experiment Manning Document to determine where to look for information. No subject matter expert list was developed.

No coordination of the knowledge production efforts within KBD was performed by KM, in contradiction to the KM Plan requirements.

COI 1.3 – How and to what degree was the KM support used?

Overall Assessment: The KM staff adapted their support to the needs of the CTF HQ staff but their support was hindered by limitation of the tools available.


a. KM Staff Roles

Finding 21 – The KM staff adapted their role to support the CTF HQ staff.

Observations

The KM organization for MNE 4 differed slightly from the one proposed by the KM Plan. In addition to the KM officers displayed in Figure D.1, a KM deputy was added.

The role of the KM training officer, assistant KR officer, and KM record management officer was modified at the end of the first week. These officers were assigned to support the EBA KM liaison officer (KMLO), the EBP KMLO and the EBE KMLO respectively. This decision to reassign these KM officers was made following request from the EBA and EBP KMLOs for additional support.

The KM staff work during MNE 4 focused on: 1. Developing process to deal with KRs and injects, 2. Managing the KR process, 3. Developing new business rules to regulate knowledge activities, 4. Modifying the Portal design; and, 5. Providing direct knowledge support to the functional groups through the KMLOs.

b. KMLOs Support to Functional Groups

The KM liaison officers (KMLOs) were delegated to support each functional team: Command Group, effects-based planning, effects-based execution, effects-based assessment, and the multi-national interagency group (MNIG). According to the KM Plan, the KM liaison officers duties include: setting up meetings (keeping calendar up-to-date and preparing agenda), ensuring the availability of staff for the meetings, encouraging the adherence to the business rules within their team, monitoring the battle rhythm and information flow (verifying the availability of inputs for team EBO steps), and facilitating virtual meetings within IWS.

Finding 22 – The KMLOs provided a necessary support to their functional group, particularly within MNIG, and were a useful link between the KM team and the EB groups.

Observations

A few times during the event, the KMLOs provided feedback to the KM team with regard to KM effectiveness. For instance, on 2 March, the major issues identified by some KMLOs were as follows:

1. EBP KMLO: 1. IWS stability, 2. Documents not being stored in Xythos, 3. Unfiltered injects.

2. MNIG KMLO: 1. Unfiltered injects, 2. Email overload from folder subscription, 3. IWS stability.


3. EBA KMLO: 1. Lack of data available for MOPs/MOEs evaluation, 2. Storage of information is not logical, 3. Lack of a good search engine.

The KMLOs not only supported the information management within their functional group but also supported the interpretation of the information as indicated by the MNIG analyst description of the MNIG KMLO activities. In particular, the MNIG KMLO activities included:

1. Setting up of meetings (including location, time and agenda); 2. Running the MNIG/UN/NGO meetings and MNIG internal meetings; 3. Chairing all meetings concerning medical issues; 4. Acting as point of contact for technical issues; 5. Supporting the search for documents and information; and 6. Acting as translator for military nomenclature, acronyms, and concepts.

The KMLO workload was likely impacted by the poor search engine available. In particular, the CG KMLO workload shows a strong correlation with the number of requests for documents observed on IWS text chat ( b = 0.611).

c. Knowledge Request Process

Finding 23 – The KR process was convoluted and the KR tool did not support many of the needs that the KRs were to fulfil.

Observations

Two functional groups were involved in dealing with KRs: the KM team and the Knowledge Base Development (KBD) team.

Although a KR process had been proposed by the KM Plan, the KR tool did not easily support the KR process.

The specific roles of the participants involved in processing the KRs was not clarified until 8 March when it was decided to unify the KM-KR cell and KBD-KR cell under a single KR cell. Although the decision was made that this cell would report to the KBD team, they reported to the KM team.

According to the KM Plan, the KRs were to encompass the functions of Commander’s Critical Information Requirements (CCIR), Priority Information Requirements (PIR) and Request for Information (RFI). However, since every KR needed a single request date, the tool was not suited to handle recurring needs.

Although it would have been suitable in some instances to split KRs, the tool did not offer this possibility.

The inability to split KRs combined with the serial process imposed longer delays on the KR process. The KRs could only be pushed to one person at a time and a long convoluted process was sometimes observed where the KR went from the originator to his group


KMLO, then to the KR manager then to the assistant KMO-KR (who acted as a postmaster), then to the System Of System Analysis (SOSA) lead, then back to the KMO-KR, then to the KS Intel Sup, then back to the KMO-KR, then to the grey cell, then back to the KMO-KR and finally back to the originator.

d. Injects

Finding 24 – An ad-hoc process emerged to deal with incoming information (as a result of injects).

Observations

The process to deal with injects was finalized on 7 March. The process was mainly determined by the KMLOs. For the EBA and MNIG team, the injects would be sent by email to the KMLOs who would then distribute them to all their team members. For the other teams, the injects would go through the KM Record Management Officer (KMRMO) who would post them on the Portal. A KBD staff member would also go through the injects to integrate them within the other databases (ONA, MindManager, Helix).

The selected process, in particular regarding the direct distribution to the MNIG, is at odds with the MNIG KMLO observation on 2 March that the unfiltered injects were an issue.

The selected process did not involve a proper verification and validation of the injects. Only the KMRMO provided a prioritization of the information contained in the inject for the EBE team.

Long delays between the receipt of the injects and their distribution by the KMLO were observed by the EBA analyst.

e. Business Rules

Finding 25 – New business rules were developed during the event, in general to address technical issues.

Observations

Following IT support recommendations, the KM team distributed new business rules during the experiment. These business rules included:

o A new file naming convention. Because the Portal could only display the file list based on name order, a new file naming convention was selected to put the date of the file at the beginning of its name rather than at the end. This new file naming convention created some issues as some KMLOs continued to follow the old convention, some followed the new one, and others did not follow any.


o IWS shared-view interruption rule. This business rules was developed to improved IWS stability. However, some teams did not understand the exact meaning of the rule and would ask to its entire member to regularly close down the shared view.

o Limit of two IWS rooms per participants. To further improve IWS stability, the participants were requested to monitor no more than two IWS rooms at a given time.

o Daily clean up of the IWS file cabinets. Some teams kept all draft documents in their IWS filing cabinet and only stored completed documents into the Portal. To facilitate the access of all staff to the documents, it was requested that at the end of each day the KMLOs move all documents from the IWS filing cabinets onto the Portal.

In addition to developing these rules, the core KM team reminded the KM liaison officers of various established business rules (in particular, regarding the use of the calendar), looked for information management issues and tried to find solutions to issues identified.

f. Portal Design

Finding 26 – The Portal was modified during the event to improve KM support for the CTF HQ staff.

Observations

During the second week a Notepad was introduced on the Portal to display links to important documents recently introduced to the Portal. The Notepad was visible from all pages of the Portal.

On 13 March, a link to the Xythos search engine was added to the Portal to provide a better search engine than the one initially provided even thought the KM team was made aware of this capability on 2 March.

COI 1.4 – What was the quality of the KM support to the knowledge activities?

Overall Assessment: KM support to the knowledge formalization activity was particularly poor. In many cases, amalgamated information was provided and the staff had a very limited knowledge of the assumptions made or the reliability (facts or opinion) of the received information.

a. Knowledge Creation

Finding 27 – KM failed to provide adequate support for the access to information.

Observations

To improve the ability of the CTF HQ staff to access information, the KM team added a link to the Xythos search engine on the Portal. However, this link was added only on 13 March while they had been aware of the issue with the Portal search engine and of the availability of Xythos search engine since the first week of the event.


The CTF HQ staff issued a total of 128 knowledge requests (KRs). Although these KRs were quickly acknowledged, it took many hours (in some cases days) before an answer was provided (see Figure D.5). A total of 26 KRs were still outstanding at the end of the experiment.

The KRs were issued almost uniformly throughout the 3 weeks with an average of 9 KRs per day. (The Haussdorf dimension associated with the distribution of the KRs was 0.92; a uniform distribution would have a Haussdorf dimension of 1 while an amalgamation of the KRs at a few periods of time would have a Haussdorf dimension of 0).

The adequacy and timeliness of knowledge request products was assessed as low by a majority of participants (see Figure D.6). Furthermore, a majority of participants were unhappy with answers provided.

The KR process was not streamlined and involved many participants. The established process was for the KR to go from the originator to the KM liaison officer, then to the KR manager, who would send it to the assistant KMO KR, who would send it to someone who could provide an answer. The identified information providers were SOSA, Intel, MNIG, or Control (CNTL). When answered, the KMO KR would send the answer directly to the originator. During the event, the established process was not always followed. This would usually cause further delays in processing the KR.

A large variation among the response times of the people involved in processing the KRs was observed. Some staff responded generally within 26 minutes while others took consistently more than 5 hours (see Figure D.7). The KS Intel Sup was particularly slow in responding to KRs. Furthermore, since the Intel player indicated that he had a low workload during the MNE 4 event, the long delay does not seem to be due to time constraints. Based on the Intel analyst, the reason for the delay was because the Intel player was not aware of his involvement in the KR process.

Some KRs were withdrawn before being answered. There appears to be two causes for this: 1) Long time delays before an answer was received, and 2) The originator did not follow the KR procedure and the request was returned by the KMO-KR.


-10

10

30

50

70

90

110

130

-20 0 20 40 60 80 100 120

Elapse Time (Experiment Hours)

Num

ber o

f KR

s

AcknowledgedStandingManagedKMSOSAINTELMNIGCONTROLDroppedClosedTotal

Figure D.5 Time evolution of the number of KRs that were issued, acknowledged, closed, managed, dropped, and answered by specific groups (KM, SOSA, Intel, MNIG, Control).

Figure D.6 displays an assessment provided by the participants regarding the quality of the KR answers and the timeliness of the answer. The scale goes from 1 (completely dissatisfied) to 7 (completely satisfied). The red boxes and lines display the results obtained from the 16 March survey; the blue ones display the results from the 9 March survey. The line inside a box shows the median from the group and the lower and upper edge of the box displays the 25th and 75th percentile respectively. Note that only the players having initiated some KRs were requested to answer the survey. The results show that there was in most groups a wide distribution of satisfaction levels regarding the KR with a majority of participants dissatisfied (4 corresponds to a neutral feeling).


Command Grou

EBA Group

EBE Group

EBP Group

Green TeamKMLO

KS-INTKS-KM

KS-SOSAMNIG

Red Team

CATEGORY

1

2

3

4

5

6

7

VALU

E

Command Grou

EBA Group

EBE Group

EBP Group

Green TeamKMLO

KS-INTKS-KM

KS-SOSAMNIG

Red Team

CATEGORY

1

2

3

4

5

6

7

VAL

UE

9-316-3

DAY

KR Response Satisfaction KR Response Timeliness

Figure D.6 Quality of KR answers and timeliness of answers as perceived by participants

0 5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Elapsed Time (Experiment hours)

Cum

ulat

ive

Pro

babi

lity

Originator

SoSA

MNIG

KRM

KMLO

KBD Coord

Intel

Control

LEGEND

Figure D.7 Response time for the participants involved in processing the KRs

The cumulative probability for the response times of the participants involved in processing KRs is displayed in Figure D.7. The response time corresponds to the delay between the time a KR was pushed to an individual and him pushing it back to someone else. The time shown is the experimental elapsed time. Thus 1200 minutes (20 hours) corresponds to 2.5 days of experimental time. From the graph, it is obvious that the KBD Coord, which acted as a postmaster, processed KRs most often within 200 minutes. Obviously, the participants (Intel, SOSA, MNIG, and CNTL) that had to find the answer to the KRs needed more time to


process them. However, the Intel player did not answer any KRs and his response was limited to proposing a different source of information, indicative of a weak participation to the KR process. Note that only cumulative probabilities for response times less than 40 hours are shown on the graph. In one instance, Control took up to 58 hours to respond to a KR. The SoSA group (in purple) responded to KRs in a relatively timely manner, as compared to Intel and MNIG.

For the MNIG, there was some frustration noted by their members since questions were tasked to them but they did not have usual information sources at hand, or through open source internet.

It should be noted that SoSA staff as also provided direct knowledge-creation support to teams within the CIE. However, the quality of this support was judge insufficient and the SoSA Team lead reported that SoSA support was mostly reactive, based primarily on short-term intelligence needs. It was not qualified as anticipatory systemic analysis as implied by the concept for continuous systemic analysis. This outcome may have been due to resource constraints, but nonetheless a recognition and desire to look beyond the planning horizon of the HQ staff would be required. A well-defined series of questions or avenues of inquiry to guide efforts should guide this activity.

Finding 28 – KM did not support, except in a few instances, an efficient decision-making process. The lack of stated principles to support an efficient decision-making process undermined the KM support.

Observations

One major issue regarding the deficiency of the decision-making process appears to be the lack of consideration given to some staff members’ contributions. Analysts identified the lack of knowledge of the authorities of the staff as an explanation for this issue.

The MNIG did not contribute to effects development (lack in proceeding through the ED2 step).

b. Knowledge Interpretation

Finding 29 – Even though KMLOs provided some support for the interpretation of information, additional resources would have been useful.

Observations

No glossary or list of acronyms was distributed to the CTF HQ staff. These would have useful, particularly to support the interaction between the military and the non-military staffs. As shown by the human factor survey, the MNIG was somehow dissatisfied with its ability to interpret and understand the information.


Too often, the source of the information was not specified, as acknowledged by the CTF HQ staff regarding answers to KRs. Knowing the source would have helped staff to interpret the information adequately.

c. Knowledge Formalization

Finding 30 – The KM Plan does not address the support of knowledge formalization.

Observations

The KM Plan does not specify any process to support the knowledge formalization activities. However, it is expected that a more consistent specification of the source of the information and/or the assumptions made in reaching given conclusions would have helped support the formalization activities.

Finding 31 – There is the lack of a clear process in the EBAO CONOPS to deal with inconsistencies.

Observations

Although the KI2 step in the CONOPS refers to the identification of inconsistencies in the description of the KI2 step (Merging of KB Inputs), a clear process to support this identification is lacking in the description of the KI2 procedures. These procedures were not followed.

d. Knowledge Prioritization

Finding 32 – KM provided limited support to knowledge prioritization. Various KM processes proposed by the KM Plan to support knowledge prioritization were not performed.

Observations

Two main tools were used to prioritize the information: the Portal Notepad was used to display hyperlinks to important documents that were of significant and broad interest, and the participants could subscribe to specific folders18 based on own priorities to be acknowledged when the content of the folder was modified.

Although the Portal Notepad appeared useful in prioritizing the information, many staff members still felt overloaded with email through the simple subscription function.

No criteria were documented for the prioritization of KRs.

KM and KBD teams did not work together. In particular, KM did not help KBD in prioritizing the information production effort.

18 IT tools available on the portal offered the user to register for being informed when the folder was updated. The user would click on a hyperlink beside the folder and a window would be displayed where different options for the type of subscription were offered to the user.


e. KM Satisfaction Level

Finding 33 – There was a wide distribution of the satisfaction within the KM team in supporting the knowledge activities, but all agree that better tools would have been useful.

Observations

At the end of the experimentation, the KM team was asked to evaluate their own capability to support knowledge activities and to identify possible resources that could have helped them to better perform. A large distribution of answers was received from the various team members, ranging from ‘Very Satisfied’ to ‘Very Dissatisfied’.

One staff member, who was very dissatisfied with his own performance, indicated that one of his tasks was to monitor the information flow within the CTF HQ. Unfortunately, he felt that he could not perform this task to his desired level due to the lack of supporting tools.

The KM team identified resources that would have helped them to perform better: an improved Portal capable of handling hyperlinks and metadata, an improved search engine, an improved KR process, a list of subject matter experts, and some tools to monitor process flow (battle rhythm).

RECOMMENDATIONS

Knowledge Management failed to appropriately support the knowledge activities during MNE 4. Many recommendations to improve KM can be made based on the above analysis and observations by KM staff:

1. KM is more than information management and the KM concept should be refined considering the need to support: knowledge creation, knowledge interpretation, knowledge formalization, and knowledge prioritization.

2. Sound KM principles to support KM decision-making processes need to be defined. In particular, the five principles introduced by Pavitt19 should be considered:

a. Groups outperform individuals for brainstorming and recalling processes.

b. Based on transactive memory theory, efficient communication allows groups to know all relevant information known by their members.

c. Groups do not perform as well as rational models predict. In particular, actual group performance is expected to lie between the Condorcet group20 and ideal rational group performance.

19 C. Pavitt. Why we have to be reductionists about group memory. Human Communication Research 29, pp. 592-599, 2003. 20 A Condorcet group is a group where each member of the group makes choices independently and the choice with simple majority is selected as the group’s decision.


d. The degree of underperformance, compared to the ideal rational group, increases with group size.

e. Unfacilitated group interaction is not an especially efficient method for pooling members’ knowledge. In particular, researchers have observed that knowledge-sharing processes can break down under certain conditions and have identified ineffective group interaction leading to such breakdowns.

3. Similarly, a clear knowledge formalization process should be developed and incorporated within the knowledge integration step.

4. Considering the large amount of information needed, improved information access tools such as multi-source search engine and targeted information distribution are required to support EBAO.

5. The introduction of tools for the KM team to monitor information flow, collaboration and knowledge activities should be explored. For example, Social Network Analysis should be considered to identify potential issues with regard to staff collaboration and to take adequate measures to resolve these issues.

6. The impact of various factors that impact on knowledge activities should be studied further, factors including past experience, language, organizational culture, the understanding of the commander’s intent, and group size.

CONCLUSION

In summary, the EBAO process during MNE 4 was not supported by a robust knowledge management system. Furthermore, due to the lack of adherence to the KM Plan, the experimental hypothesis cannot be tested. However, within the introduced concept of KM, it is apparent that the KM Plan needs further refinement to support EBAO. In particular, the KM Plan should give more than a definition of KM and a proposed KM organization with its roles and responsibilities. The KM Plan should provide sound and timeless KM principles on which implementing procedures can be based.

Major issues observed during MNE 4 included the lack of support for the formalization of information products and the lack of knowledge of who is responsible for what. Participants identified this last factor as a major hindrance to information sharing and also one cause for the ineffective KR process. From an analytical point of view, the impact of factors such as past experience, workload, and degree of involvement within the EBAO process, as well as the competition between vertical (tactical, operational, and strategic levels of command) and horizontal (EBP, EBE, EBA, KBD, MNIG as well as extending outside the EBAO organization) shared of information, are worth noting and exploring.


Appendix 1 Correlations Among Factors Impacting on Knowledge Activities

This appendix provides the list of correlations that were calculated for the KM analysis. Some factors were evaluated through surveys sent to the CTF HQ staff and some through a social network analysis performed by UK analysts. The survey question and the date when the data were captured are indicated.

List of factors considered:

Quality of the information

1. Perceived accuracy of the information (Survey HF3, 9 March, question 1: In EBO, the information I received was accurate.)

2. Perceived appropriateness of the information (Survey HF3, 9 March, question 2: In EBO, the information I received was appropriate for the purpose at hand.)

3. Perceived ability to access the information (Survey HF3, 9 March, question 3: In EBO, I was able to access information from my communications system.)

4. Perceived relevance of the information (Survey HF3, 9 March, question 4: In EBO, I was able to access information from my communications system.)

5. Perceived relevance of the information (Survey HF3, 9 March, question 5: In EBO, the information I received was relevant for the purpose at hand.)

6. Perceived timeliness of the information (Survey HF3, 9 March, question 6: In EBO, the information I received was timely for the mission.)

7. Perceived completeness of the information (Survey HF3, 9 March, question 7: In EBO, the information I received was complete for my task.)

8. Perceived sufficiency of the information (Survey HF3, 9 March, question 8: In EBO, the information I received was sufficient for me to complete my task.)

9. Perceived conciseness of the information (Survey HF3, 9 March, question 9: In EBO, the information I received was concise in the format in which it was presented.)

10. Perceived interpretability of the information (Survey HF3, 9 March, question 10: In EBO, I was able to interpret the information presented to me for this task.)

11. Perceived ability to understand the information (Survey HF3, 9 March, question 11: In EBO, I was able to understand the information presented to me in this mission.)

12. Perceived adequacy of amount of information (Survey CUL2, 16 March, question 4.5: Do you feel that you received too much or too little information in order to make decisions?)


13. Perceived quality of the received information (Survey CUL2, 16 March, question 4.6: In general, how content were you with the information you received?)

Information Sharing

14. Perceived adequacy of information sharing (Survey CUL2, 16 March, question 4.1: Most of the time, were you provided with the information you needed or did you have to seek it out yourself?)

15. Amount of information shared with superior (Survey CUL2, 16 March, question 4.4a: Rate how much you shared information with your superior.)

16. Amount of information shared with peers (Survey CUL2, 16 March, question 4.4b: Rate how much you shared information with peers.)

17. Amount of information shared with subordinates (Survey CUL2, 16 March, question 4.4c: Rate how much you shared information with subordinates.)

18. Organizational stovepipe (Survey CUL2, 16 March, question 6.1b: Where in the organization did the person you work with most work (same group or different group)?)

Quantitative evaluation of the individual involvement in sharing information within the CTF HQ as measured by the UK Social Network Analysis based on the third week emails and text chat.

19. Degree of the node associated with the individual

20. Betweeness of the node associated with the individual

21. Closeness of the node associated with the individual

22. Harmonic closeness of the node associated with the individual

23. Eigenvector of the node associated with the individual

24. 2-Local eigenvector of the node associated with the individual

25. Alpha-attenuation of the node associated with the individual

26. Density of the node associated with the individual

27. Effective size of the node associated with the individual

28. Constraint of the node associated with the individual

29. Hierarchy of the node associated with the individual


Language

30. Native English speaker (Survey CUL2, 16 March, question 5.1: Are you a native English speaker?)

31. Impact on persuasive ability (Survey CUL2, 16 March, question 5.3: Do you ever feel that it is harder to persuade colleagues of your ideas in English than in your native language?)

32. English speaker dominance (Survey CUL2, 16 March, question 5.7b: Regardless of rank and number of persons, do you believe that native English speakers tend to dominate cooperative situations more than others?)

33. Perceived language difficulties (Survey CUL2, 16 March, question 5.7a: Have you ever believed that it is more difficult to understand non-native English speakers' point of view than native English speakers?)

34. Perceived language difficulty (Survey CUL2, 16 March, question 5.7c: How often do you experience your colleagues having problems understanding English?)

Culture

35. Culture differences (Survey CUL2, 16 March, question 7.2: Were there any differences in how you related to people from different cultures?)

36. National stovepipes (Survey CUL2, 16 March, question 7.1: Who (which nationality) did you work with the most in this exercise?)

Duties and Workload

37. Perceived difference between MNE 4 duties and usual duties (Survey CUL2, 16 March, question 1.1: In this experiment, did you feel that your tasks and responsibilities were different from what you are used to (I.e., compared to your daily job)?)

38. Impact of difference between usual duties and MNE 4 duties (Survey CUL2, 16 March, question 1.3a: Did the change in tasks and responsibilities affect how you cooperated with your colleagues?)

39. Perceived amount of workload (Survey HF1, average over 12 days, question 8: How hard you worked (mental & physical)?)

40. Perceived adequacy of workload (Survey CUL2, 16 March, question 1.5: Do you feel that you had too much, too little, or the right amount of work to do in your position in this exercise?)


Previous Experiences

41. Years of military experiences (Survey Dem1, Week 0, question 6 & 7: Do you have military experience?, How many years of military experience do you have?)

42. Effects-based planning experience (Survey Dem1, Week 0, question 9: Do you have experience working with Effect Based Planning? If yes describe.)

43. Collaborative information environment experience (Survey Dem1, Week 0, question 10: Have you ever worked in a distributed collaborative environment? If yes, describe.)

44. Joint task force headquarters experience (Survey Dem1, Week 0, question 11: Have you ever worked in a Joint/Joint Task Force HQ staff?)

45. Coalition headquarters experience (Survey Dem1, Week 0, question 12: Have you ever worked in a Coalition HQ staff?)

46. MNE3 participation (Survey Dem1, Week 0, question 16: Indicate whether or not you participated in the Multinational Experiment 3 event.)

47. KBD Limited Objective Experiment participation (Survey Dem1, Week 0, question 20: Indicate whether or not you participated in the KBD LOE.)

48. MNE 4 process rehearsal participation (Survey Dem1, Week 0, question 24: Indicate whether or not you participated in the MNE 4 Process Rehearsal (November 2005)?)

49. MNE 4 Rock Drill participation (Survey Dem1, Week 0, question 25: Indicate whether or not you participated in the MNE 4 Rock Drill event (January 2006)?)

50. Participation in Effects based plan 1 development (Survey Dem1, Week 0, question 22: Indicate whether or not you participated in the CTF Effects Based Plan 1 Build Conference (September 2005)?)

Training

51. Training adequacy for CIE tools (Survey KM2, 14 March, question 6: Did you receive adequate training to use the CIE tools?)

Decision Making

52. Quality of the decisions made (Survey CUL2, March 16, question 3.4: In general, how would you rate the quality of decision making by your team?)

53. Success of the decisions made (Survey CUL2, March 16, question 3.5: In general, in terms of outcome, how successful were the decisions made by your team?)


Situational Awareness

54. Average individual’s Situational Awareness score based on survey3 KBD1, KB2, and KBD3.

Commander’s Intent

55. Average individual’s Commander Intent score based on surveys HQORG1 to HQORG10.

The correlation among all the factors listed above was calculated. The table is not displayed here due to its size, but an electronic copy could be provided upon request. More precisely, the Kendall τb correlations were calculated since they provide a non-parametric evaluation of the correlations and they possess a simple and intuitive interpretation. Briefly, τb is calculated by making pairwise comparison. For example, to calculate the correlation between an individual’s years of military experience and his perceived ability to interpret the information, a pairwise comparison is made considering all pairs of individuals. For each pairwise comparison, a check is made to verify if more years of military lead to a higher perceived ability to interpret the information (concordant comparison) or to a lower ability to interpret the information (discordant comparison). τb is finally computed by looking at the following ratio between the number of concordant comparisons (Nc) and the number of discordant comparisons (Nd):

dc

dcb NN

NN+−

=τ

If τb = 1/3, then an individual with more years of military experiences is twice more likely to indicate a higher ability to interpret the information. Similarly, if τb = 0.5, an individual with more years of military experience is three times more likely to indicate a higher ability to interpret the information. The most significant correlations obtained in the KM analysis are mentioned in the main body of this annex.


ANNEX E: Culture Analysis

Introduction

Military operations are becoming increasingly joint, interagency, and multinational in character (Sutton & Pierce, 2003). As a result, military teams and headquarters staff must incorporate cultural differences into their information-sharing and decision-making processes. Unfortunately, cultural differences can pose significant barriers to successful team performance (Klein, Pongonis, & Klein, 2000). For example, differences in a group’s social and political surroundings, values, beliefs and cognitions, can affect the manner in which information is shared and interpreted (Sutton & Pierce, 2003). Such differences among team members could ultimately lead to sub-optimal team performance.

In addition to the impact of potential cultural differences between team members, whether teams are collocated or distributed can impact team performance (Grosse, 2002). Being collocated or distributed might affect the nature of cooperation, information sharing, and decision-making. Differences, real or perceived, could ultimately affect overall team performance. Unfortunately team performance was not measured during MNE 4.

COI Experiment Aim and Objective

The aim and objective of this portion of MNE 4 was to determine whether cultural differences existed among the participants of MNE 4 in terms of information processing abilities and whether working in a collocated vs. distributed headquarters affected perceptions of team processes. Any difference observed in these measures would then need to be considered when examining the hypotheses put forth in MNE 4.

Experiment Design with respect to the COI

Prior to the start of MNE 4, the participants from the Effects Based Planning, Effects Based Execution, Effects Based Assessment, Command, and Multinational Interagency Groups from both the Coalition Task Force (CTF) and NATO Response Force NRF Headquarters (HQ), were provided with five questionnaires designed to assess how individuals contend with ambiguous information and the risk of making errors when making decisions. The pre-experiment questionnaires used were The Need for Cognitive Structure and The Ability to Achieve Cognitive Structure Scales (NCS & ACCS; Bar-Tal, 1994), The Uncertainty Response Scale (URS; Greco & Roger, 2001), The Personal Need for Structure and Personal Fear of Invalidity Scales (PNS & PFI; Thompson, Naccarato, Parker, & Moskowitz, 2001). At the conclusion of the experimental part of MNE 4, these participants were required to answer the rating portions of an organizational survey designed to assess potential perceived differences in issues related to group roles and responsibilities, organization, decision-making, information-sharing, language, identity, and culture, in the respective HQs.

Survey Participants


A total of 130 people from Canada (22), Denmark (3), Germany (31), Finland (7), France (4), Sweden (3), United Kingdom (10), United States (38), and Turkey (12) provided responses to the five pre-experiment surveys. A total of 182 persons, 72 from the NRF site and 111 from the CTF site provided responses to the Organizational survey at the conclusion of MNE 4 (a breakdown of participants by nation was not possible because some of these data were missing).

Analysis Methodology

COI Findings

COI 1 Contending with Ambiguous Information

The mean response and standard deviation for each country is shown in Table C1. Separate 1-way analyses of variance (ANOVAs) were conducted on the mean ratings for each of the pre-experiment surveys. Interpretation of these findings must be made cautiously due to the unequal number of respondents across countries and the low response rates from some of the countries.

TABLE E.1 MEAN RATINGS (MR) AND STANDARD DEVIATIONS (STD) FOR PRE-EXPERIMENT SURVEYS

Country NCS

MR / Std

ACCS

MR / Std

URS

MR / Std

PNS

MR / Std

PFI

MR / Std

Canada 3.66 / .85 3.66 / .55 4.46 / .48 4.16 / .56 3.85 / .57

Denmark 3.66 / .57 3.4 / .43 4.17 / .28 4.18 / .51 3.49 / .48

Germany 3.68 / .64 3.36 / .18 4.14 / .40 3.98 / .29 3.31 / .48

Finland 3.81 / .78 3.51 / .59 4.16 / .16 4.13 / .55 3.64 / .43

France 2.88 / .03 3.89 / .77 4.06 / .48 3.23 / .09 3.74 / .53

Sweden 3.38 / .72 3.61 / .36 4.31 / .39 3.95 / .42 3.89 / .49

UK 3.78 / .64 3.61 / .43 4.43 / .37 4.12 / .36 3.84 / .52

US 3.08 / .92 3.47 / .31 4.13 / .51 3.33 / .83 3.47 / .91

Turkey 4.13 / 1.01 3.51 / .82 4.58 / .44 4.44 / .76 3.93 / .76

COI 1 Overall Assessment Results


The analyses revealed a number of significant differences across some of the countries with respect to the URS and PNS rating scales. However, to conclude that these observed differences are in fact real differences may be premature because of the unequal response rates across the countries and the small number of responses provided by some of the countries. Moreover, close inspection of the data show that in general, each country provided ratings for each survey that demonstrate moderate levels of each variable. At best, we can conclude that there are trends in the data that show that the countries might differ in a number of cognitive aspects (e.g., URS and PNS), which could in turn affect team information sharing and decision-making.

Finding 1. Analysis of the Need for Cognitive Structure (NCS) survey revealed that statistically, the participants from the respective countries did not differ in their ratings of NCS (see Figure E.1), F(8,127) = 1.64, MSe = .85, p = .12.

Finding 2. Similar to the NCS ratings, the participants did not differ in their mean Ability to Achieve Cognitive Structure (ACCS) ratings (see Figure E.2), F(8,129) = .87, MSe = .22, p = .54.

Finding 3. Examination of the Uncertainty Response Scale (URS) revealed there to be a significant effect of country on the responses (see Figure E.3), F(8,126) = 2.35, MSe = .35, p < .02. Post-hoc analyses of the data using the Least Significant Difference (LSD) with p < .05, revealed there to be significant differences in mean responses between CA and DA, DA and UK, and between TU and DA, DE, and FR.

Finding 4. Analysis of the Personal Fear of Invalidity (PFI) data revealed there to be no statistically significant differences in mean response between the participants from the respective countries (see Figure E.4), F(8,128) = 1.9, MSe = .57, p = .07.

Finding 5. The assessment of the Personal Need for Structure (PNS) data showed a significant effect of country (see Figure E.5), F(8,126) = 3.02, MSe = .76, p < .005. Post-hoc analyses using the LSD measure with p < .05, showed France to have significantly lower mean rating than all other countries except the US. Moreover, the US was observed to differ from CA, DA, FI, UK, and TU on this measure.

COI 2 Characteristics associated with Collocated and Distributed Teams

The analysis of the Organizational Survey data was conducted with a focus on comparing responses between participants from the NRF and the CTF. This was done to determine whether characteristics associated with collocated and distributed teams would impact the participants’ views on the daily workings of their headquarters environment. Potential differences between the two headquarters would most certainly have to be taken into account when explaining possible differences in performance measures between the two sites.

The analysis of these data was undertaken with the chi squared ( 2) methodology. This statistical measure indicates whether differences in frequency of response exist for a given category. In order to determine whether the frequency of responses differed across the two


HQs, the Mann-Whitney U statistic was used. For these analyses, the Mann-Whitney U statistic is only reported in cases where differences in response frequency between the sites were observed. Although it was the frequency of the ratings that were subjected to the statistical analyses, it is the percentage of ratings from the total number of respondents in the respective headquarters that is reported in the Tables.

a. COI 2 Overall Assessment Results

Overall, the data from the organizational survey indicated that there were not many differences between members of the NRF and CTF in terms of various aspects of team processes. However, differences were observed on a number of issues: members of the CTF perceived their HQ as more hierarchical and less flexible than members of the NRF. Also, members of the CTF rated the quality of their HQ as poorer than members of the NRF. Although the data indicated that the leaders from the CTF were more likely to rely on team members for decision-making than in the NRF, members from the CTF rated the speed of decision-making in the CTF as slower in comparison with the ratings provided by the members of the NRF. In terms of information sharing, compared with the members of the NRF, the members of the CTF indicated that they did not receive enough information to make decisions and were less satisfied with the information that they did receive. The participants from the CTF provided ratings that indicated that they were less concerned about potential problems associated with communicating in English environments and relating to people from other cultures than indicated by members of the NRF. Finally, the rating data revealed that members of the CTF had more experience with exerting downward control in teams than members of the NRF, suggesting that members of the CTF might be more authoritarian than members of the NRF.

Finding 1. Group Roles and Processes

Q1.1. In this exercise, do you feel that your tasks and responsibilities are different from what you are used to (i.e., compared to your daily job)?

Yes, very different Yes, a bit different No, no difference 2

NRF 45% 36% 19% 2 = 8.58, p < .01

CTF 40% 38% 22% 2 = 7.19, p = .03

Not surprisingly, the analysis of Q1.1 revealed that participants from both the NRF and CTF viewed their tasks and responsibilities in MNE 4 as different from their daily tasks and responsibilities.

Q1.2a. Does this change in tasks and responsibilities mean that you have more, less, or the same amounts of responsibilities as you are used to (i.e., compared to your daily job)?

A lot more

Somewhat more

Same Somewhat less

A lot less 2

NRF 5% 17% 36% 25% 17% 2 = 15.13, p < .004


CTF 8% 23% 30% 23% 16% 2 = 11.91, p = .02

Q1.2b. Has the change in the amount of responsibilities been to the better or the worse for you?

Better Somewhat better Same Somewhat worse Worse 2

NRF 10% 20% 44% 17% 9% 2 = 24.14, p < .001

CTF 9% 13% 45% 22% 24% 2 = 37.54, p = .001

Analysis of the data from Q1.2a and Q1.2b showed that the majority of members from both the NRF and CTF rated their task and responsibilities as less than that of their daily jobs (Q1.2a) and that this was of no real consequence to each group (Q1.2b).

Q1.3a. Does the change in tasks and responsibilities affect how you cooperate with your colleagues?

Yes, very much

Yes, somewhat

Difficult to say

Not much

No, not at all

2

NRF 2% 32% 22% 32% 12% 2 = 20.75, p < .001

CTF 7% 20% 21% 25% 27% 2 = 11.22, p = .02

Q1.3b. Is this for the better, worse, or the same?

Better Somewhat better Same Somewhat worse Worse 2

NRF 10% 20% 56% 10% 4% 2 = 51.93, p < .001

CTF 5% 19% 55% 16% 5% 2 = 75.13, p = .001

The data from Q1.3a and Q1.3b indicated that for the majority of the participants from both the NRF and CTF, the changes in tasks and responsibilities experienced in MNE 4 did not affect how they collaborated with colleagues.

Q1.4. Compared to what you are used to (i.e., your daily job), how much do you feel like you are integrated into the central processes at this headquarter?

Very integrated

A bit integrated

No difference

A bit less integrated

Not at all integrated

2

NRF 12% 35% 21% 21% 11% 2 = 12.72, p < .01


CTF 23% 19% 17% 23% 18% 2 = 1.65, p = .79

The data from Q1.4 showed that while the majority of participants from the NRF felt a bit more integrated into the central processes of their HQ, there was a more equal distribution of ratings about HQ integration among the members of the CTF. Interestingly, this difference in responses was not statistically significant between sites.

Q1.5. Do you feel like you have too much, too little or the right amount of things to do in your position?

Too much A bit too much About right A bit too little Too little 2

NRF 3% 13% 54% 22% 8% 2 = 58.97, p < .001

CTF 6% 17% 37% 18% 22% 2 = 27.15, p < .001

Although the majority of NRF participants indicated that they had just about the right amount of work to do, a sizeable proportion of members from both the NRF and CTF did indicate that they had a bit too little to do in their HQs.

Finding 2. Organization

Q2.1. In your job here, would you describe the organization around you as different from what you are used to?

Yes, very different Yes, a bit different No, no difference 2

NRF 65% 28% 7% 2 = 37.5, p < .001

CTF 62% 28% 10% 2 = 46.11, p < .001

Similar to Q1.1, the data from Q2.1 revealed that the members from both the NRF and CTF viewed the MNE 4 organization as different from what they are used to.

Q2.2. How would you describe the level of hierarchy here? U = 2619, p < .001

Much flatter

A bit flatter

No difference

A bit more hierarchical

Much more hierarchical

2

NRF 19% 47% 18% 10% 6% 2 = 38.14, p < .001

CTF 8% 27% 30% 24% 11% 2 = 21.56, p < .001

Analysis of Q2.2 revealed a significant difference in ratings between the NRF and CTF sites, Mann-Whitney U = 2619, p < .001. The participants from the NRF site viewed their level of hierarchy as more flat than was indicated by the participants from the CTF.


Q2.3. How is the level of centralization/decentralization (self-organization) here?

Much more decentralized

A bit more decentralized

No difference

A bit more centralized

Much more centralized

2

NRF 10% 40% 22% 22% 6% 2 = 26.42, p < .001

CTF 16% 27% 32% 14% 11% 2 = 18.68, p < .001

The majority of members from both the NRF and CTF characterized the level of self-organization as more decentralized than centralized.

Q2.4. How is the flexibility (between centralization/decentralization) here? Mann-Whitney U = 2651, p < .001

Much more flexible

A bit more flexible

No difference

A bit less flexible

Much less flexible

2

NRF 15% 39% 28% 14% 4% 2 = 26.19, p < .001

CTF 6% 24% 27% 28% 15% 2 = 18.51, p < .001

A significant difference between the two sites (Mann-Whitney U = 2651, p < .001), revealed that the participants from the NRF staff perceived their headquarters as more flexible than members from the CTF staff. This is opposite to what the hypothesis states.

Q2.5. How would you rate this organization compared to what you are used to? Mann-Whitney U = 3225, p < .02

Much better

Somewhat better

No difference

Somewhat poorer

Much poorer

2

NRF 7% 32% 28% 26% 7% 2 = 21.05, p < .001

CTF 3% 19% 33% 35% 10% 2 = 44.9, p < .001

Analysis of Q2.5 revealed that a larger proportion of members from the NRF perceived their organization as better than what they are used to in comparison with member from the CTF where a large proportion of the CTF members felt that their MNE 4 organization was poorer than their daily headquarters (Mann-Whitney U = 3225, p < .02).

Q2.6. Have you experienced that a superior makes decisions that is within your area of responsibility?

Very Often Often Sometimes Rarely Never 2

NRF 4% 15% 41% 25% 15% 2 = 26.23, p < .001

CTF 6% 13% 32% 32% 17% 2 = 27.96, p < .001


The data from Q2.6 shows that members from both HQ sites typically have not experienced superiors making decisions within the members’ areas of responsibility.

Finding 3. Decision-Making

Q3.1. Who makes most of the decisions in your environment in this exercise?

Superior You Subordinate 2

NRF 65% 27% 8% 2 = 36.58, p < .001

CTF 64% 32% 4% 2 = 61.33, p < .001

The majority of members from both the NRF and CTF indicated that a “Superior” made most of the decisions in the exercise.

Q3.2a. Most of the time does a team or one person make decisions in your environment?

Leaderless Team Team with leader Leader/one person 2

NRF 22% 63% 15% 2 = 28.08, p < .001

CTF 14% 71% 15% 2 = 71.56, p < .001

Most of the participants from the both the NRF and CTF HQs indicated that most of the decisions were made within the team context (i.e., team with leader).

Q3.2b. To what degree is the decision of the leader contingent on the work and advice of others? (this question was for those who did not answer Leaderless Team in Q3.2a) Mann-Whitney U = 2212, p < .07

Not at all

Little Somewhat Much Completely Not applicable 2

NRF 0% 5% 44% 36% 11% 4% 2 = 38.18, p < .001

CTF 1% 8% 19% 54% 15% 3% 2 = 110.13, p < .001

Although the majority of members from both the NRF and CTF indicated that, in general, the decision of the leader was contingent on the work and advice of others, there was a trend among members of the CTF that the decision of the leader was more contingent upon others than was perceived to be the case with members from the NRF (Mann-Whitney U = 2212, p < .07).

Q3.3. In your opinion are decisions made too slow, too fast, or just right in your environment? Mann-Whitney U = 2935.5, p < .001


Too slow

A bit slow

Just right

A bit fast

Too fast

Not applicable

2

NRF 1% 6% 71% 10% 6% 6% 2 = 153.67, p < .001

CTF 12% 23% 51% 9% 2% 3% 2 = 109.05, p < .001

Although a large proportion of members from each HQ viewed decision times as just right, further analysis showed that a larger proportion of participants from the CTF felt that the decisions were being made more slowly than was the situation with members of the NRF (Mann-Whitney U = 2935.5, p < .001).

Q3.4. In general, how would you rate decision quality in your environment?

Very good Good Neutral Poor Very poor Not applicable 2

NRF 16% 61% 13% 7% 0% 3% 80.08, p < .001

CTF 17% 49% 28% 4% 0% 2% 85.71, p < .001

Q3.5. How successful have decisions been in general in your environment?

Very successful

Partly successful

Neutral Somewhat unsuccessful

Not at all successful

Not applicable

2

NRF 33% 47% 14% 3% 0% 3% 55.77, p <.001

CTF 30% 45% 16% 5% 0% 4% 69.73, p < 001

Examination of the data from both the NRF and CTF revealed that the majority of members at both sites rated their decisions as being of good quality (Q3.4) and successful (Q3.5).

Finding 4. Information-Sharing

Q4.1. Most of the time, are you provided with the information you need or do you seek it out yourself?

Provided by others Seek it out myself Both, equally much 2

NRF 13% 49% 38% 15.08, p < .001

CTF 12% 49% 39% 25.29, p < .001


Q4.2. Most of the time, what do you do with the information you get?

Keep until it is requested

Distribute to a few

Distribute to many

Post it on the web (or the like)

2

NRF 1% 51% 31% 17% 39.3, p < .001

CTF 5% 41% 36% 18% 37.86, p < .001

Analysis of Q4.1 and Q4.2 revealed that most of the time, the participants from both the NRF and CTF sought information on their own and then distributed it to just a few persons.

Q4.3. How is the general policy on distributing information in the organization under this exercise?

As usual Somewhat different Very different Difficult to say 2

NRF 18% 56% 19% 7% 38.55, p < .001

CTF 15% 42% 35% 8% 33.39, p < .001

In general, a majority of the participants from both the NRF and CTF perceived the policy of distributing information during the MNE 4 exercise as different than what they are used to.

Q4.4. For each of the following categories below, please rate how much you:

Share information with:

Superior Equal Subordinate

Little Somewhat A lot

2 Little Somewhat A lot


2

NRF 8% 27% 65% 36.58, p < .001

1% 38% 61% 39.08, p < .001

21% 28% 51% 11.08, p < .004

CTF 7% 24% 69% 66.51, p < .001

6% 30% 64% 57.45, p < .001

20% 29% 51% 17.56, p < .001

Receive information requests from:





2

NRF 14% 60% 26% 24.25, p < .001

18% 50% 32% 11.08, p < .004

37% 42% 21% 11.08, p < .001


CTF 29% 44% 27% 5.89, p < .05

22% 47% 31% 11.73, p < .003

41% 41% 18% 17.56, p < .001

Seek information from:





2

NRF 22% 54% 24% 14.08, p < .001

15% 54% 31% 16.58, p < .001

39% 42% 19% 6.33, p < .04

CTF 21% 64% 15% 47.35, p < .001

11% 57% 32% 36.7, p < .001

35% 37% 28% 1.51, p = .46

The majority of participants from both the NRF and CTF indicated that they shared information rather equally with superiors, equals, and subordinates. A similar pattern of results was observed with regard to receiving information requests and seeking information.

Q4.5. Do you feel you get too much or too little information in order to make decisions? Mann-Whitney = 3375, p < .05

Too much

Somewhat more than I need

Appropriate amount

Somewhat less than I need

Too little

2

NRF 0% 4% 60% 29% 7% 57.11, p < .001

CTF 5% 5% 37% 36% 17% 59.05, p < .001

Analysis of Q4.5 revealed a significant difference in the frequency of ratings between the members of the NRF and the CTF (Mann-Whitney = 3375, p < .05). Examination of the data showed that the NRF had a greater proportion of persons who felt the amount of information received to make decisions was, whereas a greater proportion of members of the CTF felt that they did not receive enough information to make decisions.

Q4.6. In general, how content are you with the information you receive (i.e., does it make you able to make the best possible decisions)? Mann-Whitney = 3008.5, p < .003

Very content

Somewhat content

Neutral Somewhat discontent

Very discontent 2

NRF 11% 43% 38% 8% 0% 27.44, p < .001

CTF 5% 34% 32% 23% 6% 41.21, p < .001

A between-group difference (Mann-Whitney = 3008.5, p < .003) showed that a greater proportion of members of the CTF felt less satisfied with the information they received for


decision making than members from the NRF, who were generally more content with the information that they received for decision making.

Q4.7. In your opinion, what are the greatest obstacles for you to share information in this exercise? 1 = lowest, 5 = highest.

Technical difficulties 1 2 3 4 5 2

NRF 13% 34% 42% 8% 3% 42.31, p < .001

CTF 4% 9% 29% 29% 29( 35.53, p < .001

Language barriers Mann-Whitney U = 3135.5, p < .01

NRF 18% 35% 37% 7% 3% 35.77, p < .001

CTF 39% 28% 24% 9% 0% 20.14, p < .001

Organizational/procedural hinders

NRF 11% 25% 36% 22% 6% 20.77, p < .001

CTF 12% 17% 34% 26% 11% 22.28, p < .001

Differences in national culture

NRF 31% 42% 19% 7% 1% 39.53, p < .001

CTF 33% 35% 23% 6% 3% 49.95, p < .001

Differences in organizational culture

NRF 7% 36% 33% 21% 3% 32.58, p < .001

CTF 15% 31% 31% 21% 2% 30.39, p < .001

Time constraints

NRF 18% 36% 22% 20% 4% 18.64, p < .001

CTF 13% 26% 24% 23% 14% 8.23, p < .08

Questions of priorities

NRF 14% 33% 32% 17% 4% 22.31, p < .001

CTF 9% 27% 48% 14% 2% 72.28, p < .001

Approachability of commander

NRF 29% 38% 18% 8% 7% 25.22, p < .001

CTF 33% 29% 24% 11% 3% 36.52, p < .001

Lacking knowledge about who needs the information


NRF 5% 31% 47% 14% 3% 50.22, p < .001

CTF 14% 25% 32% 21% 8% 20.31, p < .001

Influence and power

NRF 14% 33% 46% 6% 1% 51.75, p < .001

CTF 23% 23% 44% 10% 0% 26.76, p < .001

In general, the members from both the NRF and CTF HQs demonstrated similar patterns of responses to each of the potential obstacles to information sharing. However, the analysis revealed that a larger proportion of members from the CTF rated technical difficulties as being a greater obstacle to information sharing (Mann-Whitney U = 1704, p < .001). Also, a larger proportion of members from the CTF rated language barriers as less problematic to information sharing (Mann-Whitney U = 3135.5, p < .01). Finally, a greater proportion of participants form the CTF rated time constraints as more problematic to information sharing (Mann-Whitney U = 3200, p < .02).

Finding 5. Language

Q5.1. Are you a native English speaker?

Yes No

NRF 44 28

CTF 54 57

Q5.2. Even though you may have a very good command of the English language, how often do you :

Feel that you get more easily stressed when working in an English-speaking environment rather than in your native language?

Never Seldom Sometimes Often Very often 2

NRF 18% 36% 36% 7% 3% 22.59, p < .001

CTF 25% 33% 32% 6% 4% 22.11, p < .001

Feel that you get more tired when working in an English-speaking environment rather than in your native language?


NRF 21% 27% 39% 11% 2% 17.36, p < .002

CTF 20% 20% 37% 13% 10% 12.28, p < .02


Become more reserved about presenting your point of view in English than in your native language?


NRF 11% 32% 34% 21% 2% 16.00, p < .003

CTF 24% 26% 36% 7% 7% 16.18, p < .003

Q5.3. Do you ever feel that it is harder to persuade colleagues of your ideas in English than in your native language?


NRF 14% 27% 43% 14% 2% 21.68, p < .001

CTF 17% 22% 41% 11% 9% 17.29, p < .002

Q5.4. Do you experience being interrupted more often when communicating in English than in your native language?


NRF 29% 45% 21% 5% 0% 15.45, p < .001

CTF 48% 24% 24% 4% 0% 21.41, p < .001

Analyses of Qs 5.2, 5.3, and 5.4 revealed that there was a trend among participants from both the NRF and CTF to rate the potential problems of working in English speaking environments as low.

Q5.5a. In your job here, whom do you communicate with most?

Native English speakers Non-native English speakers No difference 2

NRF 26% 55% 19% 15.22, p < .001

CTF 33% 49% 18% 15.62, p < .001

Q5.5b. What do you think this is due to?

Organizational composition Personal preference Easier/more efficient 2


NRF 81% 15% 4% 72.08, p < .001

CTF 83% 12% 5% 128.00, p < .001

Examination of the rating data from Qs 5.5a and 5.5b showed that most of the participants from both the NRF and CTF communicated with non-native English speakers and that this pattern of communication was due overwhelmingly to the organizational composition of the HQs.

Q5.6. Do you prefer oral or written medias of communication, when:

Giving information to non-native English speakers

Oral Written No preference 2

NRF 23% 26% 51% 9.47, p < .01

CTF 17% 24% 59% 34.75, p < .001

Receiving information from non-native English speakers

NRF 21% 32% 47% 7.58, p < .02

CTF 9% 34% 57% 38.00, p < .001

Giving information to native English speakers

NRF 31% 15% 54% 16.58, p < .001

CTF 23% 19% 58% 32.00, p < .001

Receiving information from native English speakers

NRF 27% 22% 51% 10.75, p < .005

CTF 16% 24% 60% 35.18, p < .001

Analysis of the rating data from Q5.6 showed that the participants from both the NRF and CTF showed a general trend toward no preference for either written or oral communication media when giving and receiving information to and from native and non-native English speakers.

Q5.7a. Have you ever felt that it is more difficult to understand non-native English speakers’ point of view than native speakers? Mann-Whitney U = 3316.5, p < .04



NRF 6% 24% 49% 17% 4% 2 = 45.5, p < .001

CTF 12% 31% 46% 9% 2% 2 = 75.17, p < .001

Q5.7b. Irrespective of rank and numbers, do you feel that native English speakers tend to dominate cooperative situation more than others?


NRF 6% 17% 53% 20% 4% 2 = 54.11, p < .001

CTF 6% 25% 44% 20% 5% 2 = 57.6, p < .001

Q5.7c. How often do you experience your colleagues having problems understanding English? Mann-Whitney U = 2592, p < .001


NRF 1% 36% 52% 10% 1% 2 = 73.56, p < .001

CTF 11% 57% 29% 3 0% 2 = 76.7, p < .01

The analyses of Qs 5.7a,b, and c revealed again that the majority of members from both the NRF and CTF indicated that there were few problems working in native English speaking environments. Interestingly, the analysis from Q5.7a (Mann-Whitney U = 3316.5, p < .04) and Q5.7c (Mann-Whitney U = 2592, p < .001) revealed that a larger proportion of members from the CTF than the NRF rated the potential problems associated with communicating in English as less problematic.

Finding 6. Identity

Q6.1a. Who do you work with most in this exercise?

Superior(s) Equal(s) Subordinate(s) Difficult to say 2

NRF 18% 49% 8% 25% 25.44, p < .001

CTF 16% 49% 13% 22% 37.51, p < .001


Q6.1b. Where in the organization does (do) this (these) person(s) work?

Different group than you

Same group as you Both (same and different group)

2

NRF 6% 51% 43% 25.75, p < .001

CTF 13% 42% 45% 21.56, p < .001

In both the NRF and CTF HQs, the largest proportion of participants indicated that most of their work was carried out with equals (Q6.1a) and that these persons resided predominantly in the same group as the respondent or could be located in a combination of their own or a different group.

Q6.1c. Which gives you the strongest sense of belonging in this work environment here in the exercise?

Nationality/culture 1 2 3 4 5 2

NRF 21% 28% 44% 4% 3% 43.42, p < .001

CTF 15% 21% 41% 19% 4% 39.58, p < .001

Age and gender

NRF 17% 33% 43% 6% 1% 45.92, p < .001

CTF 22% 23% 41% 13% 1% 47.69, p < .001

Team

NRF 1% 3% 50% 31% 15% 60.36, p < .001

CTF 3% 4% 33% 37% 23% 89.05, p < .001

Assignment

NRF 3% 11% 44% 35% 7% 48.97, p < .001

CTF 3% 8% 47% 32% 10% 80.23, p < .001

Rank

NRF 15% 27% 44% 11% 3% 37.31, p < .001

CTF 23% 28% 41% 6% 2% 58.14, p < .001

Language

NRF 7% 33% 46% 10% 4% 49.38, p < .001


CTF 22% 155 44% 18% 1% 54.18, p < .001

Service

NRF 13% 20% 50% 13% 4% 45.5, p < .001

CTF 19% 19% 4% 14% 4% 49.14, p < .001

The analysis of Q6.1c showed that the participants from each HQ provided similar ratings for each of the categories of belonging.

Q6.2a. How important do you feel a group-closeness/belonging is for you?

Very important Somewhat important Little importance Unimportant 2

NRF 32% 56% 8% 4% 48.77, p < .001

CTF 38% 47% 12% 3% 59.48, p < .001

Q6.2b. Closeness/belonging to a group helps me get the job done

Strongly agree Agree Neutral Disagree Strongly disagree 2

NRF 14% 57% 27% 1% 1% 43.42, p < .001

CTF 25% 52% 19% 4% 0% 39.58, p < .001

Examination of Qs 6.2a and 6.2b shows that, for members of both the NRF and CTF, a sense of belonging is important and helps to get the job done.

Q6.2c. I find it harder to get the job done cooperating with people who are different from me in:

Nationality/culture Strongly agree

Agree Neutral Disagree Strongly disagree

2

NRF 0% 8% 33% 38% 21% 15.00, p < .001

CTF 0% 9% 28% 40% 23% 20.27, p < .001

Age and gender

NRF 0% 3% 40% 43% 14% 33.89, p < .001

CTF 1% 3% 30% 43% 23% 72.74, p < .001

Team


NRF 0% 22% 36% 31% 11% 10.22, p < .02

CTF 0% 16% 42% 35% 6% 56.79, p < .001

Assignment

NRF 0% 10% 50% 29% 11% 30.77, p < .001

CTF 0% 16% 42% 35% 7% 36.85, p < .001

Rank

NRF 1% 11% 44% 31% 13% 42.86, p < .001

CTF 0% 5% 37% 43% 15% 43.13, p < .001

Language

NRF 1% 18% 36% 38% 7% 39.11, p < .001

CTF 1% 15% 35% 39% 10% 39.32, p < .001

Service

NRF 1% 8% 42% 35% 14% 43.42, p < .001

CTF 2% 9% 32% 41% 16% 58.77, p < .001

Similar to Q6.1c, participants from both the NRF and CTF provided similar ratings of potential obstacle to cooperation. In general, the ratings show that the majority of participants showed little concern for these categories as possible obstacles to cooperation.

Finding 7. Culture

Q7.1. Who (which nationality) do you work most with in this exercise?

Same nationality as myself Different nationality than myself

NRF 9% 91%

CTF 15% 85%

This finding replicates the finding from Q5.5a, insofar as the participants were interacting/communicating primarily with persons different from themselves in language and culture.

Q7.2. Are there any differences in how you relate to people from different cultures? Mann-Whitney U = 3263.5, p < .03


Yes, very much

Yes, somewhat

Difficult to say

Not much

No, not at all

2

NRF 1% 24% 26% 35% 14% 23.56, p < .001

CTF 2% 21% 13% 36% 28% 40.04, p < .001

The majority of respondents from both the NRF and CTF sites indicated that they do not relate differently to people from different cultures than from their own culture. However, a significant difference between the sites (Mann-Whitney U = 3263.5, p < .03) indicated that there might be a bit more ambivalence on this issue with members from the NRF than the CTF. Moreover, a larger proportion of CTF members than NRF members indicated a stronger “No, not at all” response to this question.

Q7.3. When I have an important task at hand I am most likely to depend on people who are similar to me in:

Nationality/culture Strongly agree

Agree Neutral Disagree Strongly disagree

2

NRF 0% 18% 40% 21% 21% 9.11, p < .001

CTF 1% 24% 29% 32% 14% 34.72, p < .001

Age and gender

NRF 0% 4% 43% 31% 22% 23.10, p < .001

CTF 0% 4% 36% 42% 18% 41.25, p < .001

Team

NRF 8% 47% 28% 13% 4% 44.81, p < .001

CTF 9% 38% 28% 20% 5% 39.67, p < .001

Assignment

NRF 3% 25% 47% 18% 7% 44.53, p < .001

CTF 3% 31% 39% 22% 5% 56.25, p < .001

Rank

NRF 1% 7% 46% 33% 13% 51.05, p < .001

CTF 0% 3% 41% 44% 12% 55.83, p < .001

Language


NRF 1% 22% 39% 28% 10% 31.47, p < .001

CTF 1% 16% 37% 34% 12% 52.02, p < .001

Service

NRF 1% 14% 39% 28% 18% 28.97, p < .001

CTF 1% 7% 41% 36% 15% 68.23, p < .001

Analysis of Q7.3 indicated that the participants from both the NRF and CTF sites were rather neutral in terms of whom they are most likely to depend on.

Q7.4. Do you have any experience of having to increase downward control in order to better master the situation under you command? Mann-Whitney U – 3303.5, p < .03

Strongly agree

Agree Neutral Disagree Strongly disagree 2

NRF 14% 26% 51% 6% 3% 2 = 56.47, p < .001

CTF 18% 39% 39% 3% 3% 2 = 77.96, p < .001

The analysis of Q7.4 shows that a large proportion of persons from both the NRF and CTF have experience with the issue of downward control. However, there appears to be a larger proportion of participants from the CTF that have experience in this area (Mann-Whitney U – 3303.5, p < .03).

Conclusions

The culture and organizational data showed some trends toward differences in basic information processing and in perceptions of team-related issues between the collocated (NRF) and distributed (CTF) HQs. Accordingly, these differences should be taken into account when attempting to explain the various performance and process measures collected in MNE 4. However, due to issues of unequal group sizes and less than optimal response rates, incorporating these findings into other aspects of the MNE 4 research effort should be made cautiously. Nevertheless, previous research has shown this type of data to be important indicators of team performance and, therefore, these data should continue to be collected in these research environments where multinational teams are required to work together to achieve different goals.

Recommendations

It is recommended that the cultural analysis be conducted for future experiments that involve large groups of people from various cultures (particularly military and civilian organizations). It is clear that culture models need to be developed to fully understand impact of the cultural variables on team performance.


References

Bar-Tal, Y. (1994). The effect of mundane decision-making of the need and ability to achieve cognitive structure. European Journal of Personality, 8, 45-58.

Bjørnstad A L (2005): Part I: Allied Warrior 2004 - Pilot study and analysis of cross-cultural organizational issues. FFI/RAPPORT-2005/01709.

Greco, V. & Roger, D. (2001). Coping with uncertainty : the construction and validation of a new measure. Personality and Individual Differences, 31, 519-534.

Grossse, C. U. (2002). Managing communication within virtual teams. Business Communication Quarterly, 65, 22-38.

Klein, H. A., Pongonis, A., & Klein, G. (2000). Cultural barriers to multinational C2 decision-making. Paper presented at the 2000 Command and Control Research and Technology Symposium.

Sutton, J. L. & Pierce, L. G. (2003). A framework for understanding cultural diversity in cognition and teamwork. Proceeding of the 8th International Command and Control Research and Technology Symposium.

Thompson, M. M., Naccarato, M. E., Parker, K. C. H., & Moskowitz, G. B. (2001). The personal need for structure and personal fear of invalidity measures: Historical perspectives, current application, and future directions (pp. 19-39). In G. B. Moskowitz (Ed.), Cognitive Social Psychology: On the Tenure and Future of Social Cognition. Erlbaum.


ANNEX F: Common Intent

Introduction

Common Intent (CI) is a relatively new concept developed by McCann and Pigeau (2000). CI is the combination of explicit and implicit intent amongst two people, a team, or an organization such as coalition operational level headquarters. Common Intent, like situational awareness or decision-making, can be viewed as an intervening variable within cognitive structures that exists somewhere between perceivable information and observable behaviours. At the same time, CI can be thought of as a pre-condition that helps a team perform well. The term, organizational “health”, has been coined to describe what CI might measure. That is, a healthy organization or team should have a common understanding of the common, high level objective because they have common personal expectations, training, tradition, ethos, and common cultural values that will ultimately lead to high team performance.

I-I1

DM1

I-I2

DM2

g1

e1

b1

+

-

g2

e2

b2

s

+

-

p1 p2

Team Information Processing (TIP)

W⊗ ⊗

Figure F.1. Two-person depiction of a multiple-human interaction model

A postulate for CI has been developed based on a model of Team Information Processing (TIP) as shown in Figure F.1 (Farrell, 2006). The figure shows two control loops where each control loop represents a Perceptual Control Theory model of cognition (Powers, 1973). The control loop variables are:

g – a vector of goals p – a vector of perceptions e – a vector of perceptual errors b – a vector of behaviours (includes actions and communications) s – a vector of sensory information

The control loop conceptual transformations or functions are:

I-I – Information Interpreting Function – Comparator Function


DM – Decision Making Function W – World Function

Conceptually, any number of control loops (each representing a single person’s cognitive processes) can be added to this model where the interaction of multiple persons occurs via the World Function. This interaction is coupled.

The advantage of expressing the interaction as control loops is that all the power of Control Theory can be used to explain this interaction. Two key Control Theory concepts are stability and optimization. In terms of stability, coupled systems would have more regions of instability than de-coupled systems. In terms of optimization, partially-coupled systems may produce faster and more accurate responses than de-coupled systems. It is postulated that Common Intent has a tendency to de-couple the team interaction, while at the same time partially-couple the team interaction so as to improve team performance.

Common Intent is more than just having a common high-level goal. McCann, Pigeau, and English (2003) decompose Common (or shared) Intent into two parts:

Explicit Intent: “intent that is made publicly available through orders, briefings, questions and discussions”; and,

Implicit Intent: “the interpretation of explicit intent … derive[d] from personal expectations, military training, tradition and ethos and from deep cultural values”.

Explicit Intent is a restatement of the highest-level team goal expressed as the Commander’s orders. Team members need to perceive, internalize, and understand their own competencies, authorities, and responsibilities (CAR) with respect to the Commander’s orders. Explicit Intent also involves providing guidance for lower-level shared goals. A Commander may give his or her staff specific guidance on the areas of responsibility within which certain goals must be achieved. When a person’s CAR matches their assigned goals then there would be a high probability that the interaction will be stable, and conversely when CAR does not match then the system would likely become unstable. This parsing of goals and responsibilities is akin to having separate feedback control loops. This is a step towards de-coupling the system.

Pigeau, McCann, and English introduced the notion of a Balanced Command Envelope where there exists a balance of Competency, Authority, and Responsibility (CAR) within the organization. The equivalent notion in the TIP model is that each control loop (representing a team member) has the appropriate information interpreting (competency), behaviour (competency), and decision-making (authority) structures for the assigned set of goals (responsibility) within a specific portion of the world (area of responsibility) towards de-coupling the interaction, and therefore system stability would depend primarily on the individuals’ cognitive and behavioural capabilities.

Implicit Intent is intangible and yet it seems to turn ‘good’ teams into ‘excellent’ teams. Ideally, if a team member can interpret and anticipate other members’ goals and behaviours, respectively, in the context of their own goals and behaviours, then they could make adjustments to their decision-making so that goal achievement is optimized (faster with a smaller perceptual error). The interpretation and anticipation, however, involves the team member’s own personal expectations that are shaped by training, tradition, and culture. The


more that training, tradition, and culture are common amongst team members, the more likely their actions will be consistent with the highest level goal as well as with each other.

In summary, Explicit Intent would tend to de-couple the team interaction by having separate areas of responsibility while Implicit Intent would tend to optimize the system by having common expectations that lead to action consistency. Thus, CI would contribute to the de-coupling and optimization of human-human interaction. MNE 4 provided an excellent opportunity to explore the de-coupling and optimization aspects of CI.

Common Intent was previously measured within MNE 3 (Farrell et al., 2005). Those measurements indicated CI between medium and low (Farrell, 2005b). Farrell (2005b) assumed that 100% action consistency is the highest level of CI, 0 to 50% action consistency would be considered low, and medium would be around 70%. The MNE 3 CI results were associated with poor performance in the process steps and an ineffective organization, while the technology was deemed moderately useful. It will be difficult to make any comparisons between CI measured in MNE 3 and MNE 4 because 1) the survey instrument needs to be calibrated, and 2) there were several differences in the experimental design between the two events. However, the general conclusions may be compared.

Experiment Aim for Common Intent

Common Intent, like workload and culture measures, is a generic measurement that is applicable for any experiment involving human subjects performing a task or set of tasks. There is a general belief that CI is critical for the successful completion of the operation. Thus, the experimental aim for Common Intent is to provide some indication of the level of CI. This indication may be used to help explain other observables such as mission success as well as organizational “health” and how well EBAO processes, organization, and technologies were designed and integrated into a single event.

Military warfighting problem Common Intent is trying to solve

There are indications that high performance teams have high levels of Common Intent in order to achieve their goals. Likewise, the CI proposition is that a military team must have high levels of Common Intent to successfully complete their mission. This is even more critical when the team is distributed, includes military and non-military personnel, and involves strategic, operational, and tactical levels of Command and Control as with EBAO.

Stakeholders (if applicable)

Stakeholders include the military community as well as the CD&E and human factors science communities.

Explicit Intent Experiment Design

For MNE 4, the assertion is that if staff members have clear roles and responsibilities then Explicit Intent would be high. The seven Explicit Intent question were as follows:

To what extent do you (an observer) believe that… 1. CG (Command Group)


2. EBP (Effects Based Planning) 3. EBE (Effects Based Execution) 4. EBA (Effects Based Assessment) 5. KBD (Knowledge Base Development) 6. KM (Knowledge Management) 7. MNIG/CAG (Multinational Interagency Group/Civil Agency Group)

…understood the Commander's orders and guidance? Scale: (I don’t know) 0 (Not at all) 1 2 3 4 5 6 7 (Completely)

Methodology and Venues with respect to Explicit Intent

The assertion suggests that both Explicit Intent measures and roles and responsibilities measures would have to be designed, collected, and analysed. Unfortunately, only Explicit Intent measures were purposefully designed. The Explicit Intent survey was given to both CTFHQ (distributed headquarters) and NRF (collocated headquarters). Although, the multinational analyst team requested the expected roles and responsibilities for the entire headquarters, only roles and responsibilities for KM were given in enough detail to perform the analysis. A few general roles and responsibilities questions were asked as part of the Culture and KBD surveys as well.

Participants with respect to Explicit Intent

This survey was given to analysts, observers, and data collectors who made observations on headquarters staff meetings throughout the event. This was the first time in the MNE series that observers were asked to fill out a survey. A scale from 1 to 7 captures their opinion with the option of marking a “0” that indicated that they did not know. This set of questions was presented to observers six times over a three-week period, as Explicit Intent was likely to change over the course of the event.

Analysis Methodology with respect to the COI

The main comparison was Explicit Intent between CTF and NRF, although analyses could be performed across the various staffs as well. An attempt was made to compare any roles and responsibilities results with the Explicit Intent results in order to provide some evidence that the team interactions were de-coupled.

Experiment Design with respect to Implicit Intent

The assertion for Implicit Intent is that if there were high action consistency (an indicator of Implicit Intent) then EBAO process would be optimized, and this would be evident by improvements in products and decision-making. There were two sets of seven Implicit Intent survey questions for each staff. The first set of questions about consistency with Commander’s Intent provides an indication whether the team is moving in harmony with Commander’s Intent. The second question ascertains whether the team is moving in harmony with each other.

To what extent do you (on staff A) believe staff B’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)


To what extent do you (on staff A) believe staff B’s actions are consistent with staff A’s actions? (Not at all) 1 2 3 4 5 6 7 (Completely)

For example, for the CG staff:

To what extent do you believe CG’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe EBP’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe EBE’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe EBA’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe KBD’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe KM’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe MNIG/CAG’s actions are consistent with the Commander’s intent? (Not at all) 1 2 3 4 5 6 7 (Completely)

And

To what extent do you believe the CG team's actions are consistent with each other? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe the EBP team's actions are consistent with the CG team’s actions? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe the EBE team's actions are consistent with the CG team’s actions? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe the EBA team's actions are consistent with the CG team’s actions? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe the KBD team's actions are consistent with the CG team’s actions? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe the KM team's actions are consistent with the CG team’s actions? (Not at all) 1 2 3 4 5 6 7 (Completely)

To what extent do you believe the MING/CAG team's actions are consistent with the CG team’s actions? (Not at all) 1 2 3 4 5 6 7 (Completely)


There were ninety-eight questions in total. However, one must be careful how the data are combined since each question is measuring a separate aspect of action consistency. The first question is the same for all staff members (similar to the Explicit Intent question), but the second question is different for each of the seven staffs within each headquarters (i.e., each question is a different dimension).

Methodology and Venues with respect to Implicit Intent

The plan was to repeat the EBAO process three times, and so one might be able to track quality of the products and decision-making. However, the purpose of the experiment was to test the robustness (vice effectiveness) of the EBAO process. Even though the plan was to repeat the EBAO process three times with increasing complexity, the product quality was not measured and only a few questions related to decision-making were designed as part of the Culture and Human Factors surveys.

Participants with respect to Implicit Intent

This survey was given to the experiment players within the CTF and NRF headquarters.

Analysis Methodology with respect to Implicit Intent

The survey scale is a bi-polar ordinal scale for these sets of questions. That is, even though the absolute ends of the scale are nominal (named), the names were designed to be the polar opposite of each other. Thus, each question’s scale represents a single dimension. Given the wording of the question (to what extent), it is assumed that the respondent perceives equal intervals between each number, and so the data are treated as interval data. This is an approximation of the theoretically continuous nature of Common Intent between two extreme values. In future studies, respondents should be allowed to place a mark on a bi-polar scale (no numbers) thus indicating their perception of a percentage of the maximum value.

Given that each question is a single dimension, they must be treated as separate distributions. Moreover, the sample size for each distribution is different because the group sizes vary between 13 and 34 for the Explicit Intent questions and 3 to 34 for the Implicit Intent questions. Nevertheless, tables of means and standard deviations with key pair-wise t-tests are produced where it would make sense to do so.

The vector method (Farrell, 2005a) was developed specifically for responses to questions that come from multiple dimensions. Whereas traditional statistics use arithmetic operations on multiple values along a single dimension, the vector method treats each datum as a number along its orthogonal dimension, thus multiple orthogonal values form a single vector in a multiple dimensional space. In other words, treating the data as multiple orthogonal elements means that ordinarily different variables (i.e., “apples and oranges”) can be added using vector algebra. The Common Intent survey was designed with the vector method analytical technique in mind.

Calculating the mean and standard deviation for this set of data requires two assumptions: 1) the data come from interval scales, and 2) each respondent has close to the same context and interpretation of the question, so that they answer the question on the same single dimension.


However, many of the analysts, observers, and data collectors in MNE 4 did not have the same context or interpretation. Although they all had access to all the IWS meetings, it was physically impossible to track all the meetings. And so, observers chose to be involved in those conversations that were pertinent to their specific analyses. For example, one EBA observer only answered the EBA question while other EBA observers gave their opinion on other staffs they might have observed on that particular day. The context was not uniform.

The vector method does require data to come from a bi-polar scale, however it need not assume that the data comes from a single dimension. The vector method treats each individual answer as a single value on a single dimension and then uses vector algebra to combine responses for either a single question across multiple respondents (Figure F.2a), a single respondent across multiple questions (Figure F.2b), or multiple questions across multiple respondents.

The resultant measured vector is compared to its reference vector. They have the same dimensions. In this case, the reference vector is defined as having a “7” indicated for every respondent and every question. This would constitute a complete understanding of the Commander’s orders and guidance; that is 100% Explicit Intent. Therefore any measured vector’s magnitude would be less than or equal to the reference vector’s magnitude, while the angle between the measured and reference vectors may vary.

Subject 1

Subject 2

71

7 Question A

100%

x%

measured vector reference

vector

Question A

Question B

71

7 Subject 1

100%

x%

measured vector reference

vector

a) b)

Figure F.2. Multiple responses to a single question OR a single respondent’s responses to multiple questions may form the measure vector OR a combination of multiple respondents

and multiple questions can be compared to their corresponding reference vector.

COI 1 – Measuring Common Intent

The Critical Operating Issue is that high levels of Common Intent as the combination of explicit awareness and implicit interpretation of Commander’s Intent is essential for CTF and NRF headquarters staff to effectively perform Effects-Based Operations (although the effectiveness of outcomes were not measured during this event). The surveys described above attempt to measure Common Intent, while measures of EBAO performance will be derived from other roles and responsibilities and decision-making questionnaires.

1.a Explicit Intent Overall Assessment Results


Table F.1 lists the mean and standard deviation for the Explicit Intent results (to what extent did staff A understand the Commander’s orders and guidance):

TABLE F.1 MEANS AND STANDARD DEVIATION FOR EACH QUESTION Mean Standard Deviation

CTF CG EBP EBE EBA KBD KM MNIG CG EBP EBE EBA KBD KM MNIG 28-Feb 4.9 4.2 4.2 4.5 3.9 3.7 3.6 1.4 1.4 1.3 1.3 1.3 1.3 1.4

2-Mar 4.5 4.5 3.6 5.0 4.4 4.3 3.5 1.6 0.9 1.5 1.1 1.2 1.0 1.1 7-Mar 5 4.9 4.5 5.1 4.2 3.7 4.5 1.3 1.1 1.3 1.1 1.2 1.4 1.2 9-Mar 4.8 4.8 4.6 4.9 4.3 4.1 4.4 1.6 1.0 1.3 1.3 1.2 1.2 1.4

14-Mar 4.8 4.9 4.8 4.9 4.5 3.9 4.6 1.2 1.1 1.3 1.3 1.4 1.2 1.2 16-Mar 5.0 4.9 4.8 5.2 4.3 4.0 4.6 1.4 1.3 1.3 1.0 1.4 1.1 1.3 NRF CG EBP EBE EBA KBD KM CAG CG EBP EBE EBA KBD KM CAG 28-Feb 5.2 5.1 4.4 4.3 4.4 4.8 4.2 1.4 1.4 1.4 1.5 1.5 1.3 1.8

2-Mar 5.0 5.5 4.9 5.0 5.1 5.2 4.7 1.3 0.8 1.1 1.2 1.3 1.1 1.1 7-Mar 5.7 5.5 5.1 5.1 4.7 4.8 4.1 0.9 1.0 1.2 1.3 1.1 1.1 1.7 9-Mar 5.8 5.5 5.4 5.6 4.9 4.6 4.2 1.0 1.1 1.0 1.0 1.0 1.1 1.1

14-Mar 5.9 5.6 5.5 5.6 5.2 5.2 4.8 0.8 0.8 0.8 1.0 0.9 0.9 1.3 16-Mar 6.0 5.6 5.7 5.8 5.3 5.2 4.4 0.7 0.9 0.6 0.7 0.6 0.7 1.3

1.b Explicit Intent Findings

Finding 1. The minimum mean corresponded to the CTF MNIG staff on 2 Mar.

Finding 2. The maximum mean corresponded to the NRF CG staff on 16 Mar.

Finding 3. The CTF means were generally lower than the NRF means.

Finding 4. Mean values generally increased from 28 Feb to 16 Mar.

Finding 5. CTF standard deviation generally stayed constant while NRF standard deviation generally became smaller as the event progressed in time.

Pair-wise t-tests for samples of different sizes were performed and only significant differences (p<0.5) are reported below.

Finding 6. The CTF EBE mean on 2 Mar (3.6) was significantly different from the staff’s means on 7 Mar (4.5), 9 Mar (4.6), 14 Mar (4.8), and 16 Mar (4.8). The low level on 2 Mar may be due to the frustration reported during week one.

Finding 7. The CTF EBA mean on 28 Feb (4.5) was significantly different from the staff’s mean on 16 Mar (5.2). The EBA staff’s understanding steadily increased over the course of the three weeks.

Finding 8. The CTF MNIG mean on 28 Feb (3.6) was significantly different from the staff’s mean on 2 Mar (3.5), which was significantly different from the staff’s means on 7 Mar (4.5), 9 Mar (4.4), 14 Mar (4.6), and 16 Mar (4.6). MNIG staff had the lowest understanding, but it increased during weeks 2 and 3.


Finding 9. The NRF CG mean on 2 Mar (5.0) was significantly different from the staff’s means on 14 Mar (5.9) and 16 Mar (6.0). The CG staff’s understanding of their own orders started relatively high and increased to the highest level. This may be a key finding towards explaining the apparent success of the NRF EBAO.

Finding 10. The NRF CAG mean on 7 Mar (4.1) was significantly different from the staff’s mean on 9 Mar (4.2), which was significantly different from the staff’s mean on the 14 Mar (4.8), which was significantly different from the staff’s mean on the 16 Mar (4.4). While the CAG staff increased their understanding, there is a curious dip towards the end of week 3 (perhaps it was fatigue).

The pair-wise correlation between questions per day was calculated. A correlation of 1 would indicate that both staffs would either completely or not at all understand the Commander’s orders and guidance, while a correlation of –1 would mean that the two staffs would be completely opposite in their understanding. For Common Intent to exist, the correlation would need to tend towards 1. The following table are correlation values for KM and | R | > 0.5. Similar tables may be generated for all the staffs.

TABLE F.2 CORRELATIONS BETWEEN QUESTIONS FOR KM AND |R| > 0.5 Date Staff A Staff B R

CTF 28 Feb KM CG -0.70

KM KBD -0.90 2 Mar KM CG 0.76

KM EBP -0.60 EBA KM -1.00

7 Mar KM CG -0.50 9 Mar KM CG 0.67

16 Mar KM CG -0.6 NRF

2 Mar KM EBA -0.8 KM KBD -0.6 CAG KM -0.5

14 Mar KM EBE 0.58

Finding 11. For the CTF, an interesting pattern emerged between KM and CG staffs. The correlation sign reversed each day. Such swings in understanding may indicate that Explicit Intent was unstable between these two staffs.

Finding 12. The CTF and NRF KM and KBD were negatively correlated on 28 Feb and 2 Mar, respectively. The sub-groups had opposite understandings of the Commander’s orders and guidance even though they comprise of a single Knowledge Support staff.

Finding 13. Similarly, the CTF and NRF KM and EBA were strongly negatively correlated.

Table F.3 lists the measured vector magnitude as a percent of the reference vector as well as the angle between measured and reference vectors. Two variables are required to fully define a vector in multiple dimensional spaces. Note that these results do not provide any


information on the relative positions between measured vectors since the reference vectors have different magnitudes and directions relative to each other. Relative differences between measured vectors would require a cluster analysis in a multi-dimensional space. These ideas were introduced in Farrell (2005a) but require further investigation.

TABLE F.3 MEASURED VECTOR MAGNITUDE AS A PERCENT OF THE REFERENCE VECTOR AND THE ANGLE BETWEEN MEASURED AND REFERENCE VECTORS magnitude (percent of reference vector) angle between measured and reference vector CG EBP EBE EBA KBD KM MNIG All CG EBP EBE EBA KBD KM MNIG All CTF CTF

28 Feb 69 58 58 62 53 49 50 58 13 12 12 12 12 12 13 13 2 Mar 64 61 50 70 60 57 46 58 15 8.6 14 10 11 9.1 10 12 7 Mar 69 67 62 70 57 51 61 63 12 11 12 9.9 12 12 12 12 9 Mar 68 66 63 69 58 55 61 63 14 9.1 12 12 11 11 13 12

14 Mar 66 67 67 69 62 52 63 64 11 10 12 12 13 11 11 12 16 Mar 70 68 67 72 60 54 64 65 13 12 12 9.5 13 10 12 12

NRF NRF 28 Feb 74 71 61 61 62 67 61 66 13 13 13 14 14 12 16 14

2 Mar 70 76 68 69 72 73 64 70 12 7.1 10 11 12 9.9 10 11 7 Mar 79 77 71 71 64 66 58 71 8.4 9.4 11 12 10 10 15 12 9 Mar 81 78 75 78 67 63 56 72 8.9 10 9.2 9.2 9.4 10 9.9 11

14 Mar 83 78 76 78 71 72 66 76 7.6 7.3 7.3 9.0 7.9 8.2 11 9.0 16 Mar 84 78 79 81 72 71 60 76 6.7 8.1 5.5 6.0 5.5 6.2 11 8.5

A convenient (but not unique) way to combine both the relative magnitude and the angle between the measured and the reference vectors is the Euclidean product (or the dot product) that denotes the projection of one vector onto another (x% in Figure F.2). In this case, the projection value represents the percentage of a reference vector where Commander’s orders and guidance were completely understood.

TABLE F.4 PROJECTION OF MEASURED VECTOR ONTO REFERENCE VECTOR (%) Projection (%) CG EBP EBE EBA KBD KM MNIG All CTF

28 Feb 65 53 54 58 48 44 44 532 Mar 58 59 44 67 57 55 42 547 Mar 66 64 58 68 53 46 58 599 Mar 63 64 59 65 55 52 56 59

14 Mar 63 64 63 65 58 48 60 6116 Mar 67 65 64 70 56 51 60 62

NRF28 Feb 71 68 57 56 57 64 54 61

2 Mar 67 75 66 67 69 71 61 687 Mar 78 75 68 68 62 64 51 679 Mar 79 75 73 77 65 60 53 70

14 Mar 82 76 75 77 70 70 63 7416 Mar 83 77 78 80 71 70 56 74


There is a linear relationship between the mean values in Table F.1 and the projection in Table F.4. In fact, it can be shown that the projection and the mean have an identical formula (Farrell, 2005a). The projection calculation may be used for data that comes from a single or multiple dimensions while the mean can only be calculated for data coming from a single dimension. Therefore Findings 1 to 13 would be the same whether the data were reduced using mean and standard deviation or the vector method. Also, the correlation between the magnitude values and the projection values is 0.99. That is, the projection alone is a good indicator of the similarity between the measured and reference vectors.

Table F.3 and F.4 has an additional column “All”. That is, all the responses on a given day are combined using vector algebra and compared to its reference vector. This metric would indicate a global value for Explicit Intent for the headquarters on that given day. Additional findings are reported below based on the “All” projections.

Finding 14. As a whole, the two staffs’ understanding of the Commander’s orders and guidance steadily increased over the course of the event (53% to 62% for the CTF, and 61% to 74% for the NRF).

Finding 15. In every case, except for 28 Feb, the NRF reported a higher level of understanding than the CTF.

2.a Implicit Intent Overall Assessment Results

Recall that players are the respondents to the Implicit Intent survey. All players on the CTF and NRF staffs are asked the same first question – to what extent do you believe staff A’s actions are consistent with Commander’s Intent – seven times (once for CG, EBP, EBE, EBA, KBD, KM, or MNIG/CAG) on 6 separate dates. Table F.5 provides the means and standard deviations for the first Implicit Intent question. Note the sample sizes for the statistics.

TABLE F.5 ACTION CONSISTENCY WITH RESPECT TO COMMANDER’S INTENT Average Standard deviation CTF CG EBP EBE EBA KBD KM MNIG CG EBP EBE EBA KBD KM MNIG sample size 28 Feb 4.8 4.9 4.7 4.7 4.7 4.8 4.6 1.3 1.3 1.3 1.4 1.4 1.6 1.4 116

2 Mar 4.7 4.8 4.4 4.6 4.4 4.4 4.5 1.3 1.3 1.4 1.4 1.4 1.4 1.3 121 7 Mar 5.1 5.0 4.9 4.9 4.7 4.7 5 1.2 1.3 1.4 1.3 1.5 1.4 1.3 120 9 Mar 5.1 5.2 5.0 5.0 4.8 4.7 4.8 1.4 1.3 1.4 1.3 1.4 1.4 1.5 121

14 Mar 5.3 5.3 5.3 5.3 5.0 4.9 5.1 1.2 1.2 1.2 1.1 1.3 1.3 1.1 115 16 Mar 5.4 5.4 5.4 5.2 5.1 5.0 5.1 1.2 1.2 1.1 1.2 1.4 1.4 1.2 113 NRF CG EBP EBE EBA KBD KM CAG CG EBP EBE EBA KBD KM CAG sample size 28 Feb 5.0 5.1 4.9 4.8 4.9 4.8 4.8 1.3 1.2 1.2 1.2 1.2 1.4 1.3 65

2 Mar 5.1 5.1 5.2 5.0 4.9 4.9 4.9 1.3 1.1 1.1 1.2 1.2 1.3 1.3 62 7 Mar 5.2 5.1 5.2 5.0 4.9 4.8 4.9 1.3 1.2 1.3 1.2 1.2 1.3 1.3 60 9 Mar 5.4 5.2 5.3 5.4 5.3 5.0 5.1 1.1 1.1 1.1 1.1 1.1 1.1 1.3 60

14 Mar 5.3 5.3 5.5 5.4 5.4 5.1 5.2 1.1 1.1 1.0 1.0 1.1 1.2 1.3 51 16 Mar 5.7 5.7 5.5 5.5 5.4 5.4 5.3 1.1 1.0 1.1 0.9 1.1 1.0 1.4 32

Finding 1. The minimum mean corresponded to the CTF EBE, EBA, and KM staffs on 2 Mar.


Finding 2. The maximum mean corresponded to the NRF CG and EBP staffs on 16 Mar.

Finding 3. The CTF means were generally lower than the NRF means.

Finding 4. Mean values generally increased from 28 Feb to 16 Mar.

Pair-wise t-tests for samples of different sizes were performed and only patterns in the mean with significant differences (p<0.5) are reported below.

Finding 5. The CTF CG, EBE, and EBA means on 28 Feb were significantly different from their respective means on 7 Mar, 14 Mar, and 16 Mar, and their means on the 2 Mar were significantly different from their respective means on the 7 Mar, 9 Mar, 14 Mar, and 16 Mar. Also, EBA staff’s mean on 9 Mar was significantly different than the staff’s mean on the 14 Mar. This indicated a dip in their responses on the 2 Mar but it recovered after the final two weeks.

Finding 6. The CTF EBP and MNIG means on 28 Feb were significantly different from their respective means on 14 Mar and 16 Mar, and the EBP means on the 2 Mar were significantly different from their means on the 9 Mar, 14 Mar, and 16 Mar while the MNIG means on the 2 Mar were significantly different from their means on the 14 Mar and 16 Mar. This indicated a similar dip as before.

Finding 7. The CTF KBD staff’s mean on 2 Mar was significantly different from their means on 9 Mar, 14 Mar, and 16 Mar, while CTF KM staff’s mean on 2 Mar was significantly different from their means on 7 Mar, 14 Mar, and 16 Mar. Again, this indicated a dip in their responses on the 2 Mar.

Finding 8. Although the NRF staff yielded 31 significant differences, they were not concentrated on a single day signifying a dip in responses, but were spread fairly evenly across all pair-wise matches of days and across all staffs. The pattern of significant differences confirm Finding 4 where NRF means started at 4.8 and increased steadily to 5.7 throughout the event.

The pair-wise correlation between questions per day was calculated. Correlation values that satisfy | R | > 0.5 are reported below.

Finding 9. The NATO staff on 2 Mar yielded a correlation of R = 0.5 between KM and EBP, and this was the only correlation that satisfied the criteria. This would indicate that the questions represented separate dimensions.

For the second question, each staff within the CTF and NRF headquarters were asked different questions – to what extent do you believe staff A’s actions are consistent with Staff B’s actions – seven times on 6 separate dates depending on what staff the player belonged to (CG, EBP, EBE, EBA, KBD, KM, or MNIG/CAG). Thus, the data on a particular day cannot be combined. Table F.6 provides the means and standard deviations for the second Implicit Intent question. Note the sample sizes.


TABLE F.6 MEAN VALUES FOR IMPLICIT INTENT QUESTIONS

Average Standard deviation

CTF Staff CG EBP EBE EBA KBD KM MNIG CG EBP EBE EBA KBD KM MNIG sample size

28 Feb CG 6.3 6.0 6.0 6.0 6.0 6.0 6.0 1.0 1.2 1.2 1.2 1.2 1.2 1.2 4 EBA 5.3 5.2 4.8 5.6 5.0 5.0 4.3 0.8 1.0 1.3 1.0 1.1 1.1 1.4 12 EBE 3.9 3.3 4.2 3.9 3.5 3.7 3.7 1.8 1.7 1.5 1.7 1.5 1.3 1.6 19 EBP 4.9 4.9 4.5 4.6 4.4 4.4 4.1 1.0 1.1 1.2 1.3 1.2 1.3 1.4 32 KBD 4.3 4.1 3.9 4.0 4.4 4.3 3.8 1.3 1.0 1.0 1.0 1.3 1.4 1.1 22 KM 4.3 4.5 4.2 4.7 4.1 4.8 4.0 1.6 1.5 1.5 1.7 1.5 1.6 1.1 18 MNIG 3.8 3.9 3.9 3.9 4.3 4.3 5.4 1.3 1.2 1.2 1.2 1.1 1.1 1.3 9

2 Mar CG 5.8 5.8 5.8 6.0 5.8 5.8 6.0 1.3 1.3 1.3 0.8 1.3 1.3 0.8 4 EBA 5.0 4.8 4.4 5.7 4.3 4.4 4.5 1.3 1.0 1.3 1.2 1.2 1.2 1.2 12 EBE 5 4.6 4.8 3.9 4.0 4.2 4.7 1.3 1.2 1.1 1.2 1.3 1.2 1.0 22 EBP 4.4 4.8 4.0 4.4 4.0 3.9 4.2 1.2 1.5 1.1 1.1 1.2 1.4 1.1 34 KBD 3.9 3.9 3.4 3.7 4.7 4.3 3.9 1.6 1.5 1.4 1.6 1.5 1.2 1.4 21 KM 4.4 4.4 4.2 4.7 4.8 4.8 4.2 1.1 1.2 1.5 1.3 1.1 1.4 1.2 18 MNIG 4.0 4.0 3.9 3.3 3.8 4.0 5.6 0.9 1.2 1.2 1.3 1.1 0.9 1.2 10

7 Mar CG 6.8 6.8 6.8 6.8 6.8 6.8 6.8 0.5 0.5 0.5 0.5 0.5 0.5 0.5 4 EBA 5.0 5.1 5.1 5.6 4.5 4.7 5.1 1.3 1.5 1.1 1.4 1.6 1.6 1.3 11 EBE 5.1 4.5 4.9 4.4 4.5 4.3 4.8 1.2 1.2 1.2 1.1 1.1 1.1 0.9 21 EBP 5.2 5.3 4.6 4.7 4.5 4.3 4.8 1.1 1.1 1.1 1.2 1.1 1.3 1.1 34 KBD 4.5 3.9 3.7 4.0 4.7 4.4 4.4 1.3 1.4 1.3 1.3 1.4 1.3 1.3 21 KM 4.3 4.6 4.5 4.8 5 4.8 4.4 1.5 1.5 1.7 1.5 1.5 1.5 1.3 18 MNIG 4.3 4.3 4.1 4.4 3.8 4.1 5.5 1.5 1.1 1.4 1.3 1.3 1.2 1.4 11

9 Mar CG 6.8 6.8 6.8 6.8 6.8 6.8 6.8 0.5 0.5 0.5 0.5 0.5 0.5 0.5 4 EBA 5.4 4.9 5.2 6 4.6 4.8 4.9 1 1.6 1.3 1.4 1.6 1.5 1.4 12 EBE 5 4.8 5 4.5 4.5 4.9 4.6 1.6 1.6 1.2 1.4 1.3 1.1 1.4 21 EBP 5.3 5.3 4.7 4.7 4.5 4.5 4.9 1.1 1 1.3 1.3 1.3 1.5 1.3 33 KBD 4 4.1 4.1 4.1 4.7 4.2 4 1.5 1.5 1.3 1.5 1.5 1.6 1.4 22 KM 4.5 4.4 4.1 4.6 4.3 4.8 4.4 1.7 1.4 1.6 1.6 1.3 1.4 1.5 18 MNIG 4.5 3.8 4.5 4.3 4.2 3.7 5.9 1.2 1.5 1.2 1.3 0.8 1.1 1.1 11

14 Mar CG 6.8 6.8 6.8 6.8 6.8 6.8 6.8 0.5 0.5 0.5 0.5 0.5 0.5 0.5 4 EBA 5.8 5.6 5.7 5.7 5.4 5.1 5.1 0.8 0.8 0.8 0.8 1.3 1.4 1.5 10 EBE 5.1 4.8 5.6 4.7 4.8 5.2 5.1 1.4 1.5 1.1 1.3 1.1 1.1 1.3 20 EBP 5.6 5.4 5.1 5.1 4.8 4.7 5.0 0.9 1.0 1.1 1.2 1.2 1.5 1.2 34 KBD 4.3 4.7 4.5 4.4 5.2 4.8 4.7 1.4 1.4 1.4 1.5 1.1 1.0 1.2 21 KM 4.1 4.6 4.2 4.7 4.6 4.8 4.2 1.5 1.5 1.5 1.5 1.1 1.3 1.6 18 MNIG 4.8 4.0 4.8 4.5 3.8 3.9 6.0 1.2 1.8 1.4 1.3 1.5 1.6 0.9 8

16 Mar CG 6.8 6.8 6.8 6.8 6.8 6.8 6.8 0.5 0.5 0.5 0.5 0.5 0.5 0.5 4 EBA 5.8 5.3 5.8 5.9 5.2 5.4 5.3 0.9 1.3 0.6 0.7 1.0 1.0 1.1 10 EBE 5.6 5.1 5.7 5.0 4.7 4.7 5.3 1.0 1.3 0.9 1.0 1.3 1.3 1.1 20 EBP 5.4 5.5 5.4 5.3 5.2 5 5.4 1.3 1.3 1.0 1.4 1.4 1.6 1.1 34 KBD 4.8 4.3 4.5 4.5 5.1 4.7 4.4 1.3 1.6 1.6 1.6 1.6 1.6 1.4 20 KM 4.8 4.9 4.7 4.5 4.6 5.2 4.4 1.6 1.5 1.5 1.7 1.5 1.4 1.4 18


MNIG 5.0 3.4 4.9 5.0 3.9 3.9 6.4 1.0 1.8 0.7 0.6 1.3 1.3 0.5 7

NRF CG EBP EBE EBA KBD KM CAG CG EBP EBE EBA KBD KM CAG 528 Feb CG 5.8 5.4 5.0 4.8 5.2 5.2 5.2 0.8 1.3 1.0 0.8 1.1 1.3 1.3 8

EBA 4.9 4.1 4.0 5.3 4.0 4.3 4.1 1.0 0.8 0.9 1.0 0.9 0.9 1.2 18 EBE 4.7 4.8 5.1 4.7 4.6 4.7 4.4 1.5 1.6 1.6 1.6 1.5 1.4 1.3 16 KBD 4.3 4.6 4.2 4.3 4.7 5 4.3 1.4 1.2 1.1 0.9 1.1 1.3 1.3 12 KM 4.7 4.7 4.4 4.5 4.9 5.1 4.5 1.7 1.3 1.2 1.2 1.3 1.4 1.6 6 CAG 5.8 5.0 5.5 5.5 5.3 5.3 6.5 1.2 1.1 1.2 1.2 1.4 1.4 0.5 5

2 Mar CG 6.6 5.8 6.0 5.0 5.8 6.0 6.0 0.5 0.4 0.7 0.7 0.8 0.7 0.7 8 EBA 4.8 4.8 4.9 5.9 4.1 4.4 4.1 0.9 1.0 1.0 0.6 0.8 0.7 0.6 18 EBE 4.8 4.7 4.8 4.7 4.6 4.6 4.7 1.2 1.4 1.5 1.3 1.5 1.5 1.0 15 KBD 4.4 4.2 4.3 4.2 4.8 4.9 4.3 1.4 1.0 0.8 0.9 1.4 1.4 1.1 10 KM 5.3 5.0 4.9 4.5 5.0 5.5 4.6 1.2 1.1 1.2 1.1 1.2 1.3 1.2 6 CAG 6 5.2 5 5.2 5.3 5.2 6.3 0.6 1.2 1.7 1.2 1.2 1.2 0.5 5

7 Mar CG 6.2 5.6 5.8 4.8 5.6 5.4 5.8 0.8 0.5 0.8 0.8 0.5 0.9 0.8 8 EBA 4.6 4.1 4.5 5.9 4.5 4.5 4.3 0.7 1.2 1.7 1.0 1.2 1.2 0.7 17 EBE 4.8 4.9 5.4 4.9 4.5 4.6 4.4 1.3 1.2 1.4 1.1 1.4 1.3 1.3 14 KBD 5.1 4.9 4.8 5.0 5.2 5.1 4.9 1.1 1.0 1.0 1.0 1 0.9 1.1 10 KM 4.9 4.8 5.1 5.1 4.9 5.0 4.6 1.7 1.6 1.8 1.8 1.7 1.6 1.7 6 CAG 5.5 5.7 5.5 5.3 5.3 5.2 5.7 1.0 1.0 1.0 1.0 1.0 1.2 1.0 5

9 Mar CG 6.2 5.8 6.0 5.0 6.0 5.6 6.2 0.4 0.4 0.7 1.2 0.7 0.9 0.8 8 EBA 4.6 4.0 4.3 5.9 4.3 4.1 4.1 0.7 1.2 0.7 0.8 1.2 1.0 0.6 17 EBE 5.1 5.2 5.4 5.1 5.0 5.0 4.7 1.0 1.1 1.2 1.0 1.1 1.1 1.1 14 KBD 5.3 5.3 5.2 5.2 5.5 5.1 5.1 1.1 0.9 1.3 1.1 0.9 0.9 0.9 10 KM 5.2 4.8 4.9 4.9 5.3 5.3 5.0 1.8 1.7 1.7 1.7 1.8 1.8 1.8 6 CAG 5.3 4.8 4.7 5.2 5.0 5.0 5.5 1.2 1.2 1.2 0.4 0.9 0.9 1.2 5

14 Mar CG 6.4 5.4 5.6 5.6 6.0 5.6 5.6 0.5 1.1 0.9 1.1 1.0 1.3 1.3 7 EBA 5 4.7 4.7 6.0 4.3 4.3 4.4 0.8 1.0 1.3 0.8 0.8 0.8 1.0 14 EBE 4.6 5.2 5.6 5.2 4.9 4.7 4.5 1.2 1.1 1.0 0.9 1.3 1.1 1.2 14 KBD 5.1 5.2 5.1 5.1 5.3 4.9 4.9 1.1 0.9 0.9 0.9 1.0 1.1 1.3 8 KM 5.6 5.3 5.5 5.5 5.6 5.5 5.5 1.1 1.0 0.9 0.8 1.1 0.9 1.1 3 CAG 5.7 5.7 4.3 4.3 5.3 5.0 6.3 1.2 1.2 0.6 0.6 1.5 1.0 1.2 4

16 Mar CG 6.3 5.8 6.0 5.8 5.8 5.5 5.5 1.0 1.0 1.2 1.0 1.0 1.9 1.7 5 EBA 5.0 4.6 5.0 6 5 4.6 4.2 0.0 0.5 0.7 0.7 0.7 0.9 0.8 5 EBE 5.4 5.2 5.2 4.8 4.4 4.2 4.0 1.3 1.3 1.1 1.1 1.1 1.5 1.9 11 KBD 5.2 5.3 4.9 5.0 5.5 5.2 5.2 0.9 0.9 1.0 1.0 0.9 0.9 1.1 4 KM 6.5 6.0 6.3 6.0 6.0 5.3 5.8 0.6 0.8 0.5 0.8 0.0 1.7 1.3 3 CAG 5.7 6.0 5.7 5.7 6.0 6.0 7.0 1.5 1.0 1.5 1.2 1.0 1.0 0.0 4

Finding 1. NRF EBP staff data were missing from the raw data set.

Finding 2. Each mean value represents a single dimension. The staff sizes that responded varied from 3 to 18.

Finding 3. The minimum mean value was 3.3 on 28 Feb where the CTF EBA staff’s action consistency with respect to CTF EBE staff was reported.


Finding 4. The maximum mean value was 7 on 16 Mar where all four NRF CAG staff members gave both questions a 7 score. The standard deviation was zero.

If desired, statistical significance can be found for 903 pair-wise t-tests. However, the key analysis is to compare CTF and NRF globally. The vector method is used to combine the data from all staffs regardless if the data comes from a single dimension (i.e., the first Implicit Intent question) or multiple dimensions (the second Implicit Intent question). As before, the reference vector would be equivalent to each respondent answering “7” to each question for each group of responses being combined. The percent action consistency for each headquarters on each day is given in Table F.7, and then plotted in Figure F.3.

TABLE F.7 ACTION CONSISTENCY PROJECTION OF MEASURED VECTOR ONTO REFERENCE VECTOR (%)

Action Consistency with respect to (%) Commander’s Intent Each other Number of Dimensions CTF

28 Feb 62 56 8122 Mar 59 56 8477 Mar 65 61 8409 Mar 66 62 847

14 Mar 70 66 80516 Mar 70 68 791

NRF 28 Feb 65 62 455

2 Mar 67 64 4347 Mar 67 66 4189 Mar 71 68 420

14 Mar 72 69 35716 Mar 75 72 224

Action Consistency (%)

55

60

65

70

75

28-Feb 2-Mar 4-Mar 6-Mar 8-Mar 10-Mar 12-Mar 14-Mar 16-MarDate

perc

ent o

f Com

plet

ely

Con

sist

ent

CTF wrt Cmd IntentCTF wrt Each OtherNRF wrt Cmd IntentNRF wrt Each Other

Figure F.3. Action Consistency results as an indicator of Implicit Intent

Finding 5. Action consistency generally increased until the end of the event. Clearly, time is required for Implicit Intent to develop within a headquarters.


Finding 6. The NRF staff reported consistently higher levels of action consistency than the CTF. This finding agrees with the Explicit Intent findings and may be a key indicator for the apparent success of the NRF.

Finding 7. Action consistency with respect to Commander’s Intent was given a higher rating than action consistency with each other, although the patterns are fairly similar. This may indicate that only one question is required to determine the level of Implicit Intent.

COI 2 – Explicit Intent and Roles and Responsibilities

Recall that the Team Information Processing model suggests a link between high Explicit Intent and clear roles and responsibilities. Two roles and responsibilities results were used to shed some light on this assertion. The two questions were:

R&R1 – rate the extent to which “my team was effective in assigning specific responsibilities to members (every person knew what to do)”

R&R2 – rate the extent to which “The CONOPS in combination with the job descriptions appropriately describes how to conduct the tasks for which I am responsible (KBD staff only)”

Using the vector method technique, the roles and responsibilities data are combined across the two headquarters and compared to their respective reference vector (as before, “7” indicated the highest extent possible). The projection results are given in Table F.8 along side and as near as possible to the corresponding Explicit Intent results.

TABLE F.8 COMPARING EI AND ROLES AND RESPONSIBILITIES RESULTS Date 13 Mar 14 Mar 15 Mar 16 Mar

Question R&R1 EI R&R2 (KBD only) EI (KBD only) NRF 75% 74% 50% 71%

CTF 68% 61% 38% 56%

Finding 1. Both the roles and responsibilities and the Explicit Intent results seem to correspond with each other, however, too few data points are available to draw any firm conclusions.

COI 3 – Implicit Intent and Decision-Making

The TIP model asserts that outcomes and decisions are optimized due to high levels of Implicit Intent. However, for MNE 4, the actual plan, execution of the plan, or the assessment of the plan was not the focus, but rather whether the process, organization, and technologies were robust enough to support EBAO. Thus, no specific questions were designed to assess EBAO outcomes, but some questions were asked to assess decision-making such as:

Q1. In general, how would you rate the quality of decision making by your team? Q2. In general, in terms of outcome, how successful were the decisions made by your team? Q3. Rate your mental demand (includes thinking, remembering, searching, deciding)?

Again the vector method was used to generate the projection of measured vector onto the reference vector. Questions 1 and 2 were only given once while data from four and six dates were available for question 3 and so a correlation value was calculated. The results are given


in Tables F.9 and F.10. One would expect that Implicit Intent would be directly related to the first two questions, and inversely related with the third question.

TABLE F.9 COMPARING IMPLICIT INTENT AND DECISION-MAKING Q1 AND Q2 RESULTS

Date 13 Mar 14 Mar 14 Mar 16 Mar 16 Mar 16 Mar

Q1 Implicit Intent wrt Cmd Intent

Implicit Intent wrt Each Other

Q2 Implicit Intent wrt Cmd Intent

Implicit Intent wrt Each Other

NRF 67% 72% 69% 72% 75% 72%

CTF 66% 70% 66% 71% 70% 68%

TABLE F.10 CORRELATING IMPLICIT INTENT RESULTS WITH DECISION-MAKING Q3 RESULTS

Implicit Intent wrt Cmd Intent and Q3 Implicit Intent wrt each other and Q3 NRF (four dates) -0.92 -0.97 CTF (six dates) -0.42 -0.51

Finding 1. Nevertheless, the NRF answers are consistently higher than the CTF for both Implicit Intent and decision-making (although the decision-making differences are in the order of 1%). That is, Decision-making quality seems to be very similar across NRF and CTF, and not vary as widely as Implicit Intent. Not enough measures are available to make any definitive statement.

Finding 2. Implicit Intent and Mental Demand are inversely related as predicted by the model. That is as high levels of Implicit Intent are obtained, decision-making becomes easier.

3. Recommendations

It is recommended that a limited objective experiment be designed that would calibrate the Common Intent measurement in order to determine what minimum value of Common Intent is necessary to optimize team performance.

Common Intent along with Roles and Responsibilities, Decision-making, and Outcomes must be intentionally designed before the experiment in order to determine any correlations between these variables.

It is also recommended to use the Common Intent instrument as frequently as possible in military exercises and large-scale experiments. Because of its simplicity, it is easy to administer and seconds to complete. As data are gathered on this measurement, the confidence in its ability to accurately monitor the organization’s “health” increases.

4. Conclusions

A Team Information Processing model was developed based on Perceptual Control Theory, and a relationship between TIP and Common Intent was postulated. The model suggested that Explicit Intent would stabilize team interaction by having clear roles and responsibilities,


while Implicit Intent would optimize team interaction and this would be evident in the improvement of products and decisions. MNE 4 provided an excellent opportunity to measure Common Intent in a complex environment. The MNE 4 experimental results indicated that both Explicit Intent and Implicit Intent increased over the 3-week exercise indicating that it takes time to develop Common Intent. NRF Common Intent was consistently higher than CTF in all cases indicating that Common Intent may have a key role in the apparent success of EBAO conducted by the NRF headquarters. After measuring Common Intent in MNE 3 and MNE 4, the general impression is that a Common Intent value of 70-75% seems to be a minimum value required for a headquarters to move forward with some degree of success. Also, there were indications that Explicit Intent and roles and responsibilities were directly related to each other. Similarly, Implicit Intent and decision-making seem to be related as predicted by the model. However, there were insufficient data to make any definitive statements.

5. References

Farrell, P.S.E. (2005a). Calculating Effectiveness using Bi-Polar Scales and Vector Algebra. DRDC Toronto TR 2005-148, Toronto, Canada: Defence R&D Canada. 71 pp.

Farrell, P. S. E. (2005b). Measuring Common Intent During Effects Based Planning. DRDC Toronto TM 2005-150, Toronto, Canada: Defence R&D Canada, June 2005.

Farrell, P. S. E., P. Comeau, R. Grossman-Vermaas, L. Lichacz, R. Patrick, S. Villeneuve and K. Wheaton (2005). CFEC Analyst Final Report for Multi-National Experiment 3. DRDC Toronto TM 2005-149, Toronto, Canada: Defence R&D Canada, June 2005.

McCann, C., R. Pigeau, and A. English (2003). Analysing command challenges using the command and control framework: pilot study results. DRDC Toronto TR 2003-034, Toronto, Canada: Defence R&D Canada, February 2003.

Pigeau, R. and C. McCann (2000). The Human in Command: Exploring the Modern Military Experience, edited by McCann and Pigeau, Kluwer Academic/Plenum Publishers, New York, 2000. Chapter 12: Redefining Command and Control. pp.165 – 184.

Powers, W.T. (1973). Behavior: The Control of Perception. Hawtorne, New York: Aldine De Gruyter.


List of Symbols ADRM Action development and Resource Matching

AAMOP Actions Analysis by use of Measures of Performance

ANA Afghanistan National Army

AS Australia

C2 Command and Control

C&C Control and Coordinate Operations

CA Campaign Assessment

CA Canada

CAG Civilian Agency Group

CD&E Concept Development and Experimentation

CD-CIE Cross Domain Collaborative Information Environment

CF Canadian Forces

CFC-A Combined Forces Command - Afghanistan

CG Command Group

CFBLNet Combined Federated Battlelab Network

CFEC Canadian Forces Experimentation Centre

CFXnet Canadian Forces Experimentation Network

CI Common Intent

CIDA Canadian International Development Agency

CIE Collaborative Information Environment

CJTF Combined Joint Task Force

COI Critical Operational Issue

CONEMP Concept of Employment

CONOPS Concept of Operations

COP Common Operating Picture

COS Chief of Staff

CRC Communications Research Canada

CROP Common Relevant Operational Picture

CSA Continuous Systemic Analysis

CTF Coalition Task Force


CTFHQ Coalition Task Force Headquarters

DCEE Distributed Continuous Experimentation Environment

DFAIT Department of Foreign Affairs and International Trade

DIME Diplomacy, Information, Military, Economic

DISA Defense Information Systems Agency

DND Department of National Defence

EAMOP Effects Analysis by use of Measures of Effectiveness

EBA Effects Based Assessment

EBAP Effects Based Assessment Planning

EBAO Effects Based Approaches to Operations

EBE Effects Based Execution

EBO Effects Based Operations

EBP Effects Based Planning

ED Effects Development

ESA End State Analysis

EST Eastern Standard Time

FI Finland

FR France

GE, DE Germany

GMT Greenwich Mean Time

GRA Guidance and Requirements Analysis

HIG Hezb-e-Islami Gulbuddin

HQ Headquarters

IDI Identify Issues

ISAF International Security Assistance Force

IWS InfoWorkSpace

JFCOM Joint Forces Command

JIMP Joint, Inter-agency, Multinational, Public

KB Knowledge Base

KBD Knowledge Base Development

KM Knolwedge Management

KP Knowledge Processing


KR Knowledge Request

KS Knowledge Support

LNO Liaison Officer

LOE Limited Objective Experiment

Degree of belief

M&S Modelling and Simulation

MMA Measures of Effectiveness – Measures of Performance Analysis

MNE Multinational Experiment

MNIG Multinational Interagency Group

MNIS Multinational Information Sharing

MN KB Multinational Knowledge Base

NATO North Atlantic Treaty Organization

NDHQ National Defence Headquarters

NGA Non-government Agencies

NRF NATO Response Force

OGD Other Government Departments

ONA Operational Net Assessment

OPP Operational Planning Process

P Proposition

PMESII Political, Military, Economic, Social, Information, and Infrastructure

PRT Provincial Reconstruction Team

POT Process, Organization, and Technology

RG Red and Green Teaming

SCD Senior Concept Developer

SE Sweden

SJFHQ Standing Joint Force Headquarters

SME Subject Matter Expert

SoSA System of Systems Analysts

SPR Synchronization and Plan Refinement

START Stabilization and Reconstruction Task Force

TD Travel Duty

TTPs Tactics, Techniques, and Procedures


UK United Kingdom

UN United Nations

US United States of America

WebCOP Web Common Operating Picture

WESTT Workload, Error, Situation awareness, Time and Teamwork

Wmodel World Model

WSCC Wargaming, Simulation and Cultural Center


Glossary

Effects Based Operations An alternative means to pursue foreign and defence policy objectives that relies heavily on the injection of specific inter-agency, academic, corporate, diplomatic, economic and coalition intelligence knowledge, as well as technology to assist the decision maker in understanding the end-states, or, ‘effects’, required to attain strategic objectives.

Effects Based Planning Involves inter-agency coordination and assistance in developing the ONA, creating potential ‘effects’ and actions linkages, and pursuing actions based on capabilities

Operational Net Assessment Assessment and exploitation of network nodes and linkages between nodes of interest.


This page intentionally left blank

UNCLASSIFIED SECURITY CLASSIFICATION OF FORM

(highest classification of Title, Abstract, Keywords)

DOCUMENT CONTROL DATA (Security classification of title, body of abstract and indexing annotation must be entered when the overall document is classified)

1. ORIGINATOR (the name and address of the organization preparing the document. Organizations for whom the document was prepared, e.g. Establishment sponsoring a contractor’s report, or tasking agency, are entered in section 8.)

Defence R&D Canada – Ottawa Ottawa, ON, Canada, K1A 0Z4

2. SECURITY CLASSIFICATION (overall security classification of the document,

including special warning terms if applicable) UNCLASSIFIED

3. TITLE (the complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S,C or U) in parentheses after the title.)

Mult-National Experiment 4 on Effects-Based Approach to Operations; CFEC Analysis Report (U)

4. AUTHORS (Last name, first name, middle initial)

Philip S. E. Farrell, Dave Allen, Paul Burrows, Paul Comeau, Steven Hughes, John Kachuik, Paul Labbé, Fred Lichacz

5. DATE OF PUBLICATION (month and year of publication of document)

December 2006

6a. NO. OF PAGES (total containing information. Include Annexes, Appendices, etc.)

193

6b. NO. OF REFS (total cited in document)

75

7. DESCRIPTIVE NOTES (the category of the document, e.g. technical report, technical note or memorandum. If appropriate, enter the type of report, e.g. interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.)

Technical Report

8. SPONSORING ACTIVITY (the name of the department project office or laboratory sponsoring the research and development. Include the address.)

Canadian Forces Experimentation Centre Department of National Defence (Shirley's Bay) 101 Colonel-By Drive, Ottawa, ON, Canada, K1A 0K2

9a. PROJECT OR GRANT NO. (if appropriate, the applicable research and development project or grant number under which the document was written. Please specify whether project or grant)

20cq

9b. CONTRACT NO. (if appropriate, the applicable number under which the document was written)

10a. ORIGINATOR’S DOCUMENT NUMBER (the official document number by which the document is identified by the originating activity. This number must be unique to this document.)

DRDC Ottawa TR 2006-230

10b. OTHER DOCUMENT NOS. (Any other numbers which may be assigned this document either by the originator or by the sponsor)

CFEC TR 2006-001

11. DOCUMENT AVAILABILITY (any limitations on further dissemination of the document, other than those imposed by security classification) ( X ) Unlimited distribution ( ) Distribution limited to defence departments and defence contractors; further distribution only as approved ( ) Distribution limited to defence departments and Canadian defence contractors; further distribution only as approved ( ) Distribution limited to government departments and agencies; further distribution only as approved ( ) Distribution limited to defence departments; further distribution only as approved ( ) Other (please specify):

12. DOCUMENT ANNOUNCEMENT (any limitation to the bibliographic announcement of this document. This will normally correspond to

the Document Availability (11). However, where further distribution (beyond the audience specified in 11) is possible, a wider announcement audience may be selected.)

UNCLASSIFIED

SECURITY CLASSIFICATION OF FORM DDCCDD0033 22//0066//8877

UNCLASSIFIED SECURITY CLASSIFICATION OF FORM

13. ABSTRACT ( a brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the information in the paragraph (unless the document itself is unclassified) represented as (S), (C), or (U). It is not necessary to include here abstracts in both official languages unless the text is bilingual).

This report documents the Canadian analysis within Multinational Experiment 4 (MNE 4), and addresses the three Canadian Objectives that included the relevance of an Effects-Based Approach to Operations (EBAO), assessment of Multinational Interagency Group (MNIG) and Knowledge Support (KS) concepts for possible application to the Canadian Forces (CF) and other Canadian government departments. The report introduces the EBAO, MNIG, and KS concepts, provides an overview of the experimental design for MNE 4, and specifies the key Critical Operational Issues that were relevant to the Canadian Objectives. The results are summarized for each relevant COI. The discussion section addresses the Canadian Objectives in light of Canada’s International Policy Statement as well as MNE 4 assessments. The report concludes with a series of implications and recommendations for the CF. The main recommendation is for Canada to continue being engaged in EBAO research, concept development, and experimentation. It is critical that Canada keep abreast of the latest ideas and thoughts on this concept as our major allies (US and NATO) move toward adopting EBAO ideas. Moreover, it is now time to think about “Canadianizing” the EBAO concept such that it addresses Canada’s unique challenges and exploits unique opportunities.

14. KEYWORDS, DESCRIPTORS or IDENTIFIERS (technically meaningful terms or short phrases that characterize a document and could be helpful in cataloguing the document. They should be selected so that no security classification is required. Identifiers such as equipment model designation, trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus. e.g. Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurus-identified. If it is not possible to select indexing terms which are Unclassified, the classification of each should be indicated as with the title.)

Command and Control Concept Development and Experimentation (CD&E) Common Intent Collaborative Environments Continuous Systemic Analysis Coalition Task Force Culture Diplomacy, Information, Military, Economic (DIME) Effects-based Joint, Inter-agency, Multinational, Public (JIMP) Knowledge Base Knowledge Base Development Knowledge Management Operational Net Assessment Political, Military, Economic, Social, Information, and Infrastructure (PMESII) System of Systems

UNCLASSIFIED

SECURITY CLASSIFICATION OF FORM

Multi-National Experiment 4 on Effects Based Approach to ...

Documents

Transcript of Multi-National Experiment 4 on Effects Based Approach to ...