Post on 22-Sep-2020
Communication in Joint Activity
Investigating Teams’ Communication Pattern in a
Dynamic Decision Making Environment
Master’s thesis 30hp
Nicoletta Baroutsi
2014-07-20
ISRN LIU-IDA/KOGVET-A--14/012--SE
Linköpings Universitet Institutionen för Datavetenskap
Communication in Joint Activity Investigating Teams’ Communication Pattern in a Dynamic Decision
Making Environment
Master’s thesis 30hp
Nicoletta Baroutsi
2014-07-20
ISRN LIU-IDA/KOGVET-A--14/012--SE
I
Acknowledgements
I want to take this opportunity to thank all the wonderful people who have
supported me throughout this project. First I want to extend a grand thank you
to my tutors at FOI, Peter Berggren and Björn Johansson, without whom this
would not have been possible. Thank you for all the support and advice, but
also for making this project fun and enjoyable. I also want to thank Christopher
Palm, a dear friend that shared this experience of working at FOI with me, and
who continuously discussed and dwelled the problems I encountered with me.
Not to forget all the people working at FOI who made me feel so welcome, it
was always a good feeling to show up at work. And of course, a big thank you
to all my friends and family that always listened to me when I was rambling on
about my thoughts and concerns, even when they had no idea what I was
talking about.
II
Sammanfattning
I en värld av ständigt ökande komplexitet, som karaktäriseras av ofullständig
information och dynamiska, tidskritiska miljöer, strävar människor efter att fatta
rätt beslut – inte som individer – utan även som ett team. I denna gemensamma
aktivitet behöver medlemmarna synkronisera sina handlingar, vilket utförs med
hjälp av kommunikation. Kommunikationen är den dominerande formen av
interaktion inom ett team, och är även en externalisering av teamets kognitiva
processer (Letsky, Warner, Fiore & Smith, 2008).
I en tidigare studie har oerfarna deltagare tränats i team om tre, för att bli
högpresterande inom mikrovärlden C3Fire (Baroutsi, Berggren, Nählinder och
Johansson, 2013). I denna mikrovärld står teammedlemmarna inför ett
dynamiskt beslutsproblem - att bekämpa en skogsbrand. Rollerna i teamet är
ömsesidigt beroende av varandra, vilket kräver att de samordnar och lägger upp
strategier på en teamnivå för att på ett framgångsrikt sätt kunna lösa uppgiften.
Dessa sex tränade team jämfördes sedan med sex otränade team i ett
experiment. Flera mått användes för att bedöma teamen (CARS, DATMA,
Shared Priorities, m.fl.), vilket visade att de tränade teamen skilde sig både
avseende prestation, men även inom andra viktiga teamaspekter (Baroutsi,
Berggren, Johansson, Nählinder, Granlund, Turcotte, & Tremblay, 2014;
Berggren, Baroutsi, Johansson, Turcotte, & Tremblay, 2014; Berggren,
Johansson, Baroutsi, & Dahlbäck, 2014; Berggren, Johansson, Svensson,
Baroutsi, & Dahlbäck, 2014; Baroutsi, Berggren, Johansson, manuskript).
Syftet med denna rapport är att undersöka hur kommunikationsmönstret
påverkas av dessa skillnader.
Kommunikationen analyserades med hjälp av ett kodningsschema där
innehållet i teamens uttalanden kategoriseras. De två olika typerna av team
uppvisade ingen skillnad i antalet uttalanden, men skillnader fanns för olika
kommunikationskategorier. De tränade teamen kommunicerade oftare angående
sammanhanget och situationen, medan de otränade teamen oftare
kommunicerade om de aktiviteter som pågick. Detta kan tolkas som en brist i
den gemensamma förståelsen, styrbarheten och förutsägbarheten mellan
teamets medlemmar (Klein, Feltovich & Bradshaw, 2005) hos de otränade
teamen. Kommunikationsinnehållet förklarade 88,3 % av variationen i
prestationen.
III
Summary
The complexity in the world is continuously increasing. Teams are faced with
imperfect information in uncertain, dynamic, and time critical environments as
they strive to make the right decisions, not just as individuals, but as a team. In
this joint activity the members choreograph their actions and synchronize their
behavior through the use of communication. Communication is the predominant
form of interaction within teams – it is not only a window into team cognition –
it is an externalized cognitive process at a team level (Letsky, Warner, Fiore &
Smith, 2008).
In an earlier study, non-professional participants were trained in teams of three
to become high-performing within the C3Fire microworld (Baroutsi, Berggren,
Nählinder and Johansson, 2013). In this microworld the team members are
faced with the dynamic decision problem of fighting a forest fire. They have
interdependent roles, requiring them to coordinate and strategize on a team
level, making C3Fire a suitable platform for investigating dynamic decision
making in teams. These six trained teams were compared to six untrained teams
in a final experiment through a variety of measures, showing that the trained
teams differed significantly in terms of both performance and in other important
team aspects (Baroutsi, Berggren, Johansson, Nählinder, Granlund, Turcotte, &
Tremblay, 2014; Berggren, Baroutsi, Johansson, Turcotte, & Tremblay, 2014;
Berggren, Johansson, Baroutsi, & Dahlbäck, 2014; Berggren, Johansson,
Svensson, Baroutsi, & Dahlbäck, 2014; Baroutsi, Berggren, Johansson,
manuscript). These differences were thought to have an impact on the
communication shared among the team members. Hence, the purpose of the
present report was to investigate how the communication pattern was affected
by these differences.
The communication was analyzed using a coding scheme that categorized the
content of the teams’ utterances. No difference was found in terms of
communication frequency between the two types of teams. However, the
trained and untrained teams did differ in communication content. The trained
teams communicated more frequently about the context and the situation, while
the untrained teams communicated more about the activities of the team. This
can be interpreted as a deficiency in common ground, directability, and
interpredictability (Klein, Feltovich & Bradshaw, 2005) among the untrained
teams. Also, the communication content explained 88.3 % of the variance in
performance.
IV
Table of Contents
1 Introduction 1
1.1 Background ....................................................................................... 2
1.2 Research questions .......................................................................... 3
2 Theoretical background 4
2.1 Common Ground in Joint Activity ...................................................... 4
2.1.1 Requirements for Joint Activity ..................................................... 5 2.1.2 Criteria for Joint Activity ................................................................ 6 2.1.3 Choreography of Joint Activity ...................................................... 7
2.2 Team communication and coordination ............................................ 8
2.2.1 Communication frequency ............................................................ 9 2.2.2 Closed-loop communication .......................................................... 9 2.2.3 Error detection ............................................................................... 9 2.2.4 Team training .............................................................................. 10
2.3 Team effectiveness ......................................................................... 10
2.4 Dynamic, time-critical, high stake situations ................................... 11
2.5 Dynamic Decision Making ............................................................... 12
2.6 Microworlds ..................................................................................... 14
2.6.1 From microworlds to reality ......................................................... 15 2.6.2 C3Fire.......................................................................................... 15
2.7 Synthesis ......................................................................................... 16
3 Method 18
3.1 Participants ...................................................................................... 18
3.2 Experimental design ........................................................................ 18
3.2.1 Sensor range ............................................................................... 19 3.2.2 Role configurations ..................................................................... 19 3.2.3 Scenario ...................................................................................... 20 3.2.4 Script commentaries ................................................................... 21
3.3 Dependent measures ...................................................................... 23
3.3.1 Simulation performance .............................................................. 23
V
3.3.2 Communication ........................................................................... 23 3.3.3 Level of transcription ................................................................... 24 3.3.4 Coding procedure ........................................................................ 24
3.4 Procedure ........................................................................................ 26
3.4.1 Preparing the untrained teams .................................................... 27 3.4.2 Session procedure ...................................................................... 28
3.5 Apparatus ........................................................................................ 28
4 Results 29
4.1 Coding scheme reliability ................................................................ 29
4.2 Simulation performance .................................................................. 32
4.3 Communication ............................................................................... 33
4.3.1 Team type comparison................................................................ 34 4.3.2 Sensor range comparison ........................................................... 36 4.3.3 Communication frequency and performance .............................. 38 4.3.4 Communication pattern as a predictor of performance ............... 38
4.4 Summary ......................................................................................... 39
4.4.1 Trained and untrained teams ...................................................... 39 4.4.2 Full view and limited view sensor range ..................................... 40 4.4.3 Communication and it’s relation to performance ........................ 40
5 Discussion 42
5.1 Results discussion .......................................................................... 42
5.1.1 Communication patterns and team type ..................................... 42 5.1.2 Communication pattern and visual conditions ............................ 43 5.1.3 Communication frequency and performance .............................. 44 5.1.4 Communication content and performance .................................. 44
5.2 Method discussion .......................................................................... 45
5.2.1 Transcriptions and the coding scheme ....................................... 45 5.2.2 Reliability of the coding scheme ................................................. 45 5.2.3 Adopting a grammatical approach to diminish ambiguity ........... 46 5.2.4 Strategies and planning .............................................................. 47 5.2.5 Reaching consensus ................................................................... 49 5.2.6 Summary of the proposed coding scheme ................................. 51
6 Conclusions 54
6.1 Results ............................................................................................ 54
VI
6.2 Method ............................................................................................ 55
6.3 Future research ............................................................................... 56
7 References 57
Appendix 1 62
1
1 Introduction
In a world of increasing complexity, the central role of teams becomes
progressively more important. However, all teams are not efficient by nature,
only some teams are able to grasp the unpredictable world around them, and for
the team members to do so in symphony with each other. Understanding what
makes a team successful is thusly highly valuable. It is not satisfying enough to
only measure team outcome – i.e. performance – since many factors not related
to the team may be influencing the outcome of a situation. Besides, a lack of
feedback on the effect of the team’s actions may also make it impossible to
accurately measure performance.
Real world situations offer imperfect information in uncertain, dynamic and
time critical environments (Klein, Orasanu, Calderwood & Zsambok, 1993).
Dynamic decision making takes place as events are unfolding, requiring the
decision maker to make sense of a world that changes, not only as a result of
their actions, but spontaneous as a consequence of time (Brehmer, 2000). As a
team, the members also strive to make the right decisions as a team, not just as
individuals. Team cognition (Cooke, Gorman & Winner, in press) is profoundly
different from individual cognition in many ways. Communication is the
predominant form of interaction within teams – it is not only a window into
team cognition – it is an externalized cognitive process at a team level (Letsky,
Warner, Fiore & Smith, 2008). In a joint activity the control and coordination of
team members actions are choreographed through the use of communication
(Klein et al., 2005). It allows the participants to transpose through the phases of
the activities together, as they continuously recognize each other’s signals. “If
we take language use to include such communicative acts such as eye gaze,
iconic gesture, pointing, smiles, and head nods – and we must – then all joint
activities rely on language use” (Clark, 1996, p. 58).
The purpose of the study is to deeper investigate team communication in an
effort to find what type of content it is that relates to successful teams, content
that signifies proper dynamic decision making among team members. A coding
scheme originally developed by Svenmarck & Brehmer (1991), and later
modified by Johansson, Trnka, Granlund & Götmar (2010), will be the tool
used to code the communication. The chosen platform for the experiment is a
microworld called C3Fire. C3Fire faces the participants with the task of
fighting a forest fire. The decision problem is dynamic, complex and time
critical, making it a suitable choice for studying dynamic decision making.
2
1.1 Background
The Swedish Defense Research Agency (FOI) has an interest in dynamic
decision making for teams operating in complex and uncertain environments.
Within this framework, a series of studies have been conducted in order to
investigate important team aspects, team development and team
communication. This was carried out within the Swedish Armed Forces
research and development (R&D) project AVALO at FOI.
One experiment was conducted that has led to three separate studies, including
the current one. For the first study, six three-person teams of non-professionals
were put through ten sessions of training (Baroutsi, Berggren, Nählinder and
Johansson, 2013). The purpose was to train them to collaborate as cohesive
units, and to investigate whether validated measures within the domain could
measure their progress. The teams’ progress was assessed using various types
of measures, including measures of performance, tactical performance, situation
awareness and mutual awareness. Considerations taken into account were that
the team structure should be decentralized, and that coordination and
communication needed to be a central part of the teams’ behavior in order for
them to be successful. These considerations were important for the second
study where trained teams were compared to untrained teams (Baroutsi,
Berggren, Johansson, manuscript). During this second study (Baroutsi, et al.,
manuscript) a new measure called Shared Priorities was validated, and also a
new measure called Content Analysis emerged during the process.
This leads up to the third and current study. The communication was recorded
during the experiment between the trained and untrained teams, but never
analyzed. These trained teams have been monitored through their training and
compared to the untrained teams using a variety of measures, including
simulation performance, shared situational awareness, mutual awareness,
shared priorities, content analysis and tactical performance (Baroutsi et al.,
manuscript). These earlier studies suggest that important skills are developed
within the trained teams, skills not found in the untrained teams. These
differences should have an impact on the communication shared among the
team members, thus being available for further investigation. Hence, the
purpose of this study is to analyze this untapped source of information using the
adapted coding scheme from Johansson et al. (2010).
3
1.2 Research questions
Q1. How does the communication pattern differ between trained and
untrained teams?
Q2. How does the communication pattern change during diverse visual
conditions?
Q3. Can a relationship be established between the communication
frequency and performance of the teams?
Q4. Is it possible to predict performance via the communication content?
Q5. What are the limitations of the coding scheme?
4
2 Theoretical background
The scientific studies on team cognition took off during the late 80’s. New
theories emerged as the scientific focus shifted from the individual towards the
team (Cooke et al., in press), theories that could help explain the unique
behaviors’ observed in team interactions. A team is defined by Salas,
Dickinson, Converse & Tannenbaum (1992, p. 4) as “a distinguishable set of
two or more people who interact dynamically, interdependently, and adaptively
toward a common and valued goal/object/mission, who have each been
assigned specific roles or functions to perform, and who have a limited life span
of membership”.
This need to interact interdependently and adaptively toward a common goal
sets certain prerequisites on a team. These prerequisites will be the first theories
presented in this chapter. They relate to team coordination and communication,
and will be discussed in terms of common ground and joint activity (Klein et
al., 2005). Once the foundation has been discussed, the theoretical background
leads into a presentation of relevant research findings concerning teams. This
includes patterns of communications, and the benefits of conducting team
training as opposed to individual skill training. This is followed by a definition
of team effectiveness, that section describes what it means for a team to be
successful.
The next section covers the problem characterization, starting with a description
of the environment in which teams operate: dynamic, time-critical, high stake
situations (Klein et al., 1993). This environment has direct consequences on the
problems the decision maker encounters and the actions that follows. These
implications are discussed in the concept of dynamic decision making
(Brehmer, 2000), leading to a general description of microworlds and C3Fire –
the microworld used in the current experiment.
Conclusively is a synthesis relating the theoretical findings, and its
implications, to the current study.
2.1 Common Ground in Joint Activity
The coordination among team members in high performing teams can to an
outsider be seen as minimalistic and ambiguous. Building on the work of Clark
(1996), Klein, et al. (2005) interprets the ideas of common ground and joint
5
activity into the team domain. They describe the relationship between common
ground and joint activity, two closely related concepts that explain how the
observed coordination among team members is possible. This chapter is a
description of their interpretations.
Clark (1996) defines a joint activity as a set of coordinated behaviors carried
out by two or more people. The following three sections will describe the
requirements, criteria, and choreography of joint activity (see Figure 1).
Figure 1. Joint activity and its key aspects. Adapted from Klein et al. (2005).
2.1.1 Requirements for Joint Activity
Three primary requirements for achieving effective coordination in joint
activities have been found to cut across domains: sufficient common ground,
interpredictability between team members, and directability of team members.
Common ground is what makes joint activity and coordination possible. The
concept of common ground includes relevant mutual beliefs, knowledge, and
assumptions that support the interdependent actions of a team. It is a process of
continuous communication, testing, updating, and repairing of faulty
assumptions. A team that maintains a sufficient common ground will allow for
abbreviated forms of communication, i.e. it allows for ambiguous signals to be
correctly interpreted. A review on coordination in various forms of teams found
central types of common ground to be important (Klein et al., 2005):
The different roles and their related functions
Routines manageable by the team
6
Skills and competencies
Participants goals, including the commitment to the team activity
The stance of each participant, e.g. their individual perception of time
pressure, and competing priorities
Common ground is not a need for the participants to think identically. The act
of aligning members’ different perspectives to increase common ground may
even result in added effort. Nevertheless, diverse perspectives and the
acknowledgement of them may actually improve team performance (Spiro,
Feltovich, Coulson & Feltovich, 1989, in Klein et al., 2005). Teams maintain a
sufficient common ground through activities such as: structuring preparations,
insertion of clarifications and reminders, monitoring of other’s activities,
detecting and signaling anomalies, and correcting faulty assumptions.
Interpredectability is the ability to coordinate and predict each other’s’ actions.
To permit this interpredictability each team member has to make his or her
actions sufficiently predictable. Many features of a real world situation require
a team to possess this quality: the time needed to complete an action, the
physical location of certain objects, and the difficulty to complete an action.
There are many factors that contribute to the understanding and handling of a
situation, including the roles and functions held by each member. Hence, the
ability to predict each other’s actions is greatly enhanced in teams where they
are able to envision the perspective of their team members.
Directability is the ability to deliberately redirect the actions of the other team
members as the conditions changes. It has been identified as an important
aspect of coordination, because it enhances the team’s resilience (Christoffersen
& Woods, 2002).
2.1.2 Criteria for Joint Activity
For an activity to be considered a joint activity there has to be an intention to
work together, i.e. the basic compact, and the work has to be interdependent.
“It’s not cooperation if either you do it all or I do it all”
Woods 2002, in Klein et al., (2005, p. 6).
The basic compact states that the participants need to comply with an
agreement (usually tacit), and to carry out the required coordination
7
responsibilities in order to participate in a joint activity. This agreement
involves goal alignments, which entails relaxing of short-term goals in order to
allow for more global, long-term goals to be fulfilled. Another aspect of the
basic compact is the detection and correction of losses in common ground that
might impede the joint activity (Klein et al., 2005).
A joint activity requires interdependence: the activities of the different parties
must in some significant way rely on each other. Two musicians playing the
same musical piece at different locations are not involved in a joint activity.
The same goes for parallel or synchronized activities: two police officers
working in shifts with a synchronized schedule are not participating in a joint
activity. The interdependent aspect thusly puts emphasis on the interaction and
interweaving of the participant’s actions (Klein et al., 2005).
2.1.3 Choreography of Joint Activity
“Each small phase of coordination is a joint action, and the overall composite
of these is the joint activity” (Clark, 1996). A couple waltzing is involved in the
joint activity of dancing, each sequence being a joint action. As they sweep the
dance floor they must continuously recognize the cues of their dance partner,
the changes in posture and body pressure are all signals on events unfolding in
the near future. The choreography of the joint activity centers around different
phases, it is influenced by the signals expressed, and the coordination devices.
The burden of choreographing the efforts are referred to as coordination costs
(Klein et al., 2005).
What really gets coordinated during activity are the phases, and the
coordination is accomplished one phase at a time. A phase is a joint action
consisting of three constituents: an entry, a body of action, and an exit. No clear
demarcations marks the phases constituents, this is all to the parties themselves
to decide. It may be difficult to coordinate the exiting of a phase, and for this
the parties are in need of evidence on whether the exiting was performed
successfully. For example, when you push the button in an elevator you wish to
see an indication that the push is registered, e.g. the light goes on. Similarly, the
joint activity is in need of a joint closure. An engaged listener acknowledges the
reception of the information by nodding the head or making paraphrases. To
successfully synchronize entry and exit points of the numerous embedded
phases in a complex joint activity can prove to be a major challenge (Klein et
al., 2005).
8
Signals are the tools team members use to inform each other about transitions
between and within phases. Signals are also used in broader terms, covering
everything from intentions, difficulties, desires, and so forth. Attention is a
limited resource, but this resource can be redirected very quickly. Signaling is
only successful if the receiver notices the signal. Thus, in the choreography of a
joint activity it becomes relevant to use signals to direct the other team
members attention to relevant cues. For example, a team member incorrectly
believes that a phase has been completed and tries to exit. It is now up to the
other team members to signal and redirect the attention of that individual to
relevant cues, and thereby helping to correct the faulty assumption (Klein et al.,
2005).
Coordination devices are used to shape the choreography of joint activities.
Coupled with common ground, these signals increase the interpredictability
within the team (Klein et al., 2005). Examples of coordination devices are:
Agreement: explicitly communicated intentions, signs and gestures.
Convention: how parties interact based on prescriptions of various
types and degrees of authority, as well as norms.
Precedent: norms and expectations developed during the current joint
activity.
Salience: how the workspace is arranged so that the next move
becomes prominent.
2.2 Team communication and coordination
Communication can generally be defined as the information exchange between
two or more individuals, through any type of medium (McIntyre & Salas, 1995,
quoted in Salas 2005). A team needs to be able to coordinate and communicate
effectively in order to perform successfully. The ability of control depends on
the ability of the individuals to coordinate their actions, and the usual way of
achieving this coordination is through communication (Johansson, 2005).
Researchers have even been able to predict a team’s performance by looking at
their communication pattern, without knowing anything about the members of
the team (Pentland, 2012). Studies have examined explicit aspects of
communication, ranging from timing and frequency to accuracy and patterns.
9
Aspects thought relevant to the current study are communication frequency,
closed-loop communication, and error detection. These will be discussed
further, and followed by information on how team training effects
communication and coordination.
2.2.1 Communication frequency
Communication frequency was found to correlate positively with performance
when conducting an experiment involving fighter pilots (Svensson. 2002).
Similarly Obermayer & Vreuls (1974) found a correlation between
communication frequency and acquisition skill, i.e. experienced teams
communicated more frequently than inexperienced teams during weapons
delivery in air force training. However, the opposite was found during routine
tasks, where the inexperienced teams communicated more frequently than the
experienced teams. According to Salas (2005) it appears that teams over time
develop a vocabulary that reduces the lengths of the messages, resulting in a
reduction of communication.
2.2.2 Closed-loop communication
Closed-loop communication is a technique that enables teams to avoid
misunderstandings. This communication includes three steps: First the sender
communicates a message to a team member. The receiver interprets and
acknowledges the message, which means that the receiver repeats or
paraphrases the message. The sender then reassures that the message was
received as intended, commonly just by answering “yes”, or correct the
message if needed (Salas, 2005). The effect of closed-loop communication has
been investigated and more successful teams reassures the accurate information
exchange using three steps, while less successful teams seldom communicate
through more than two (Lindgren, Hirsch, & Berggren, 2006).
2.2.3 Error detection
Many factors may hinder the communication, the same message may be
interpreted differently because of an individual’s own perspective bias, and
members may also be less willing to share information if they feel that it is not
valued or used appropriately (Bandow, 2001). During team decision making,
teams differ in the frequency in which they consider contributions from their
10
team members. For teams with a more frequent consideration of opposing
views, the input was not always accepted. Still, the consideration proved to lead
to higher error detection, which in turn resulted in higher quality decisions
(Driskell and Salas, 1992, in Salas, 2005).
2.2.4 Team training
Team training is an essential part of the development of any team, since it
creates a common understanding of the situation. There are many ways to
complete a task, and numerous ways to coordinate resources. These skills are
manifested at a team level and cannot be taught individually. Through time,
teams become increasingly proficient as they learn how to work together, and
they become increasingly similar in their perceptions (Morgan, Glickman &
Woodard, 1986). Team members sharing information about the nature of each
other’s subtasks emphasizes the requirement for communication and
coordination, which will enhance the team performance (Krumm, 1958).
Training team coordination should also help teams to identify interdependencies
between different roles and the undesirable consequences that will follow if the
team fails to coordinate their efforts and resources accurately (George, 1979, in
Swezey & Salas, 1992).
2.3 Team effectiveness
Teams have been acknowledged for their strengths in comparison to single
individuals. They have been attributed the potential to offer greater
productivity, adaptability and creativity, while providing solutions found to be
more innovative, complex and comprehensive (Amabile & Fisher, 2009;
Gladstein, 1984 in Salas 2005). Successful as they potentially may be, their
failures have also proven to be widespread with far-reaching effects (Larrick,
2009). Researchers have been able to suggest that team effectiveness is
mediated through team processes (Hackman & Ruth, 2009). Team effectiveness
is distinct from team performance by means of adopting a more holistic
approach. Team performance only accounts for the outcome of the team’s
actions, e.g. completion of task. Team effectiveness however, also includes how
they might have accomplished the task, covering factors concerning team
interactions, i.e. team processes, communication, coordination, learning (Salas,
2005, Hackman & Ruth, 2009). This distinction is important since many
11
factors, not depending on the team’s influence, may have contributed to the
success, or failure, of an assignment.
2.4 Dynamic, time-critical, high stake
situations
The world around us in which we operate is an ever-changing entity. It is in this
dynamic, naturalistic situation in which dynamic decision-making unfolds.
Klein et al. (1993) defines natural decision settings by the following eight
characteristics:
1. Ill structured problems: Real world problems are rarely clean cut.
The decision maker will have to generate ideas about what is actually
happening, what options are available and what the appropriate
responses are. Complex causal links relate to each other, causes
interact, feedback loops intertwine and so on. There is typically not one
accepted procedure, and it is necessary to make a selection or invent
new ways to proceed.
2. Uncertain dynamic environments: An uncertain environment is an
incomplete world with imperfect information. Some of the information
is available to the decision maker (e.g. the status of the firefighter,
number of resources available), while other information is unavailable,
ambiguous, or of poor quality (e.g. the extent of the current fire, the
location of team members’ units). The environment is dynamic – the
conditions laying the foundation for the decision might change rapidly
– even within the time frame of the necessary decision.
3. Shifting, ill-defined, or competing goals: Well-understood goals are
rare outside of the laboratory setting, usually the decision maker is
driven by multiple goals, some opposing each other. A fire chief wants
to save a burning building, but at the same time keep his crew out of
harm’s way. The development of the fire may shift the fire chief’s
goals; saving property loses priority as lives are at stake. Usually the
larger goals direct the smaller decisions.
4. Action/ feedback loops: Series of actions stretching over time are
usually needed to deal with complex problems, developing over series
of events. It is not a matter of hording information until a valid
12
decision can be made, and early opportunities and mistakes will have
an effect on later events and decision. However, the cause and effect
relationship may only be loosely coupled, making it difficult to derive
back to the origin.
5. Time pressure: Correct decisions have to be performed during the
right time in order to achieve the desired result. Sometimes action is
needed within only minutes or seconds. High time-pressure will often
exert high level of stress in the decision maker, potentially leading to
exhaustion or loss of vigilance. Characteristically their thinking will
shift into simpler reasoning strategies as the time pressure increases.
Extensive evaluations of multiple options are simply not feasibly, only
a few options are evaluated in a non-exhaustive manner before making
the decision.
6. High stakes: Plenty of everyday decisions are made where stakes are
not high at all, these situation are not the ones of interest. The concern
lies within the cases where stakes are high, cases that matters to
participants, situations that are likely to make them feel stressed and
involved, persuading them to take an active role.
7. Multiple players: Many problems involve not a single actor, but
several decision makers who are actively involved. The team may
include hierarchical command structures, including the roles of
decision makers and subordinates. It may also be a flat command
structure where multiple individuals may act together as a single
decision maker, or behave as competitors.
8. Organizational goals and norms: As discretely indicated, dynamic
decision making usually situate within an organizational context. The
organization carries goals and norms that does not coincide with the
individuals personal preferences. It is hard to incorporate these factors
into artificial environments (Klein et al., 1993, pp. 7-10).
2.5 Dynamic Decision Making
When interacting with a dynamic, time-critical, high stake situation the decision
maker is faced with a dynamic decision problem. Brehmer (2000, pp. 233-238)
defined a list of properties fundamental for dynamic decision making:
13
Requires a series of decisions
The decisions are interdependent
The environment changes, both as an effect of the decision makers
actions, and spontaneous system changes as a consequence of time
Temporal constraints (Brehmer 2000, pp. 233-238)
As can be seen, many of these properties are direct reflections concerning the
characteristics of the before mentioned and defined dynamic, naturalistic
situation. The first three properties are closely intertwined. For example, when a
fire chief is facing the problem of a forest fire, he/she first has to make an initial
decision concerning how many resources to send in. If he would send all
available resources, then he would be left without any resources available if a
second fire strikes. The fire is now spreading as a consequence to how many
resources he decided to initially assign to the mission. But the environment also
changes as a consequence of variables he has no control over, e.g. the strength
and direction of the wind, type of vegetation, and so on. Information concerning
the situation is (hopefully) reported back, and he now has to make new
decisions regarding what to do, decisions that are highly dependent on past
decisions and spontaneous system changes.
All that is happening is restricted by temporal constrains. When faced with a
dynamic decision problem, the decision maker will lack control over the time
that the decisions have to be made. Decisions are made when required, not
when the decision maker feels satisfied with the brought up solution. Two
separate kinds of problems arise in these types of situations. First is the
handling of the “core task”, to exert control over relevant aspects of the
environment (in this case the fire). Second is the managing of the overall
decision situation, so as to remain capable of making the proper core decisions.
This involves gathering information, evaluate the options, and so on. This is
only possible if the Fire Chief comes up with a strategy allowing him to think
through and consider available options before they are executed (Brehmer,
2000).
The fire is a temporal process, the controlled process, and the mechanism the
Fire Chief constitutes another temporal process, the controlling process. The
tactical opportunities depend on the relationship between these two processes.
In the scenario depicted in Figure 2, the fire is growing at a steady pace in a
homogenous terrain without any wind present. The slopes represent the
14
efficiency of the two processes. As long as the firefighting process is more
efficient than the process of the fire, the situation is under control and the
attempted strategy is still valid. Hence, certain strategies work better under
certain circumstances. In the real world decision makers will not get immediate
feedback on their actions, thus the dynamic decision making process will
involve coping with delays (Brehmer, 2000).
Figure 2. A simplified case of a firefighting scenario. As long as the controlling process is
higher than the controlled process the fire can be sustained, meaning that it spreads in a
rate manageable for the firefighters. After the intersection has been crossed, a new
strategy is needed to control the situation. (Adapted from Brehmer, 2000)
2.6 Microworlds
A microworld is a computer simulation of a realistic event that can be used to
study complex systems in a controlled way. Researchers suggest that using
microworlds becomes a way of bridging the gap between controlled
experiments conducted in a laboratory, and field studies in the real world. Field
studies often lack the control needed when conducting science, while laboratory
studies instead holds so many variables constant that it is difficult to generalize
the findings outside of the laboratory setting. This is a well-known trade-off
between the external and internal validity. In the real world there is too much
complexity and in the laboratory there is not enough. Microworlds might not be
a perfect solution but it is minimizing the gap in between the two extremes.
They are not designed to be exact representations of the real world. Their
purpose is to present a recognizable problem, complex enough for the subject to
experience uncertainty in a dynamic situation, but yet simple enough to allow
15
for a closer analysis. Microworlds are complex, dynamic and opaque thus
allowing the researcher to present the subjects with conflicting goals with
numerous response alternatives in real time, all this and still providing stable
and replicable results (Brehmer & Dörner 1993).
2.6.1 From microworlds to reality
Generalizations from microworlds should, like all experiments, be taken with
caution and be built upon a theoretical underpinning. Resemblance between the
microworld and the target situation does not create a valid criterion for
automatic generalization. A microworld is still a simplification of the real
world, but the point has never been for a microworld to be a replica of the world
around us. The best explanation of this is the ‘cat problem’, stating that the best
simulation of a cat would be another cat. The problem is that by creating a
replica none of the complexity is reduced and it would be just as hard to
understand as using the original cat. The same goes for microworlds, they are
not meant to be replicas of the world, they are simplifications of the world
designed to allow the researcher to observe what he intends to (Brehmer, 2004).
2.6.2 C3Fire
C3Fire is a microworld that can either be used by an individual, or by a group
of people collaborating, with the goal to extinguish a forest fire (Granlund et al.
2001; Granlund 2002). It has earlier been used in a variety of experiments
(Lindgren & Smith 2006; Johansson et al. 2010; Tremblay, Vachon, Lafond &
Kramer, 2011; Persson & Rigas, 2014). C3 stands for command, control and
communication. It is a simulation where collaboration can be investigated in a
controlled way. The collaboration can be supported by different means of
communication, and it is possible to configure the simulation so that a
dependency is created between the different members. One of the strengths of
C3Fire is the flexibility that it provides: different roles can be created for the
team members, diverse kinds of terrain, customized graphical user interface and
a large variety of scenarios. The interactions with the agents are conducted
through a geographical information system, which is an interactive electronic
map (see Figure 3). The interactive map consists of a number of cells that all
contain different properties. A cell can consist of an array of items, for example:
specific types of trees can make the fire burn faster or slower in a specific
direction, a water pump where the players can refuel, or valuable properties
16
such as houses or schools. The user interface also contains information
regarding the status of the vehicles, a mail viewer, and an anemometer. Each
user controls a number of vehicles that are displayed as colored numbers in the
interface, and it is through these vehicles that they affect the outcome in the
simulation.
All events in C3Fire are saved onto log files. The software records and
produces a variety of measures later available for analysis, including the
amounts of burnt cells, and the time it takes for the participants to extinguish
the fire. In addition, the movement of all the units, and all the messages are
logged and available for further investigation.
Figure 3. The C3Fire user interface used within this study.
2.7 Synthesis
By the theoretical underpinning of joint activity a framework is set, through
which the teams can be understood on a deeper level. The criterion of intention
is already met through the voluntary participation in the study, and the
interdependence can be created through the design of the roles within the
17
microworld, see 3.2.2. However, the requirements will have to be fulfilled by
the teams in action if they are to successfully participate in, and complete their
joint activity. Klein et al. (2005) refers to communication mainly as signals
used by the team members to inform each other about transitions between
phases in a joint activity. It is the tool the members employ to coordinate their
actions.
The importance of coordination and communication have been pointed out
extensively within the research, also relating it to team performance and
proficiency (Obermayer & Vreuls 1974; Morgan, Glickman & Woodard, 1986;
Svensson. 2002; Johansson 2005; Lindgren, Hirsch, and Berggren, 2006;
Pentland, 2012). Certain aspects of team communication were emphasized since
they are relevant to the current study and can relate to the analysis of the results:
communication frequency, closed-loop communication, and error detection.
In this study, trained teams are compared with untrained teams. Through time
the trained teams should have developed similar perception, identified
interdependencies between the roles, and enhanced their performance as they
learn how to work together (Krumm, 1958; George, 1979 in Swezey & Salas,
1992; Morgan, Glickman & Woodard, 1986 ). Hence, they should be more
capable at fulfilling the requirements for joint activities, and also more efficient
at managing a satisfactory level of common ground at lower coordination costs
than the untrained teams.
Furthermore, important characteristics of a dynamic, naturalistic situation have
been defined. The real world contains ill structured problems, uncertain
dynamic environments, and shifting ill-defined goals where multiple players
interact during high time pressure (Klein et al., 1993). It is in the interaction
with these environments that decision makers find themselves to be faced with
a dynamic decision problem. Dynamic decision problems are composed of four
fundamental properties: they require a series of decisions, decisions that are
interdependent, the environment changes both spontaneous and because of the
actions of the decision maker, and are under temporal constraints (Brehmer,
2000). In C3Fire multiple players are faced with these types of dynamic
decision problems, making this microworld a suitable platform for investigating
behaviors’ related to dynamic decision making.
18
3 Method
Six teams were initially put through extensive training before the actual
experiment. Each team consisted of three members, and the members of the
teams never changed. The training consisted of ten training sessions. Session 1,
4, 7 and 10 were held constant, i.e. they had the same map and scenario
configuration, and could therefore be used for measuring the team’s progress.
The purpose of the training was for the participants to develop as a team and
become experienced within the C3Fire domain.
For the experiment the six trained teams were compared to six untrained teams.
The communication that was analyzed in this report is extracted from the
experiment, see Figure 4.
Figure 4. Layout of the experiment. Numbered tiles represents the training sessions
undergone by the trained teams, yellow tiles marks sessions where their progress was
measured. The ‘F’-tile is the experiment where the trained teams were compared to the
untrained teams.
3.1 Participants
Twelve teams of non-professionals, with three members in each team,
participated in the experiment, yielding six trained and six untrained teams.
There were 28 men and 8 women. The mean age of the participants were 28.9
years (SD = 3.56). There was no significant difference between trained and
untrained teams regarding age or gaming experience. Each participant in the
trained team was paid 1200 SKr (which included the training sessions). The
participants in the untrained teams received 2 movie tickets each (value ca
200 SEK).
3.2 Experimental design
A 2*2 split plot design was used: team type (trained vs. untrained teams) and
sensor range (limited view vs. full view, see section 3.2.1). Sensor range was
balanced over runs.
19
3.2.1 Sensor range
Sensor range is the amount of visual information available to the participants.
These two conditions provide scenarios with varying amount of difficulty, thus
allowing for a greater variation in the sample collected. This was beneficial for
the earlier study where the goal was to validate the Shared Priorities measure
(Baroutsi et al., manuscript). Two different configurations were used for this
study, full view and limited view.
Full view – all information was available on the interactive map, including
locations of the other member’s units, and the spread of the fire.
Limited view – neither the locations of the other team members’ units, or the
spread of the fires were visible, unless the information is in an adjoining cell to
the participants own units (3*3 cells vision). However, all the objects on the
map were still visible, e.g. houses, vegetation, and pumps.
3.2.2 Role configurations
The organization consisted of three roles: Fire Chief, Water Chief and Gasoline
Chief.
The simulation supplied three kinds of units; fire trucks, water trucks and
gasoline trucks. Fire Chief controlled six fire trucks; two of them were faster
than the other trucks but had smaller water tanks. These were good for scouting.
Water Chief controlled two fire trucks and three water trucks. Gasoline Chief
controlled two fire trucks and three gasoline trucks (see Figure 5). The
configuration forces the participants to coordinate their actions within the team
in order to become successful.
20
Figure 5. Role configuration.
3.2.3 Scenario
Both types of sensor range (limited/full view) used the same map configuration
and scenario script, but the map was rotated and flipped to avoid familiarity.
This enables the usage of identical map and scenario configurations, while the
subject still experiences it as new. Consequently, this allows for comparisons
between the two scenarios, since they are identical. The map is 60*60 cells,
whereas the interface only allows for a 40*40 cells view at a time. The
sequence of events is seen in
Table 1, and in Figure 6 the map is presented with the fires plotted out.
Table 1: Sequence of events with corresponding minutes into the round. The locations of
the fires on the map can be seen in Figure 6.
Time Event Time Event
0.00 Fire 1 16.00 Fire 4
5.00 Fire 2 23.15 Fire 5
7.30 Fire 3 25.00 End
9.00 Pause
21
Figure 6. The map used during the experiment. All units are located in the top left corner
when the simulation begins. “F” stands for fire, and the number corresponds to the
sequencing of the fires. The size of the letters is relative to the size of the fire when it
initiates.
3.2.4 Script commentaries
The scenario for the experiment is built to allow for the same types of strategies
to be applied as during the training.
During the first fire (F1) two villages were threatened. Diverse types of trees
surrounding the initial starting point of the fire produces different growth
patterns in the different directions. Initially the fire spread northwards because
of the pine trees that were closely located and fast burning. The pine trees were
followed by birch trees, which would slow the fire down. On the east side the
pine trees were located a bit further away from the fire’s starting point, but the
pines reached all the way to the eastern village. This meant that the fire would
reach this eastern village faster than the northern village. In the full view
scenario this might not be a problem, but it would become challenging during
the limited view condition. This calls for an understanding of how the different
22
objects were affected by the fire, in order to successfully anticipate and control
future events.
While F1 was still burning a second fire (F2) started at the opposite side of the
map. To begin with, this is a very small fire starting in only two cells. It would
be easy to contain if units were sent directly. Two fast fire units can put it out if
they leave straight away. However, the fire grows exponentially, and to
postpone actions would make the task increasingly more difficult. To make it
worse, there is a big pine forest growing on the east side of F2, with a village
right next to the forest. Stopping F2 from reaching the pine forest is therefore of
great importance. Once the fire reaches the pine trees it is very hard to control
it. Less experienced teams might try to save the house on the west side of F2
instead, with dire consequences. If so, this decision puts the entire village in
danger instead. Also, the amount of burnt out cells will cost a lot more (in score
count) than the single house.
When the next fire starts (F3) the same dilemma faces the teams again. If they
have not been able to manage F2 the best choice is to save the village,
sacrificing the forest between F2 and F3. Here the pause is implemented since
all teams will be facing competing goals, i.e., two or three fires burning. The
competing goals were of interest for the earlier study, since the new measures
were implemented during the freeze.
When the round continues, another 7 minutes will pass before a new fire starts
(F4). There were two schools closely located to the fire, which starts in the
middle of a pine forest. Here the goal would be to protect the schools that were
located north of the fire. The teams might also try to rescue the house located
on the south side of the forest. Most teams would not have finished with F2 and
F3, which by now have created one big fire. After another 7 minutes F5 will
begin. This fire is located in the top right corner. This makes it easier for the
groups that were already having difficulties handling the previous fires, but at
the same time it gives the high performing teams something to do before the
end of the game.
Some teams would be able to put out one fire before the next starts. However,
during this time they have to create strategies and prepare for the next fire. For
example, one successful strategy adopted by several of the trained teams was to
refill water and gasoline and to distribute the units over the map in smaller
groups, as they wait for the next fire.
23
3.3 Dependent measures
The performance measure was collected through system logs in C3Fire, and the
communication to be analyzed consisted of both verbal and written
communication.
3.3.1 Simulation performance
The Simulation Performance was calculated from the amount of cells that had
been burnt down or put out. Different cells resulted in different points,
depending on if it contained an object (see Table 2). Participants were briefed
about the scoring system during the introduction. For each scenario the
maximum score was calculated by allowing the fire to spread without
intervention. The performance score was then calculated by dividing the teams
achieved score with the maximum score. This score was then subtracted from 1:
giving that 0 was the worst performance, indicating that all cells that could burn
out did so, 1 was instead the optimal performance, indicating that all cells had
been rescued (it is only the optimal performance theoretically; it is not actually
possible to achieve 1). The Simulation Performance was calculated on a team
level.
Table 2. Scoring system in the game.
Object Score Object Score Object Score
Burned school -200 Burned house -50 Burned other -3
Saved school -50 Saved house -10 Saved other -1
3.3.2 Communication
In this study the relationship between communication and team effectiveness is
under focus. A coding scheme was applied to the communication of the two
team types, in an effort to find the mediating characteristics. This coding
scheme has not been developed for this particular experiment, and problems
related to generalization were expected. Johansson, Trnka, Granlund & Götmar
(2010) modified a coding scheme earlier used by Svenmarck & Brehmer (1991,
quoted in Johansson et al. 2010), to evaluate the benefits of using geographical
information systems in emergency response situations. Their experiment was
24
also conducted within the C3Fire environment, but they only treated written
communication. In addition, Johansson et al. (2010) adopted a hierarchical
command structure. The commanders performed all the planning and created
the strategies, although this communication was not recorded. The orders were
then communicated from the commanders to the ground chiefs through an email
function, which was the communication they analyzed.
In the current study a flat organization structure was developed through the
configuration of the microworld, and planning and strategies were expected to
be included in the communication (see Baroutsi et al. 2013). Also, in addition to
the email the participants communicate verbally. Problems were expected since
the coding scheme was not developed for neither verbal communication, nor
strategies and planning.
3.3.3 Level of transcription
The level of transcription can vary widely, depending on the type of discourse
and the purpose of the analysis. If the purpose is to only convey the content of
the speech, then a rough transcription would be enough. For other purposes
intonations and other verbal cues might become important and a deeper level of
transcription needs to be applied (Norrby, 1996). For the purpose of the current
study a mixture of Level I and Level II of Linell’s (1996) levels of transcription
was chosen. This was a literal transcription that identified reproduced word
occurrences, retakes, and incorrect initiations of sentences (Level II), but it also
includes hesitating sounds, and overlapping of speech (Level I). Excluded from
the transcription was length of pauses, intonations of sounds, speech rate, and
speech strength. In comparison, the last level, Level III, is completely
normalized to the written language, and only includes complete sentences.
3.3.4 Coding procedure
The participants had the option to use both verbal and written communication.
The verbal communication was recorded using three mp3-players (Olympus
VN406PC). One recorder was placed next to each participant. A mailing system
available in the C3Fire software automatically tracks and stores the written
communication in log files. These communication log files were extracted once
the session was completed. Both the verbal and written communication was
used in the analysis.
25
The coding consisted of five steps and was conducted by two raters:
1. The code scheme was applied and discussed using a transcription not
part of the sessions in focus (communication from the training of the
trained teams). An effort was made to reach a consensus concerning the
boundaries of ambiguous categories, as well as specific expansions of
categories, in order to make it more applicable (see Table 3). This step
was essential since the coding scheme was developed for written
communication in a hierarchical command structure.
2. The interpretations and alterations to the coding scheme were discussed
with one of the developer of the coding scheme (Johansson et al., 2010).
3. The transcriptions needed to be segmented into single phrases, each
phrase could later only be attributed one category.
4. Each rater individually coded all the material.
5. Inter-rater reliability was calculated using Cohen’s kappa and the results
were interpreted according to the guidelines of Landis & Koch (1977).
The new interpretations to the coding scheme in Table 3 are mainly self-
explanatory. However, category number 6 has been generalized from ‘mission
orders’ to ‘mission orders and strategies’. The difference is that a mission order
is only applicable when an assignment is to be carried out, e.g. “Fight fires in
the north”, while a strategy also includes passive actions, e.g. “Let’s ignore that
for now”.
Table 3. Coding scheme, both original interpretations and new interpretations are
included. In the “Original Category” column are the categories created by Johansson et
al. (2010). These are followed by the “New Interpretations” developed during the first step
in the coding procedure. The examples are created using the new interpretations of the
categories.
No. Original Category New
Interpretation
Examples
Questions
1 About the fire About the context “Where is the fire?” “Is
there a water pump near
the school?”
2 About other unit’s
activity
About activity
(others’ and own
“Do you have any water
left in number 12?”
26
unit’s) “Where should I go now?”
Information
3 About the fire About the context “The fire has reached X, Y”
“It is a big fire”
4 About own activity - “I am fighting the fire at X,
Y” “I am heading for Y-
town”
5 About other’s activity - “Your fuel truck is out of
fuel” “X is fighting the fire
north of me”
Order
6 Mission order Mission order and
strategies
“Fight fires in the north”
“Let's ignore that for now”
7 Direct order - “Go to X, Y” "Stop and give
me fuel first"
Other
8 Request for help - “Can you send me some
back up to X, Y?” “I need
water on X, Y”
9 Request for clarification - “Did you mean Y-town?”
“Where were you?
10 Acknowledgment (on
order or info)
- “Got it, thank you”
“Mission accomplished”
"No"
11 Misc (including system
messages,
encouragement)
- “Is there anything on TV
tonight?” “Keep up the
good work”
3.4 Procedure
The session took about 2 hours for the trained teams to complete, and about 2
hours and 30 minutes for the untrained teams. The time difference is due to the
introduction of C3Fire that the untrained teams went through.
27
The participants arrived with their team. First they were informed about the
study and the purpose. They were given a written description of C3Fire and its
current configuration (appendix 1) and other information concerning the
session. Before the session started they answered a background questionnaire
(appendix 2). This was followed by a briefing on how to respond to the Shared
Priorities/Content Analysis (see Baroutsi et al., manuscript). The experimenter
asked if they understood the instructions and clarified if needed.
3.4.1 Preparing the untrained teams
The untrained teams went through an introduction where they learned about the
functionality of the game. This consisted of two separate rounds in C3Fire,
during which they were allowed to ask all sorts of questions. During the
scenarios used for the analysis, they were only allowed to talk to the
experimenter concerning the functionality of the simulation, i.e. if something
was not functioning properly. For the first round the full view was used with a
40*40 cells map, and the scenario was 11 minutes long. All participants had the
same types of units to control; one water truck, one gasoline truck, and three
fire trucks. This configuration allowed them to try on all units. A pause was
implemented 6 minutes into the simulation, during which the participant
answered a questionnaire. This was a distractor task meant to prepare them for a
freeze implemented during the upcoming scenario. During the second round the
limited view was used with a 60*60 cells map. The roles had the same setup as
Figure 7. Participants engaging in C3Fire.
28
during the upcoming scenario (see 3.2.2). This round took 7 minutes, resulting
in a total of 18 minutes hands-on practice for the untrained teams.
3.4.2 Session procedure
The participants chose how to distribute the roles within the team. Each team
played two rounds in C3Fire á 25 minutes. After 15 minutes into the rounds a
freeze was implemented, during this freeze measures analyzed in a previous
study was collected (see Baroutsi et al. manuscript). The round then continued.
When the round was finished measures related to the previous study were again
collected. When the round was finished the experimenter took a print screen of
the map from the observer’s screen.
3.5 Apparatus
Four Dell computers were used to run the C3Fire simulation, including a server
computer that was used to control and run C3Fire. The server computer had
2.73 GHz processor and 4 GB RAM. The other three computers were used by
the participants. The participants’ computers had 2.66 GHz processor and 3 GB
RAM. All four computers were equipped with Windows XP Professional
operating system. The screens used by the participants were connected to a
power strip, allowing the experimenter to easily control when the screens
should be turned on or off. C3Fire version 3.2.7 was used in this experiment.
Each role was assigned to a specific computer. The participants were separated
by dividers to make sure that they could not see each other’s screens (see Figure
8).
Figure 8. Setup of the computers during the experiment.
29
4 Results
Initially, results concerning the participants, inter-rater reliability, and the
coding scheme are presented in order to act as a framework to interpret the
findings concerning the communication.
Univariate ANOVA’s were conducted to compare the two team types, trained
and untrained. There was no significant difference between trained and
untrained teams regarding age, gaming experience, gender, computer
experience or firefighting experience. A difference was however found in
familiarity with the other team members, F(1, 69) = 11.54, p = .001. The
untrained teams reported to be closer friends, M = 4.20 (SD = 1.05), than the
trained teams, M = 3.08, (SD = 1.65).
The raters conducting the coding had previous experience of coding, both
verbal and visual communication material. Cohen’s κ was analyzed to
determine the inter-rater reliability, and a substantial agreement (Landis &
Koch, 1977) was found between the raters, κ = .689, p = .005. The coding from
Rater 1 was used for the analysis.
4.1 Coding scheme reliability
A cross tabulation of the raters’ code was used to analyze the reliability of the
distinct categories in the coding scheme (seeTable 4).
For each category, with Rater 1 as base, the relative amount of inconsistent
categories was calculated. E.g. for category 1 the number of inconsistent codes
was divided by the total amount in that category, 92/186 = .4946. This gives
that the raters disagreed 49.46 % of the times, for the times that Rater 1
assigned a phrase with category 1. Thus, the inter-rater reliability of this
category is not very high (this does not cover the internal reliability for Rater 1).
Four categories in the schema proved to be questionable concerning the inter-
rater reliability. Categories 1, 5, 9 and 11 all displayed a large inconsistency,
ranging between 49 – 53 %. Categories 6, 7 and 8 showed a moderate
inconsistency of 34 – 36 %. Most reliable were categories 2, 3, 4 and 10
ranging between 6 – 25 % inconsistencies.
30
Table 4. Cross tabulation of the rater’s categories. Red marks indicate systematic
mismatches that are affecting the reliability of the category (read left to right). Green
marks indicate systematic errors that are not threatening the reliability of the category.
Rater 2
Total 1 2 3 4 5 6 7 8 9 10 11
Rater 1
1 94 51 14 4 0 4 0 0 17 1 1 186
2 9 794 1 47 19 12 4 20 137 5 4 1052
3 6 3 743 129 21 20 1 3 2 34 13 975
4 3 35 37 2204 85 9 20 119 8 286 16 2822
5 0 21 12 193 314 32 22 21 1 35 10 661
6 0 10 27 40 23 265 21 10 0 6 1 403
7 0 1 6 15 18 52 234 32 3 4 2 367
8 0 60 4 133 63 59 122 880 3 27 10 1361
9 13 89 9 12 9 0 1 3 145 21 4 306
10 2 4 13 88 13 3 2 1 4 2252 31 2413
11 2 7 32 122 21 3 3 10 2 94 303 599
Total 129 1075 898 2987 586 459 430 1099 322 2765 395 11145
This information was further used to find systematic errors. For each category
displaying a large inconsistency, a search was made to find with what other
categories they were mainly misinterpreted as. This analysis showed that
category 1 was usually confused with category 2 and 9, category 5 with 4,
category 9 with 2 and 10, and category 11 with 4 and 10.
Lastly, a search was conducted for any misinterpretations with a higher number
than 100 codes. These misinterpretations were not large enough in relation to
the total amount of categories to cause a reliability problem. However, they do
signal a systematic error that can prove interesting to follow up. A few
examples of utterances that become ambiguous when applying the current
coding scheme can be seen in
31
Table 5.
32
Table 5. Examples of ambiguous utterances.
Utterance Possible categories
But soon I should soon have locked it in anyway 3, 4
I put out the fire on top 3, 4
Now I am fueling number 8 4, 5
4.2 Simulation performance
The simulation performance was analyzed using a repeated measure ANOVA
with sensor range as the repeated measure and team type as the independent
measure. A main effect was found for sensor range, F (1, 10) = 29.63, p < 0.001,
where the full view condition gave an average score of 0.60 (SD = 0.25) and
the limited view condition 0.39 (SD = 0.19). There was also a main effect
found for team type F (1, 10) = 15.38, p = 0.003. The trained teams
(M = 0.65, SD = 0.06) performed better than non-trained teams
(M = 0.34, SD = 0.06). No interaction effect was found, see Figure 9. The
results of the simulation performance has previously been presented in Baroutsi
et al.(manuscript).
33
Figure 9. The Simulation Performance for the two types of teams during the different
sensor range conditions, displayed with error bars.
4.3 Communication
Each section in this chapter is directly related to a specific research question:
Q1. “How does the communication pattern differ between trained and
untrained teams?” relates to the results in section 4.3.1.
Q2. “How does the communication pattern change during diverse visual
conditions?” relates to the results in section 4.3.2.
Q3. “Can a relationship be established between the communication frequency
and performance of the teams?” relates to the results in section 4.3.3.
Q4. “Is it possible to predict performance through the communication
content?” relates to the results in section 4.3.4.
34
4.3.1 Team type comparison
Initially a univariate ANOVA concluded that the type of team did not influence
the communication frequency, F(1, 23) = .313, p = .582. The communication
frequency included all types of communication, e.g. the total amount of phrases.
A multivariate ANOVA was used to analyze the categories in the coding
scheme, the team type was the fixed factor and the communication categories
the dependent variables. A difference was found in communication pattern
between the team types, F(1, 23) = 6.257, p = .004. A test of between-subjects
effects revealed that the differences were found for code 1-6 and 8, as well as
tendencies for 9 (see Table 6 and Figure 10).
Table 6. Descriptive statistics and results of between-subjects effects on communication
for trained vs. untrained teams. * marks significant results.
Code ANOVA
Trained teams
M (SD)
Untrained teams
M (SD)
1 F(1, 23) = 4.599, p = .044* 9.50 (6.59) 6.00 (4.65)
2 F(1, 23) = 14.903, p = .001* 29.00 (11.87) 58.58 (23.12)
3 F(1, 23) = 6.002, p = .024* 49.25 (22.36) 32.00 (14.57)
4 F(1, 23) = 10.994, p = .003* 92.50 (34.02) 142.67 (38.38)
5 F(1, 23) = 6.311, p = .021* 23.25 (10.62) 31.83 (6.38)
6 F(1, 23) = 5.086, p = .035* 22.50 (12.37) 11.08 (11.51)
7 F(1, 23) = .146, p = .706 14.25 (12.81) 16.33 (12.93)
8 F(1, 23) = 7.900, p = .011* 45.42 (17.95) 68.00 (19.74)
9 F(1, 23) = 4.170, p = .055 9.08 (5.84) 16.50 (11.67)
10 F(1, 23) = .990, p = .332 108.08 (40.86) 93.00 (28.96)
11 F(1, 23) = .977, p = .335 29.00 (25.03) 20.91 (10.73)
This entails that the trained teams asked more questions concerning the context
while the untrained teams asked more questions concerning the activities of the
units (both concerning their own and others’ units). The trained teams also gave
more information concerning the context and the other team members’ activity,
while the untrained teams gave more information about their own activity. The
trained teams did also give more mission orders than the untrained teams.
Requests for help were more frequent among the untrained teams (see Figure
10).
35
Figure 10. Communication patterns for the trained and untrained teams. Displayed on the
y-axes is the phrase frequency, and on the x-axes displays the codes in the coding
scheme.
To illustrate the communication found in the different team types are two
examples of discussions within the untrained teams,
Table 7, and the trained teams, Table 8.
Table 7. Conversation among team members in one of the untrained teams.
Line Alias Utterance
1. Adam: Bob, do you have any fuel trucks?
2. Bob: Where you want?
3. Adam: Ehh… everywhere… hehe. W 53.
Whichever one is close to you.
36
4. Bob: W 50?
5. Adam: Ok, I'll just tell you the numbers. 2, 3, 4.
6. Bob: 2, 3?
7. Adam: 2, 3, 4… and 6
Table 8. Conversation among team members in one of the trained teams.
Row Alias Utterance
1. Adam: You know what Cesar, we could almost head upwards
and position ourselves in the pines close to G 18. It
would’ve been a legendary position.
2. Bob: Yeah, you could do that.
3. Cesar: If it’s gonna start burning then it would probably burn
in the middle of the pines, one can imagine
4. Bob: Yeah, that is probably where it’s gonna start, yes
5. Cesar: Spontaneously
6. Adam: I don’t fucking know, maybe it’s too risky
7. Cesar: It is damn far away from everything else then
8. Adam: Mmm
4.3.2 Sensor range comparison
The type of sensor range showed no effect on the communication frequency
according to the univariate ANOVA analysis, F(1, 23) = .545, p = .838.
A multivariate ANOVA was again used, this time with sensor range as the fixed
factor and the communication categories as the dependent variables. Sensor
range showed a significant difference in the communication patterns, F(1,
23) = 2.991, p = .048. A test of between-subjects effects revealed that the
difference was for code 1 and 3, this was due to an increase in frequency for
37
questions and information concerning the context during the limited map view
scenario (see Table 9 and Figure 11).
Table 9. Descriptive statistics and results of between-subjects effects on communication
for full view and limited view condition. * marks significant results.
Code ANOVA
Full view
M (SD)
Lim view
M (SD)
1 F(1, 23) = 22.065, p < .001* 3.92 (3.00) 11,58 (5,53)
2 F(1, 23) = .776, p = .389 40.42 (25.71) 47.17 (21.61)
3 F(1, 23) = 4.794, p = .041* 32.92 (12.59) 48.33 (24.22)
4 F(1, 23) = .796, p = .383 110.83 (44.73) 124.33 (43.62)
5 F(1, 23) = 2.999, p = .099 30.50 (9.22) 24.58 (9.46)
6 F(1, 23) = 1.958, p = .177 20.33 (14.50) 13.25 (12.02)
7 F(1, 23) = .432, p = .518 17.08 (14.57) 13.50 (10.70)
8 F(1, 23) = .117, p = .736 58.08 (23.03) 55.33 (21.34)
9 F(1, 23) = 3.627, p = .071 9.33 (7.13) 16.25 (11.10)
10 F(1, 23) = .005, p = .944 101.08 (41.36) 100.00 (30.35)
11 F(1, 23) = .192, p = .666 26.75 (23.03) 23.17 (15.50)
38
Figure 11. Communication patterns for the full view and limited view condition.
4.3.3 Communication frequency and performance
A pearson correlation was calculated to determine if there was a relationship
between the communication frequency and the simulation performance scores.
No such correlation could be found, r = -.013, p = .954.
4.3.4 Communication pattern as a predictor of performance
A regression analysis was calculated to investigate whether the communication
patterns could be used as a predictor of the performance, e.g. the simulation
performance score. The variables in the communication scheme significantly
predicted performance scores, R = .94, F(11, 23) = 8.231, p = .001. R2 = .883,
hence, 88.3 % of the variance could be explained by the communication
categories.
Four of the categories in the communication factor contributed to the
predicative model, category 2, 6, 7 and 9, p < .05. Thus, communication content
found to be predicative of performance were questions concerning the activity
39
of the own and other members’ units, mission orders, direct orders, and requests
for clarifications, see Table 10.
Table 10. The categories used in the regression analysis to predict performance. βeta* is
standardized coefficients. * marks significant results.
Category β βeta* t-value Sig.
1 .005 .117 0.471 .646
2 -.006 -.545 -2.148 .053*
3 .000 .024 0.089 .931
4 .001 .193 0.722 .484
5 .012 .483 1.727 .11
6 .009 .495 2.114 .056*
7 -.009 -.46 -2.583 .024*
8 -.001 -.089 -0.386 .706
9 -.016 -.653 -3.127 .009*
10 .002 .302 1.179 .261
11 -.005 -.363 -2.028 .065
4.4 Summary
Cohen’s kappa showed that a substantial agreement was found between the two
raters. Hence, the inter-rater reliability is high enough to continue analyzing the
communication for further interpretations. A closer investigation of the code
produced by the two raters revealed inter-rater reliability concerns for specific
categories, and systematic errors for these and some other categories. The other
systematic errors did not raise any reliability concerns for the categories in
question.
4.4.1 Trained and untrained teams
The trained teams performed better than the untrained teams during both types
of sensor range, hence they achieved a higher simulation performance. No
difference was found in the communication frequency between the types of
teams, e.g. the trained teams uttered as many phrases as the untrained teams.
However, the utterances differed in content.
40
Trained teams:
Category 1: Asked more questions regarding the context.
Category 3: Gave more information regarding the context.
Category 6: Gave more mission orders.
Untrained teams:
Category 2: Asked more questions regarding the activities of their own
and others’ units.
Category 4: Gave more information regarding their own units’
activities.
Category 5: Gave more information regarding activities of other
members units.
Category 8: Gave more requests.
4.4.2 Full view and limited view sensor range
The simulation performance varied between the two types of sensor range. The
teams achieved higher performance scores during the full view condition. No
difference was found for the communication frequency for the two conditions,
but differences in the content were again found:
Category 1: Questions regarding the context became more frequent
during the full view condition.
Category 3: Also information regarding the context increased during
the full view condition.
4.4.3 Communication and it’s relation to performance
The communication pattern could be used as a predictor of performance. Four
categories contributed to the predicative model:
Category 2: Questions about activity
Category 6: Mission orders and strategies
Category 7: Direct orders
41
Category 9: Requests for clarification
The overall communication frequency, e.g. the total amount of phrases, showed
no relation to the simulation performance.
42
5 Discussion
Earlier analysis’ has suggested that the team types (trained vs. untrained) were
different when it came to team effectiveness (Baroutsi et al. manuscript), and
the inter-rater reliability proved to be of substantial agreement (Landis & Koch,
1977). This entails that the reliability is high enough to continue analyzing the
communication for further interpretations.
5.1 Results discussion
The analysis of the results are mainly discussed in the framework of joint
activity (Klein et al., 2005), but are also related to results from earlier studies on
communication.
5.1.1 Communication patterns and team type
It is important to note that the untrained teams communicated as frequent as the
trained teams, the only difference between the two types of teams are found in
the communication content. The results show that the trained teams ask more
questions and provide more information concerning the context. The untrained
teams instead asked more questions regarding the units’ activities, and gave
more information about their own and other unit’s activities.
Hence, the trained teams communicate more frequently concerning the context,
while the untrained teams were more concerned about the units’ activities. One
of the requirements of a joint activity is that the team members share a
sufficient amount of common ground in order to coordinate properly (Klein et
al., 2005). The untrained teams continuously pay high coordination cost, which
is reflected by frequent questions and information sharing on activities as they
strive to repair and manage their unsatisfactory common ground. It also reflects
a deficiency in the interpredictability (Klein et al., 2005) for the untrained
teams, since they were not able to predict each other’s behaviors to a reasonable
degree of accuracy, they will again have to compensate in terms of coordination
costs.
Furthermore, the trained teams gave more mission orders while the untrained
teams gave more requests. This could be a reflection of the directability (Klein
et al., 2005) of the team members. The untrained teams frequently had to repeat
their requests, increasing the amount of requests stated, something not very
43
common among the trained teams. Of course, there is also a difference in self-
synchronization between the two types of teams affecting the frequency of
requests. For example, some trained teams located a water truck on top of a fuel
pump, enabling the firefighters to fill themselves up with water and fuel at the
same time. Also found was examples where the Chief responsible for a
resource, let’s say water, continuously rotated around the firefighters so that
they rarely had to ask for water. This relationship between directability and
requests is not open for further investigation using the current coding scheme.
However, possible developments of the coding scheme in this direction are
discussed in section 5.2.5 and 5.2.5.
When applying the coding scheme the closed-loop communication would entail
an increase of acknowledgements for the trained teams, or even the double
amount of acknowledgements (category 10). This since both step 2 and 3 in the
closed loop communication would in this coding scheme be interpreted as
acknowledgements, and the untrained teams would only reach step 2. No such
differences could be found. The structure of the coding scheme could also
categorize these kinds of acknowledgements as either question about other
units’ activity (2) or information about other’s activities (5). For example, if
someone asks for fuel for unit 8, the acknowledgement might be a simple
”number 8?” with an intonation at the end signaling that it is uttered as a
question. This would still be coded as a question. Another answer, bordering to
an acknowledgement is “you are close to my 13”. This answer both
acknowledges that the information has been received, and also adds additional
information to their common ground. Hence, it can be interpreted as
information regarding activities. However, the results suggest a significant
difference for both category 2 and 5, but where the untrained teams utter them
more frequently. If trained teams possess closed loop communication, but not
the untrained teams, then the opposite pattern would be found. Hence, nothing
suggests that the trained teams would differ from the untrained teams in terms
of closed-loop communication.
5.1.2 Communication pattern and visual conditions
The communication frequency did not differ between the two visual conditions,
but again differences in content were found. It was the questions and
information concerning the context that displayed an increase in frequency
during the full view sensor range.
44
It is rather surprising that no other differences where to be found, since the loss
of visual information changes the task considerably. Communication regarding
questions and information about activities did not increase during the limited
view condition, even if the loss of visual information means reduced
availability of monitoring each other’s activities. The monitoring task is one of
the activities teams often engage in to support the team’s common ground
(Klein et al., 2005). Consequently, there is no way to determine in this study
whether the choice was incorrect or not (not to compensate the loss of visual
information), and whether it was having a negative impact on the team’s
performance. It might also be possible that the gains in common ground were
outweighed by the monitoring costs, and therefore ignored.
5.1.3 Communication frequency and performance
The overall communication frequency would be expected to be lower for the
trained teams than the untrained teams (Obermayer & Vreuls, 1974; Svensson,
2002; Salas, 2005). However, no correlation could be found between
communication frequency and performance, which opposes the results found in
previous studies. Salas (2005) interpreted the findings from Obermayer &
Vreuls (1974) and said that it appears that teams reduce the lengths of their
messages over time, which results in a reduction of communication. However,
in this study an effort was made to ensure that strategies and planning was
important for the teams to become successful (Baroutsi et al, 2013). The result
of the multivariate ANOVA also suggests a different communication pattern for
the two team types. Hence, there is reason to believe that the trained teams took
the opportunity to use this extra time to discuss and strategize, as they reduced
the lengths of their messages in other areas.
5.1.4 Communication content and performance
The communication content served as a strong predictor of performance in the
created model, explaining 88.3 % of the variance. This can be contrasted to the
absence of correlation between the communication frequency and performance
in this study. Communication content that was included in this model was
questions regarding activities, mission orders, direct orders, and requests for
clarifications. The untrained teams displayed a higher frequency for questions
regarding activities, while the trained teams uttered more mission orders and
direct orders.
45
5.2 Method discussion
The C3Fire microworlds was theorized to be a suitable platform form studying
dynamic decision making because of its dynamic, uncertain and time-critical
properties, see 2.7. But to bear in mind is that the sample was small, only 12
teams participated in the study.
5.2.1 Transcriptions and the coding scheme
The verbal communication was transcribed before it was categorized. An
alternative choice would be not to transcribe the material, but instead listening
to the recordings while categorizing each sentence. The benefit of this choice
would be that the categorization would be made straight from the raw material
and no informational cues would be lost. Intonation, pauses, sound level and so
forth were all cues that can reveal intent in verbal messages. However, two
researchers independently categorized the material followed by a calculation of
the inter-rater reliability. For this calculation to be possible the material needs to
be split up into chunks, each chunk is then assigned one category (one by each
rater). The inter-rater reliability is then calculated on the amount of chunks that
have been assigned the same category by the different raters. Hence, for it to be
possible to calculate the inter-rater reliability, the material needs to be divided
before the categorization can commence.
5.2.2 Reliability of the coding scheme
Initially it is important to point out that the reliability concerns for the distinct
categories are all related to the inter-rater reliability, they are based on how the
coding of Rater 1 and Rater 2 differ from each other. The actual analysis of the
communication is only based on Rater 1, and these reliability results do not
reflect the internal reliability of this rater. What can be said is though that the
inter-rater reliability was high enough to suggest sufficient consistency to
continue with the analysis of the results. But conclusions should for these
specific categories be drawn with caution, and the ambiguity should be further
discussed and revised for future work.
Four of the categories displayed large inconsistencies when comparing the
output produced by the two raters, with up to 53% disagreements. These and
other categories were found to be systematically interpreted differently by the
raters, see Table 11. Further improvements on these categories would therefore
46
be beneficial for future developments of the coding scheme. Ideas on how the
improvements can be made are found in the next section, 5.2.3.
Table 11. Categories found to be systematically mixed when comparing Rater 1 and 2.
Nr Category
Systematic
error
Reliability
concern
1 Questions about the context 2, 9 Yes
2 Questions about activity 9
3 Information about the context 4
4 Information about the own activity 8, 10
5 Information about other’s activities 4 Yes
8 Request for help 4, 7
9 Request for clarification 2, 10 Yes
11 Miscelanous 4, 10 Yes
5.2.3 Adopting a grammatical approach to diminish ambiguity
When defining categories for coding communication the examples tend to be
very clean cut, making the distinctions easy. Unfortunately, the participants are
not following the prescribed examples, making the coding process ambiguous.
Many phrases can be interpreted in multiple ways, and it is difficult to decide
the focal point of the utterance. The same utterance could be interpreted as
being about the own units activity, someone else’s activity, and the context – all
at the same time.
Phrases in verbal communication are not distinct categories, a more fair
description would be to view phrases as belonging to a spectrum. Just like
colors, we can create distinct categories but the nuances can, and will always,
be up for debate. When coding material the purpose is to diminish the
ambiguity and increase the reliability of the categories.
One way of dealing with this type of ambiguity could be to adopt a purely
grammatical approach to the coding process. Consider phrase 3 (see Table 12),
“Now I am fueling number 8”. This utterance contains information concerning
both the speakers own unit, as well as information about the activity of vehicle
number 8’s owner. However, there is one active noun, “I am fueling”, and one
passive noun, “number 8”. A linguistic solution could be to assign the category
to the active noun in the sentence: “Now I am fueling number 8” would then
47
belong to category 4, “information about own units activity”. If the same
information would be uttered as “Number 8 is being fueled by me”, then the
active noun would be “8 is being fueled” and instead assigned to category 5
“information concerning other units activity”. In phrase 1 (Table 12) the active
noun would be “I should” even if the speaker is referring to the status of the
fire, “locked it away”, and therefore assigned category 4. The same goes for
phrase number 2, the active noun is “I put out”, hence it belongs to category 4.
Table 12. These examples are all observed in the current study, and the numbers after
the arrows are possible categories that can be assigned to the given utterance.
Utterance Possible categories
But soon I should soon have locked it in anyway 3, 4
I put out the fire on top 3, 4
Now I am fueling number 8 4, 5
This is just one example of how a grammatical approach could be used to
diminish ambiguity, but simple rules can be applied to most situations. This
would be an interesting solution to the encountered reliability concerns. It may
also be interesting to see if better performing teams more frequently talk so that
the active noun is the team member, and not themselves. Since this could
possibly be a reflection of talking and thinking through someone else’s
perspective, an ability claimed to belong to high-performing teams (Klein et al.,
2005).
5.2.4 Strategies and planning
As previously mentioned, the coding scheme is not designed to handle
strategies and planning. When dealing with dynamic decision problems, this
form of communication becomes essential. It is through the process of
deliberation between team members that the process of dynamic decision
making takes place, especially within flat command structures. In
Table 13 presents several good examples of communicational content that arise
during the joint activity of creating strategies and planning. Some of these
examples can be covered within the current coding scheme (row 2, 4, 5, 7, 8 in
Table 13) but some cannot (row 1, 3, 6).
48
Table 13. Conversation among team members in one of the trained teams.
Row Alias Utterance
1. Adam: You know what Cesar, we could almost head upwards
and position ourselves in the pines close to G 18. It
would’ve been a legendary position.
2. Bob: Yeah, you could do that.
3. Cesar: If it’s gonna start burning then it would probably burn
in the middle of the pines, one can imagine
4. Bob: Yeah, that is probably where it’s gonna start, yes
5. Cesar: Spontaneously
6. Adam: I don’t fucking know, maybe it’s too risky
7. Cesar: It is damn far away from everything else then
8. Adam: Mmm
Each example and its category, or absence of category, will be described and
clarified. First presented are the categories that were not covered in the coding
scheme. (1) Adam addresses his team mate and presents him with a possible
future state. This is very common during dynamical decision making. The
presentation of a future state anchors the conversation and offers a hypothetical
common ground. Future states can be used both within questions, information
sharing as well as orders (i.e. “if X happens then you have to Y”). (3) Cesar
further develops Adams trail of thoughts and continues to build on the possible
future state. (6) Adams is now adding a value to the made up scenario, in this
case a negative value signaling to his team mates to consider the flaws of the
future state. He is redirecting their decision making process into a more
cautious trail. As implied, a value can either be negative or positive.
Onwards to the phrases that are already covered. (2/4/8) Acknowledgements on
what has been said. These three utterances do not add any information to the
decision making process. (5) Miscellaneous, a word describing that his previous
contribution was a spontaneous one. (7) Information regarding the context, this
phrase is a description of the map and the discussed location.
49
Consequently, if a coding scheme is to support strategies and planning, valuable
additional categories would include:
Future states
o As a question
o As shared information
o As a conditional order
Adding value
o Negative
o Positive
These categories in combination with the current categories: acknowledgement,
information about the context, mission orders and direct orders would create a
good starting point for extending the coding scheme to also cover strategies and
planning.
Also, in the coding scheme used in this study, questions regarding the own
unit’s activity are coded together with questions regarding other unit’s
activities. This choice was made since the original coding scheme did not
account for these types of questions at all. A modified coding scheme should
differentiate between these types of questions and code them into separate
categories.
5.2.5 Reaching consensus
One drawback when analyzing the coding scheme is that there is nothing
indicating whether the participants were just battling the same questions or
topics repeatedly, or if they were actually progressing.
Table 14. Conversation among team members in one of the untrained teams.
Line Alias Utterance
1. Adam: Bob, do you have any fuel trucks?
2. Bob: Where you want?
3. Adam: Ehh… everywhere… hehe. W 53.
50
Whichever one is close to you.
4. Bob: W 50?
5. Adam: Ok, I'll just tell you the numbers. 2, 3, 4.
6. Bob: 2, 3?
7. Adam: 2, 3, 4… and 6
This form of communication was not unusual among the untrained teams, a
simple request for fuel required many turns before they could exit the phase in
their joint activity (Klein et al., 2005). If many signals have to be performed in
order to complete a simple request, it is arguable to claim that the team does not
possess a good directability (Klein et al., 2005). The untrained teams did utter
more requests than the trained teams, but in the current coding scheme there is
no way to tell whether this difference is a result of repeated requests. It would
be of interest to track how many turns it takes before they actually enters the
exit phase for a request and complete the joint activity, i.e. how many turns does
it take before they reach consensus in the matter. This could allow the coding
scheme to offer insights on the directability of the team members.
The coding scheme was not created to handle planning and strategies. If the
coding scheme would be further developed to incorporate relevant aspects of
planning and strategizing, then what aspects could be of interest to include? In
some of the trained teams strategies seemed to be debated, while in other teams
it was more common to only give orders or information without it being further
discussed. According to Driskell & Salas (1992) teams that more frequently
consider opposing views will have higher error detection, resulting in higher
team performance. Would this entail that teams taking more turns reaching
consensus when strategizing would reach higher performance?
As implied, calculating turns to reaching consensus would have to be applied on
categorized transcriptions in order to be valuable. Otherwise a team repeating
the same requests over and over again would seem no different than a team
debating strategies. But together with a coding scheme interesting results
concerning directability and considerations on opposing views could be
explored.
51
5.2.6 Summary of the proposed coding scheme
The analysis of the team’s communication, together with theories concerning
what enables a team to be successful has led to suggestions on how to further
develop the coding scheme. These changes are summarized in
Table 15, this summary is depicting how the coding scheme has transformed
from the beginning of the study up to the final suggestions of proposed
categories. Further descriptions on the proposed categories can be found in
section 5.2.5 and 5.2.4.
Table 15. The evolution of the coding scheme during the process of this study.
Nr Original Categories New Interpretation Proposed Categories
Questions
1 About the fire About the context
X1 - - About own unit’s activity
2 About other unit’s
activity
About activity (others’
and own unit’s)
About other units activity
X2 - - About future states
Information
3 About the fire About the context -
4 About own activity - -
5 About other’s activity - -
X3 - - About future states
X4 - - Add positive value
X5 - - Add negative value
Order
6 Mission order Mission order and
strategies
Mission order
7 Direct order - -
X6 - - Conditional order
Other
8 Request for help - -
9 Request for clarification - -
10 Acknowledgment (on
order or info)
- -
52
11 Misc (including system
messages,
encouragement)
- -
The newly proposed categories are also exemplified in an effort to clarify the
intention of each category, see Table 16.
Table 16. The newly suggested categories with corresponding examples.
Nr Proposed Categories Examples
Questions
1 About the context “Where is the fire?” “Is there a water
pump near the school?”
2 About own unit’s activity “Where should I go now?”
3 About other units activity “Do you have any water left I number
12?”
4 About future states "What if it starts burning on the other
side instead?"
Information
5 About the context “The fire has reached X, Y” “It is a big
fire”
6 About own activity “I am fighting the fire at X, Y” “I am
heading for Y-town”
7 About other’s activity “Your fuel truck is out of fuel” “X is
fighting the fire north of me”
8 About future states "If it’s gonna start burning then it
would probably burn in the middle of
the pines"
9 Adding positive value "I think it's gonna work" "That is a good
point"
0 Adding negative value "It might be too risky" "There is no
point trying to save the that village"
Orders
10 Mission order “Fight fires in the north” “Let's ignore
that for now”
11 Direct order “Go to X, Y” "Just leave that forest for
now"
53
12 Conditional order ”If the fire starts at XY, then use your
fast units to get there”
Other
13 Request for help “Can you send me some back up to X,
Y?” “I need water on X, Y”
14 Request for clarification “Did you mean Y-town?” “Where were
you?
15 Acknowledgment “Got it, thank you” “Mission
accomplished” "No"
16 Miscelanous “Is there anything on TV tonight?”
“Keep up the good work”
54
6 Conclusions
The conclusions will be related to the initial research questions: each question
will be directly responded to.
6.1 Results
Q1. How does the communication pattern differ between trained and
untrained teams?
No difference could be found between the teams in communication frequency,
but differences in communication content were found. The trained teams
communicated more frequently regarding the context, contrasted by the
untrained teams who uttered more phrases concerning unit’s activities. This can
be interpreted as a deficiency in common ground and interpredictability among
the untrained teams. They find themselves having difficulties coordinating in
their joint activity, and they have to pay coordination costs in terms of an
increased frequency in choreographing signals concerning activities.
The trained teams also gave more mission orders, while the untrained teams
uttered more requests. The increase of requests among the untrained teams
partially stems from a lack of self-synchronization, an ability that was found
among the trained teams. Repeated requests were also common among the
untrained teams. The increase in requests can therefore be interpreted as an
inability to direct each other, i.e. a deficiency in directability.
Q2. How does the communication pattern change during diverse visual
conditions?
The communication frequency did not differ between the two conditions: full
and limited view. In the communication content, the only difference found was
an increase in questions and information regarding the context. This was found
to be rather surprising since the loss of visual information would reduce the
availability to monitor each other’s activity, but no compensatory actions could
be found.
Q3. Can a relationship be established between the communication
frequency and performance of the team?
No relationship was found between communication frequency and performance,
unlike results from previous studies that found this relation to be true
55
(Obermayer & Vreuls, 1974; Svensson, 2002; Salas, 2005). However, the
communication is an important tool when a team has to make decisions
together, and the communication content did differ between the trained and
untrained teams. It is therefore feasible to believe that the trained teams took
advantage of the situation, and started discussing strategies and plans when the
communication in other areas reduced.
Q4. Is it possible to predict performance via the communication content?
The communication content could explain 88.3 % of the variance in
performance. Categories contributing to the predicative model of performance
included category 2, questions regarding activity, 6, mission orders, 7, direct
orders, and 9 requests for clarifications.
6.2 Method
Q5. What are the limitations of the coding scheme?
Systematical differences were found between Rater 1 and Rater 2, even if the
inter-rater reliability was up to a substantial degree (Landis & Koch, 1977),
reflecting ambiguity between the categories. To find ambiguity between the
categories is not surprising, since language is not naturally divided into clean
cut sections. A language rather behaves like a spectrum of colors, made up of
nuances that slowly merge into new colors. It is suggested that this ambiguity
can be diminished by adopting a grammatical approach, a way of learning from
the language sciences. For example, to let the active and passive noun decide
whether a sentence is concerning one’s own actions, or if it is about someone
else’s action.
The main deficiency of the coding scheme when it came to handling strategies
and planning was its lack of including future events, and adding value to an
idea. Also there was no differentiation between questions regarding one’s own
actions, and another member’s actions. To cover these aspects new categories
where suggested:
Future states
o As a question
o As shared information
o As a conditional order
56
Adding value
o Negative
o Positive
Split up category 2: Questions regarding activities
o Own unit’s activities
o Other unit’s activities
Another limitation with the current coding scheme is that nothing differentiates
between teams battling the same topic or questions repeatedly, and teams that
actually progress in their discussion. It would hence be of interest to calculate
the number of steps it takes for a team to reach consensus in a matter. If this
was adapted together with a coding scheme, then interesting results concerning
directability and considerations of opposing views may surface.
6.3 Future research
The proposed coding scheme needs to be tested on a new sample, or that the
same sample is coded by new raters, to find out if it is really applicable. It
would also be interesting to investigate the grammatical approach to diminish
ambiguity when coding verbal communication.
The act of coding corpuses is a common trait within the field of language
technology, and this act can be automated. A program can perform the coding
process, and what is needed is a grammatically tagged dictionary and a set of
rules on how to apply the categories. Two advantages are gained by utilizing an
automated process. (1) When creating the rules, no fussy distinctions between
the categories would be accepted by the program and the categories have to be
specified in detail. (2) The program would always code the material in the same
manner, eliminating the inter-rater reliability problem. Conversely, the
disadvantage would be that utterances that are ascribed with an incorrect
category would be coded not somewhat wrong, but terribly wrong. It is thus
advisable to still manually go through the coded material and catch these
mistakes. Hence, if a machine initially codes the material and a researcher goes
back and corrects the obvious mistakes, then the inter-rater reliability wouldn’t
completely disappear, but it would diminish substantially. Also, this method
would not actually ease the workload or save time since the researcher still
would have to manually go through the material.
57
7 References
Amabile, T., & Fisher, C. (2009). Stimulate creativity by fueling passion. In
Locke, E. (Ed.). Principles of organizational behavior (pp. 481-495).
West Sussex, UK: Wiley.
Bandow, D. (2001). Time to create sound teamwork. The Journal for Quality
and Participation, 24, 41–47.
Baroutsi, N., Berggren, P., & Johansson, B. (manuscript). Assessing the Shared
Priorities measure. (FOI). Linköping.
Baroutsi, N., Berggren, P., Johansson, B., Nählinder, S., Granlund, R., Turcotte,
I., & Tremblay, S. (2014). Assessing development of team training in
emergency management. In Proceedings of the 11th ISCRAM. The
Pennsylvania State University, University Park.
Baroutsi, N., Berggren, P., Nählinder, S. & Johansson, B. (2013). Training
teams to collaborate as cohesive units. (FOI-R--3830--SE). Linköping.
Berggren, P., Baroutsi, N., Johansson, B., Turcotte, I., & Tremblay, S. (2014).
Assessing team focused behaviors in emergency response teams using the
shared priorities measure. In Proceedings of the 11th ISCRAM. The
Pennsylvania State University, University Park.
Berggren, P., Johansson, B. J. E., Baroutsi, N., & Dahlbäck, N. (2014). The
shared priorities measure as a way of assessing team strategic awareness –
a bridge between self-assessment and the deep blue sea of field
recordings. In Proceedings of the ECCE 32nd annual conference.
Berggren, P., Johansson, B., Svensson, E., Baroutsi, N., & Dahlbäck, N. (2014).
Statistical modelling of team training in a microworld study. In
Proceedings of the HFES 58th annual meeting. Hyatt Regency Chicago in
Chicago, Illinois.
Brehmer, B. (2000). Dynamic decision making in command and control. In
McCann, C., & Pigeau, R. (Eds). The human in command: Exploring the
58
modern military experience (pp. 233-248). New York: Kluwer
Academic/Plenum Publisher.
Brehmer, B. (2004). Some reflections on microworld research. In Schifflett, S.,
Elliot, R., Salas, E., & Coovert, M. (Eds.) Scaled worlds: The
development, validation and applications (pp. 22-36). Bodmin, Cornwall:
MPG Books Ltd.
Brehmer, B., & Dörner, D. (1993). Experiments with computer-simulated
microworlds: Escaping both the narrow straits of the laboratory and the
deep blue sea of the field study. Computers in Human Behaviour, 9, 171-
184.
Christoffersen, K., & Woods, D. (1995). How to Make Automated Systems
Team Players. Advances in Human Performance and Cognitive
Engineering Research, 2, 1–13.
Clark, H. (1996). Using Language. Cambridge: Cambridge University Press.
Cooke, N., Gorman, J., & Winner, J. (in press). Team cognition. In F. Durso,
R. Nickerson, S. Dumais, S. Lewandowsky, & T. Perfect (Eds.).
Handbook of Applied Cognition, 2 Ed. Wiley.
Granlund, R. (2002). Monitoring distributed teamwork (Training Ph.D. Thesis).
Linköping University: Department of computer and information science.
ISBN 91-7373-312-1.
Granlund, R., Johansson, B., & Persson, M. (2001). C3Fire a micro-world for
collaboration training in the ROLF environment. In proceedings to SIMS
2001 the 42nd Conference on Simulation and Modelling, Simulation in
theory and practice. Porsgrunn, Norway, 8-9 October.
Hackman, R., & Wageman, R. (2009). Foster team effectiveness by fulfilling
key leadership functions. In E. Locke (Ed.) Principles of organizational
behavior (2 Ed. pp. 475-293). West Sussex, UK: Wiley.
59
Johansson, B. (2005). Joint control in dynamic situations. (Dissertation no.
972). Linköpings University, Department of computer and information
science.
Johansson, B., Trnka, J., Granlund, R., & Götmar, A. (2010) The effect of a
geographical information system on performance and communication of a
command and control organization, International journal of human-
computer interaction, 26, 228-246.
Klein, G., Orasanu, J., Calderwood, R., & Zsambok, C. (1993). Decision
making in action. Norwood, New Jersey: Ablex Publishing Corporation.
Klein, G., Feltovich, P. J., Bradshaw, J. M., & Woods, D. D. (2005). Common
ground and coordination in joint activity. Organizational simulation, 139-
184.
Krumm, R. (1958). Crew member agreement on b-52 crew operating
procedures as an index of crew proficiency. Ft. Belvoir: Defense
Technical Information Center.
Landis, J., & Koch, G. (1977). The measurement of observer agreement for
categorical data. Biometrics, 33, 159-174.
Larrick, R., (2009). Broaden the decision frame to make effective decisions. In
E. Locke (Ed.) Principles of organizational behavior (2 Ed. pp. 461-480).
West Sussex, UK: Wiley.
Letsky, M., Warner, N., Fiore S., & Smith C. (2008). Macrocognition in Teams.
Aldershot: Ashgate.
Lindgren, I., & Smith, K. (2006). Using microworlds to understand cultural
influences on distributed collaborative decision making in C2 settings.
The 11th international command and control research and technology
symposium. Cambridge.
Lindgren, I., Hirsh, R., & Berggren, P. (2007). It takes three points to define a
common ground: breathing apparatus fire-fighters’ communication during
rescue operations. Journal of pragmatics. 39, 1482-1502.
60
Marks, M., & Panzer, J. (2004). The influence of team monitoring on team
processes and performance. Human Performance, 17(1), 25–41.
Morgan, B., Glickman, S.,Woodard, A., Blaiwes, S., & Salas, E. (1986).
Measurement of team behavior in a navy training environment (Tech.
Rep. TR-86-014). Orlando, FL: Naval training systems center, Human
factors division.
Norrby, C. (1996). Samtalsanalys: så gör vi när vi pratar med varandra. Lund:
Studentlitteratur.
Obermayer, R., & Vreuls, D. (1974). Combat-ready crew performance
measurement system: Phase IIIA, Crew performance. Northridge, CA:
Manned Systems Science.
Pentland, A. (2012). The new science of building great teams. Harward
Business Reveiw, 90, 60–70.
Persson, M., & Rigas, G. (2014). Complexity: The dark side of network-centric
warfare. Cognition, Technology and Work, 16(1), 103-115.
Salas, E. (2005). Is there a “Big Five” in Teamwork? Small Group Research,
36(5), 555–599.
Salas, E., Dickinson, T., Converse, S., & Tannenbaum, S., (1992). Toward an
understanding of team performance and training. In R. Sweezy, W., & E.
Salas (Eds.), Teams: Their training and performance. Norwood, NJ:
Ablex.
Svensson, J. (2002). Communication and performance: Aspects of the
relationship between communication content, frequency and problems,
and team outcome, studied in a military multi-aircraft simulator. (Thesis).
Linkoping’s University, Department of computer and information science.
Swezey, R., & Salas, E. (1992). Teams: Their training and performance. USA:
Ablex Publiching Corporation.
61
Tremblay, S., Vachon, F., Lafond, D., & Kramer, C. (2011). Dealing with task
interruptions in complex dynamic environments: Are two heads better
than one? Human Factors, 54, 70-83.
62
Appendix 1
Instructions to players
Your main task in this computer-generated game is to put out a forest
fire in a simulated world.
Organization and Assignment
Your task during a session is to put out a forest fire and protect
important objects in the simulated world. You affect the fire
simulation by commanding the units in the game. In order to be
efficient you have to make plans for, establish a structure in the work
and synchronize the units activities. As the game begin one or more
fires will start. Your objectives are to put out the fire, to save as many
houses as possible and to save as much terrain as possible.
Ground Chiefs
X
Units, Houses, People
and the Fire, etc.
Computer
Simulation
Y
Figure 1: The simulated world.
The organisation in this game consists of three players; Fire Chief,
Water Chief and Gasoline Chief.
The game supplies three kinds of units; fire trucks, water trucks and
gasoline trucks. Fire Chief will control 6 Fire trucks, 2 of which are
faster then the other trucks, but has a smaller water tank and is
therefore not as efficient when it comes to putting out the fire. Water
Chief will control 2 Fire trucks and 3 Water trucks. Gasoline Chief
will control 2 Fire trucks and 3 Gasoline trucks.
Figure 2: The three roles with their assigned units.
The Simulated World and Interface
In the centre is a map of the simulated world and a clock is displayed
at the top left corner of the map. Unit information and the mail
function is located left of the map. On the right side of the map is the
pointer’s position, the intended position of the units and object
descriptions.
64
Figure 3: Example of a player’s user interface.
Map
The simulation divides the world in a matrix of 40x40 cells. Each cell
in the matrix is uniquely identified by a letter (indicating its column)
and a number (indicating its row). The speed and direction of the fire
depends on two factors: the type of vegetation, and the activities of
the fire trucks you command. The fire will not spread outside the
matrix.
There are three different kinds of visual fields, the experimenter will
tell you what kind of map view will be used before the session starts.
Full map view: you will be able to see each others units as well as
the fire.
Partial map view: you will see the fire but not each others units
unless they enter your visual field (3x3 cells around each of your
units).
Limited map view means that you will only be able to see the fire
and the other players units when they enter your visual field (3x3
cells). The map has a memory built in which means that it will
remember the fire where you have been, but the cells will not update
the information until you go there again.
Vegetation and Geographical Objects
There are three kinds of vegetation in the world that burn at different
rates: normal vegetation, pine, trees and birch trees.
Normal vegetation.
Pine trees (burn two times faster than normal vegetation)
Birch trees (burn at half the rate of normal vegetation)
House. Important to protect.
Water tank station. Contains an unlimited amount of water.
Water supply units and fire supply units can get water here.
Fire
A fire in one cell can spread to all eight surrounding cells, see figure
5. Cells that are no-longer burning or burned-out cannot start to burn
again.
66
Normal
Burning
No longer burning
Burned-out
Figure 4: Status of the fire Figure 5: Development of the fire
Hints for fighting fire in C3Fire efficient
The fire fighting units have to collaborate in order to be successful in
putting out the fire efficiently. An adequate collaboration technique is
to put out the fire in edges. In figure 6 the units join forces while they
put out the fire in the south. In figure 7 units do not collaborate. In
this case the fire slips between the fire fighters and will eventually
spread around them.
Figure 6: Good collaboration Figure 7: Bad collaboration
Units
The units will do exactly what you command them to do if they have
the resources available to do so.
The panel ‘Unit Info’ is on the left side of the map, figure 8. It
contains information about the unit’s position, activities, and
properties. Below is a panel called ‘Unit property’, figure 8, which
contains information about the unit’s control functionalities. Select
what unit to display by clicking on it in the ‘Unit info’ panel.
Figure 8: Unit Info
Figure 9: Unit Property
68
Control the units movements
In the map the units are identified with different colors. The unit’s
current position is exposed with a colored number, and its intended
position (where it is going) is exposed with a white number, figure
10.
Move a unit: Drag-and-drop the colored number to the intended
position. The number will appear in that cell in white. If you
want to change the destination position while a unit is going,
drag-and-drop the white number to a new destination, figure 10.
It is also possible to command a unit by selecting its white
number in the ’Unit Palette’ panel on the right side of the screen
and push the left mouse button on the intended cell on the map,
figure 11.
If the white number disappears immediately after you have
placed it on a cell at the map, then the unit has run out of fuel.
The speed of all units is independent of vegetation and objects
on the map and the fire does not affect them.
Fire trucks
The Fire trucks current position is exposed with a number in a red
colour.
Fighting fire: A fire truck will automatically start to fight the fire if it
is standing on a burning cell, it will only fight the fire in the current
cell. Putting two fire-fighting units on the same burning cell will not
put the fire out faster.
Running low on gasoline and water: Refill it by standing next to a
gasoline truck, water truck or water tank station. The units can not get
refueled if they are positioned diagonally from each other.
Activity: A fire-fighting unit can perform six different
activities.
Inactive (Doing nothing)
Moving
Mobilizing (Preparing to fight fire)
Fire fighting
Demobilizing (Ending the extinguish of fire)
Refill water
Water trucks
The Water trucks current position is exposed with a number in a blue
colour.
It can be refilled at a water tank station or by another water truck.
Also the Fire trucks are able to use the water tank stations. The Water
trucks can refill one unit at a time.
Activity: A water truck can perform four different activities.
Inactive (Doing nothing)
Moving
Tapping water
Refilling water
Gasoline trucks
The unit’s current position is exposed with a number in a yellow
colour.
70
It can be refilled at a fuel tank station and it can refill one unit at a
time. All units are able to use the fuel tank stations. The truck itself
can not run out of fuel.
Activity: A gasoline truck can perform three different activities.
Inactive (Doing nothing)
Moving
Refilling gasoline
Sending a note by mail is the only way to share information with
members of your team. The mail system has two parts:
The viewer panel: where you receive and read mails.
The editor panel: where you write and send mails.
When you receive a mail, two things happen.
1. The number at the top right-hand corner of the viewer panel
changes. The number indicates how many mails you have
received but not read.
2. The viewer panel becomes pink. You don’t get any other
notification, so keep an eye on the viewer window to see if
you have unread mails.
Click the ‘Next’ button once to open and read incoming the mail. Old
mails cannot be viewed again, they are deleted when you view the
next mail.
Any questions?
Appendix 2
This section is filled in by the experimenter
Group number: ____________
ID (InitialsÅÅMMDD):__________
Background information
Occupation:
__________________________________________________________________________
If student:
What
education/program:_____________________________________________________
Gender: _______________ Age: _______________
How many times have you played C3Fire?
0 1 2 3 4 5+
How much computer experience do you have?
1 2 3 4 5
No experience Plenty of experience
Do you have any previous experience of firefighting?
1 2 3 4 5
No experience Plenty of experience
Have you done any kind of military service?
Yes, title:_______________, for__________months. No
How well do you know the other team members?
1 2 3 4 5
Never seen before Very good friends
1 2 3 4 5
Never seen before Very good friends
How much experience do you have from gaming (computer- and videogames)?
1 2 3 4 5
No experience Plenty of experience
På svenska
Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –
under en längre tid från publiceringsdatum under förutsättning att inga
extra-ordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda
ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat
för ickekommersiell forskning och för undervisning. Överföring av
upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All
annan användning av dokumentet kräver upphovsmannens medgivande. För
att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av
teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som
upphovsman i den omfattning som god sed kräver vid användning av
dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras
eller presenteras i sådan form eller i sådant sammanhang som är kränkande
för upphovsmannens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press
se förlagets hemsida http://www.ep.liu.se/
In English
The publishers will keep this document online on the Internet - or its
possible replacement - for a considerable time from the date of publication
barring exceptional circumstances.
The online availability of the document implies a permanent
permission for anyone to read, to download, to print out single copies for
your own use and to use it unchanged for any non-commercial research and
educational purpose. Subsequent transfers of copyright cannot revoke this
permission. All other uses of the document are conditional on the consent of
the copyright owner. The publisher has taken technical and administrative
measures to assure authenticity, security and accessibility.
According to intellectual property law the author has the right to be
mentioned when his/her work is accessed as described above and to be
protected against infringement.
For additional information about the Linköping University Electronic
Press and its procedures for publication and for assurance of document
integrity, please refer to its WWW home page: http://www.ep.liu.se/
© Nicoletta Baroutsi