Expert Systems With Applications 65 (2016) 271–282

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Visual decision support for business ecosystem analysis

Rahul C. Basole

a , ∗, Jukka Huhtamäki b , Kaisa Still c , Martha G. Russell d

a Tennenbaum Institute & School of Interactive Computing Georgia Institute of Technology 85 Fifth Street NW Atlanta, GA b Tampere University of Technology Department of Mathematics Tampere, Finland c VTT Finland Espoo, Finland d mediaX and H ∗STAR Stanford University Stanford, CA

a r t i c l e i n f o

Article history:

Received 19 April 2016

Revised 13 July 2016

Accepted 10 August 2016

Available online 25 August 2016

Keywords:

Information visualization

Decision support

Business ecosystem

Cognitive fit theory

Data complexity

a b s t r a c t

This study comparatively evaluates the effectiveness of three visualization methods (list, matrix, network)

and the influence of data complexity, task type, and user characteristics on decision performance in the

context of business ecosystem analysis. We pursue this objective using an exploratory study with 14

prototypical users (e.g. executives, analysts, investors, and policy makers). The results show that in low

complexity contexts, decision performance between visual representations differ but not substantially. In

high complexity contexts, however, decision performance suffers significantly if visual representations are

not appropriately matched to task types. Our study makes several theoretical and practical contributions.

Theoretically, we extend cognitive fit theory by investigating the impact of business ecosystem task type

and complexity. Managerially, our study contributes to the relatively underexplored, but emerging area of

the design of business ecosystem intelligence tools and presentation of business ecosystem data for the

purpose of decision making. We conclude with future research opportunities.

© 2016 Elsevier Ltd. All rights reserved.

1

c

f

n

b

f

n

a

w

(

e

c

s

w

s

1

t

w

b

E

d

m

a

t

a

f

s

d

s

i

t

e

p

n

p

t

d

l

h

0

. Introduction

With a rapidly changing business environment, fast product cy-

les, and decreasing average life expectancy of firms, managers are

eeling a sense of urgency to find effective methods and tech-

iques to help understand and manage the complexity of their

usiness ecosystem ( Adner, 2012 ). With the emergence of plat-

orms and increasing digitization of products and services, busi-

ess ecosystems have become an important concept for man-

gers ( Van Alstyne, Parker, & Choudary, 2016 ). Indeed, it is now

ell accepted that companies actually compete in ecosystems

Kelly, 2015 ). Adapted from the biological/ecological sciences, the

cosystem perspective is based on the premise that industries

onsist of a heterogeneous and continuously evolving set of con-

tituents that are interconnected through a complex, global net-

ork of relationships, co-create value, and are co-dependent for

urvival ( Basole & Rouse, 2008; Iansiti & Levien, 2004; Moore,

996; Russell, Huhtamäki, Still, Rubens, & Basole, 2015; Still, Huh-

amäki, Russell, & Rubens, 2014 ).

The visualization of these relationships is an important step to-

ards understanding the complexities and tradeoffs inherent in

∗ Corresponding author.

E-mail addresses: basole@gatech.edu (R.C. Basole), jukka.huhtamaki@tut.fi

(J. Huhtamäki), kaisa.still@vtt.fi (K. Still), marthar@stanford.edu (M.G. Russell).

p

i

t

p

a

ttp://dx.doi.org/10.1016/j.eswa.2016.08.041

957-4174/© 2016 Elsevier Ltd. All rights reserved.

usiness ecosystems ( Basole, Russell, Huhtamäki, & Rubens, 2015;

vans & Basole, 2016 ). Rather than relying strictly on individual in-

icators of the state of the business ecosystem, which many deem

issing and/or insufficient, an interactive visual approach provides

n articulated “wide lens” perspective that can be shared to es-

ablish common ground on which decisions can be based, to cre-

te reference points for trade-off decisions, and to lay a foundation

or exploration, discovery, and analysis ( Thomas & Cook, 2005 ). Not

urprisingly, visual representations of ecosystems are valuable to a

iverse set of user groups, including executives that want to under-

tand their firm’s competitive landscape, venture capitalists seek-

ng investment opportunities, or policy makers examining innova-

ion dynamics ( Basole, Clear, Hu, Mehrotra, & Stasko, 2013; Still

t al., 2014 ).

Despite the well understood general notion that visual ap-

roaches can aid decision makers, little empirical evidence exists

ow as to how and to what extent visual representations can sup-

ort individuals and enhance decision making for business ecosys-

em intelligence tasks. Such tasks differ from ”normal” business

ecisions in that they can include understanding of the overall re-

ational structure as well as identification of clusters and complex

atterns embedded in ecosystem relationships. Moreover, there

s a particular lack of understanding how underlying ecosystem,

ask, and user characteristics influence the decision quality. Cor-

orate decision makers often have differing levels of visual literacy

nd diverging analysis tool comfort, preferences, and expectations

272 R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282

Fig. 1. Theory of cognitive fit.

e

d

o

v

w

(

&

r

i

p

i

r

s

t

o

t

(

p

t

2

s

p

G

c

(

v

m

p

a

i

B

a

a

o

p

e

e

c

s

v

(

D

o

s

e

l

2

( Basole, 2014 ). Further, current tools used by these decision mak-

ers are quite basic, ranging from simple reports to spreadsheets.

More sophisticated representations of (e.g. multipartite networks,

geocoded maps, flow diagrams, etc.) and approaches for business

ecosystem data are just emerging ( Basole et al., 2013; Huhtamäki,

Russell, Rubens, & Still, 2015 ). Based on these realities, it is thus

pertinent to first independently evaluate the effectiveness of each

visual representation under different ecosystem analysis contexts

(e.g. size, scope, and complexity) for these types of users.

The objective of this study is to comparatively evaluate the use

and effectiveness of three frequently employed visualization tech-

niques — lists, matrices, and networks — for making decisions that

are informed by complex business ecosystem relationships. We

pursue this objective using an exploratory study with prototypical

users in a laboratory setting.

Our study makes several theoretical and practical contributions.

Theoretically, we extend theories of cognitive and task-technology

fit and investigate the impact of ecosystem data and task com-

plexity. Managerially, our study contributes to the relatively un-

derexplored, but emerging area of the design of business ecosys-

tem intelligence tools and presentation of ecosystem data for

the purpose of decision making. In doing so, our study serves

as an important benchmark study for visual business ecosystem

analysis tool design using participants with significant decision

experience.

The remainder of this study is structured as follows.

Section 2 presents the theoretical background and research model.

Section 3 describes our research methodology, including ecosystem

analysis tasks, performance metrics, and design and execution of

our exploratory study. Results are presented and discussed in

Section 4 . Section 5 presents implications. Section 6 concludes the

study.

2. Theory and research model

2.1. Theory of cognitive fit

Based on information processing, decision-making, and cost-

benefit theory, the theory of cognitive fit was developed to help

understand how the fit between presentation format and decision-

making task influences an individual’s problem-solving perfor-

mance ( Vessey & Galletta, 1991a ). In particular, cognitive fit theory

argues that the performance of problem-solving depends on both

the information format (e.g. problem representation) and the na-

ture of the task. Different information formats, such as tables and

graphs, tend to emphasize different problem-solving tasks, such as

pattern detection or value retrieval, and different problem-solving

processes. Cognitive fit theory further suggests that when the

problem representation fits the problem-solving task, a preferable

and more consistent mental representation of the problem will be

realized, thereby facilitating the problem solving process, and con-

sequently resulting in preference for the representation, along with

faster and more accurate performance in decision-making. How-

ver, when a mismatch occurs (i.e. there is a lack of cognitive fit)

ecision-making performance suffers in terms of speed, accuracy,

r both (see Fig. 1 ).

The theory of cognitive fit has been applied and empirically

alidated in a variety of problem domains, including consumer

eb behavior ( Hong, Thong, & Tam, 2004 ), financial decisions

Frownfelter-Lohrke, 1998 ), complex managerial decisions ( Speier

Morris, 2003 ), software comprehension ( Shaft & Vessey, 2006 ),

equirements analysis ( Agarwal, Sinha, & Tanniru, 1996 ), and open

dea sourcing ( Blohm, Riedl, Füller, & Leimeister, 2016 ). Table 1

rovides a summary of prior salient work including contexts stud-

ed, visual representations used, and key findings. We limit our

eview to studies published in leading information systems, deci-

ion sciences, and human-computer interaction journals from 1990

o 2015 and exclude conference papers, books, and articles. Based

n the extant body of research, assessment of potential alterna-

ive theoretical explanations, and our own practical experience

Walsham, 2006 ), we conclude that cognitive fit theory is an ap-

ropriate theoretical lens for understanding the relationship be-

ween information formats and ecosystem analysis tasks.

.2. Visual representation of business ecosystem data

Visualization of complex data enables decision makers to

ee patterns, spot trends, and identify outliers and thereby im-

rove comprehension, memory and decision making ( Tufte &

raves-Morris, 1983 ). Visualization can make data more ac-

essible and provides a method for improved communication

Shneiderman, 1996 ). It has also been shown that well-designed

isualizations can improve comprehension, memory, and decision

aking, critical in the exploration, discovery, and analysis of com-

lex problems ( Thomas & Cook, 2005 ). The challenge, and thereby

rt and science of visualization, is to create effective and engag-

ng visual representations that are appropriate to the data ( Heer,

ostock, & Ogievetsky, 2010; Heer & Shneiderman, 2012 ).

There are many different forms of visual representation suit-

ble for business data. In addition to standard 2D-techniques such

s x − y plots, bar charts, and line graphs, there are also many

ther sophisticated visualization techniques Keim (2002) . A com-

rehensive review is beyond the scope of this paper, but inter-

sted readers are referred to ( Heer et al., 2010 ) who provide a

xamples of salient visualization techniques, including time-series

harts, stacked graphs, small multiples, maps, cartograms, matrices,

unbursts, node-link diagrams, and networks.

While there is a growing recognition of the potential value of

isualization in the business, strategy and innovation communities

Few, 2009; Huhtamäki, Russell, Still, & Rubens, 2011; Soukup &

avidson, 2002; Tegarden, 1999; Wright, 1997 ), there is a dearth

f studies evaluating the utility and effectiveness of different vi-

ualization approaches ( Zhu, 2007 ). The visualization of business

cosystems poses particular challenges as the underlying data is

arge, multi-level, multivariate, and often uncertain ( Basole et al.,

015 ). While there is initial evidence of the value and impact of

R.C

. B

aso

le et

al. / E

xpert

System

s W

ith A

pp

licatio

ns 6

5 (2

016

) 2

71

–2

82

27

3

Table 1

Summary of prior work using cognitive fit theory (1990–2015). Ordered chronologically.

Reference Context Data representation Participants Dependent variable CFT

Acad Ind Other Time Accuracy Other Support?

Vessey and Galletta (1991b) Bank account management Graphs vs. tables x x x Partial

Sinha and Vessey (1992) Programming languages LISP vs. PASCAL x x Partial

Umanath and Vessey (1994) Bankruptcy decision-making Schematic faces vs. graphs vs. tables x x x Yes

Agarwal et al. (1996) Requirements modelling Process vs. object-oriented tools x x Partial

Smelcer and Carmel (1997) Geographic information systems Tables vs. maps x x Yes

Dennis and Carte (1998) Geographic information systems Tables vs. maps x x x x Partial

Frownfelter-Lohrke (1998) Financial statements Graphs vs. tables vs. hybrid x x x x No

Hubona, Everett, Marsh, and Wauchope (1998) Language-conveyed spatial information Route vs. survey oriented text x x x Partial

Tuttle and Kershaw (1998) Employee performance evaluations Graphs vs. tables x x x Yes

Chandra and Krovi (1999) Information retrieval Networks vs. OO representation x x x Partial

Mennecke, Crossland, and Killingsworth (20 0 0) Spatial decision support SDSS vs. paper maps x x x x Partial

Dunn and Grabski (2001) Accounting models DCA vs. REA models x x x x Partial

Speier and Morris (2003) Interruptions Graphs vs. tables x x x Yes

Mahoney, Roush, and Bandy (2003) Decision making Probability density vs. tables x x Yes

Hong et al. (2004) Online shopping List vs. matrix x x x Partial

Khatri, Vessey, Ram, and Ramesh (2006) Conceptual modelling ER vs. EER modeling x x Yes

Shaft and Vessey (2006) Expertise management & visualization Tables vs. self-organizing maps vs. MDS x x Partial

Shaft and Vessey (2006) Software maintenance Accounting vs. Hydrology COBOL program x x Yes

Speier (2006) Operations management Graphs vs. tables x x x Partial

Cardinaels (2008) Accounting-based costing Graphs vs. tables x x No

Goswami, Chan, and Kim (2008) Spreadsheets Spreadsheet without vs. with visualization x x x Partial

Kamis, Koufaris, and Stern (2008) Product customization online Attribute-based vs. alternative-based DSS x x Yes

Urbaczewski and Koivisto (2008) Bank account management Graphs vs. tables x x x Partial

Brunelle (2009) Consumer channel preference Bricks-and-mortar vs. online store x x Partial

Teets, Tegarden, and Russell (2010) Preduction- quality assurance 2D graphs vs. 3D graphs vs. tables x x x Partial

Adipat, Zhang, and Zhou (2011) Mobile device (search tasks) Tree view vs . hierarchical text x x x x Yes

Chan, Goswami, and Kim (2012) Spreadsheets A1 vs. C1R1 problem presentation x x x Yes

Shen, Carswell, Santhanam, and Bailey (2012) Emergency Management Plan vs. elevation vs. 3D display x x x x Yes

van der Land, Schouten, Feldberg, van den Hooff,

and Huysman (2013)

3D Virtual environments 2D vs. 3D static vs. 3D immersive floorplans x x Yes

Xu, Chen, and Santhanam (2015) e-Commerce Product Reviews Text vs. Image vs. Video x x Yes

Li, Wei, Tayi, and Tan (2015) Online product presentation Text vs. Visual x x Yes

This Study Ecosystem analysis Lists vs. matrices vs. networks x x x Yes

274 R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282

w

a

t

d

c

i

i

H

t

i

m

n

t

n

(

s

t

c

a

o

s

b

p

c

e

t

m

r

w

e

h

t

3

3

u

d

f

(

t

B

b

G

t

f

w

interactive business ecosystem visualizations ( Basole, 2009; Basole

et al., 2013; Basole, Hu, Patel, & Stasko, 2012; Basole et al., 2015;

Russell, Still, Huhtamäki, Yu, & Rubens, 2011; Still et al., 2014 ), to

the best of our knowledge there are no studies that have empir-

ically explored and validated their effectiveness on complex deci-

sion making.

2.3. Effectiveness of business ecosystem visualizations

Measuring the effectiveness of decision support systems (DSS)

is an important topic in information systems research ( Burton-

Jones & Straub, 2006; Davis, 1989; Sharda, Barr, & MCDonnell,

1988; Todd & Benbasat, 1992 ). Through a meta-study Hung et al.

( Hung, Ku, Liang, & Lee, 2007 ) evaluated two common categories

of DSS effectiveness, namely process- and outcome-oriented mea-

sures. Process-oriented measures include frequency or length of

system usage; outcome-oriented measures include decision perfor-

mance and satisfaction.

Evaluations of visual DSS effectiveness are particularly challeng-

ing ( Morse, Lewis, & Olsen, 20 0 0; North, 20 06; Shneiderman &

Plaisant, 2006; Tory & Moller, 2005 ) and are generally conducted

through either expert assessments or user studies ( Anderson et al.,

2011 ). In our study we follow an expert participant approach and

use one measure from each category to measure visual DSS suc-

cess. Our process-oriented measure is efficiency as measured by

the decision speed of the user. Our outcome-oriented measure is

decision accuracy.

2.4. Research model and propositions

A key tenet of cognitive fit theory is to explicate how and

why a particular problem representation format fits a particu-

lar task. This demands a theoretical differentiation of the vi-

sual representations and tasks. Existing classifications of visualiza-

tions primarily focus on the nature of the data (e.g. continuous,

qualitative, etc.) ( Card & Mackinlay, 1997; Chi, 20 0 0 ), the tasks

(e.g. low-level, high-level, problem-solving, information acquisition,

etc.) ( Shneiderman, 1996 ), or the algorithms that created it ( Tory &

Moller, 2004 ).

We focus on three common data representations in our study —

list, matrix, and network. We selected these three representations

following an extensive field study, interviews, and our experiences

with decision makers and their particular decision making tasks

and environments. Given that relationships and networks are par-

ticularly important in business ecosystem analyses, we chose visual

representations that are typically used for connection/relational

data. List views are very well established; matrix and networks are

extensively used for graph data and well known too.

The most appropriate way of theoretically differentiating these

three representations is to adapt and extend Vessey and Galletta’s

conceptualization ( Vessey & Galletta, 1991b ), which argues that

representations and tasks are either spatial or symbolic in nature.

Spatial tasks, those that lead to assessing the problem area as a

whole, are facilitated by the use of graphs; symbolic tasks, those

that lead to precise data values, are facilitated by the use of ta-

bles lists ( Huang et al., 2006 ). Our study uses one symbolic (list)

and two different types of spatial representation techniques (ma-

trix and network).

Matrix and network representations share many similarities and

are highly overlapping in the problem-solving processes they can

support ( Novick & Hurley, 2001 ). They both provide insight into

the global structure and the building blocks of relational data.

For the matrix, the building block unit is a cell denoting the in-

tersection of two elements. For the network, the building block

unit is two nodes and some type of link between them. How-

ever, they differ in a few functional ways that explicitly suggest

hat tasks they are best for suited for. While both representations

re well suited to depict links and link types, the identification of

he presence/absence of a link is faster with a matrix due its or-

ered structure ( Bertin, 1981 ). The orderable nature of rows and

olumns in matrices also lends itself to identifying sets and regions

n the data; the same effect, but more challenging, can be achieved

n network representations using clustering algorithms ( Novick &

urley, 2001 ).

Perhaps the most distinguishing feature of networks over ma-

rices is their ability to traverse chains of links (i.e. connectiv-

ty/paths). Paths are naturally evident in networks, but not in

atrices. In matrices, users must identify a cell, locate its con-

ected cells by scanning through rows and columns, and continue

his process until they reach the final cell. This process is cog-

itively more resource-intensive than following a series of edges

Lee, Plaisant, Parr, Fekete, & Henry, 2006 ). These observations

uggest there is a subtle but significant difference between our

wo spatial representations. Matrices aid particularly with spatial-

onnectivity tasks (i.e., presence and absence of links); networks

id particularly with spatial-traversal tasks (i.e., identifying paths

f connectivity).

Based on the preceding discussion, we build the following re-

earch model (see Fig. 2 ). We posit that the association between

usiness ecosystem visual representation, task type, and decision

erformance — measured in terms of decision accuracy and de-

ision time — is moderated by the complexity of the business

cosystem. We further argue that the relationship can be differen-

iated by two user characteristics: perceived task load and decision

aking style. Following cognitive fit theory, we thus propose two

esearch propositions: Proposition 1: Decision time will be shorter

hen the business ecosystem visualization format matches the

cosystem analysis task. Proposition 2: Decision accuracy will be

igher when the business ecosystem visualization format matches

he ecosystem analysis task.

. Method

.1. Identification of business ecosystem analysis tasks

In order to define realistic tasks for our experimental study, we

sed a two-step approach. First, we identified business ecosystem

ecision making tasks from the literature. In particular, we drew

rom work on graph visualization and task taxonomy identified by

Lee et al., 2006 ) as well as studies on business ecosystem analysis

hat outlined common tasks decision makers pursue ( Basole, 2014;

asole et al., 2013; Basole et al., 2012 ). This literature suggests four

road task types.

• Attribute-based tasks are concerned with identifying (or filter-

ing for) attributes of both nodes and links in business ecosys-

tems. • Topology-based tasks are concerned with the topology (i.e.

structure) of the business ecosystem, including adjacency, ac-

cessibility, and connectivity. • Browsing tasks are concerned with following a path or revisit-

ing nodes in the business ecosystem. • Overview tasks are concerned with estimating the overall size

and structure of the business ecosystem as well as identifying

patterns and outliers.

Following the theoretical differentation provided by ( Vessey &

alletta, 1991b ), attribute-based tasks are thus symbolic, while

opology, overview, and browsing are considered spatial tasks.

Second, we conducted interviews with six experts that per-

orm business ecosystem analyses for their work to understand

hat types of tasks they perform and how their tasks map to task

R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282 275

Fig. 2. Research model.

c

p

W

w

t

f

m

o

b

t

v

fi

b

B

i

3

u

c

s

t

s

r

c

i

p

3

o

d

t

w

s

t

t

(

b

s

n

d

i

y

&

m

t

t

f

w

C

w

p

c

t

h

e

s

t

3

t

ategories identified in the literature. All interviews were done in-

erson except for two experts, which were conducted by phone.

e used an informal interview procedure covering the intervie-

ees’business ecosystem analysis experience, the applications and

ypes of visualizations they use, and their common tasks.

The experts further articulated that they commonly performed

our types of tasks: (1) find a fact (2) determine a pattern, (3)

ake a decision, and (4) a compound task that could combine any

f these elements, both individually and with other team mem-

er(s).

Using these findings from the literature review, the expert in-

erviews, and feedback from two expert faculty members, we de-

ised four tasks for the individual session ( T 1 − T 4 ):

• ( T 1) What is the expertise of firm i ? ( Attribute-based task ) • ( T 2) Which firm connected to firm j has the highest trustwor-

thiness? ( Topology-based task ) • ( T 3) Identify the most interconnected segment. ( Overview task ) • ( T 4) Identify a collaboration path that connects firm k and firm

l . ( Browsing task )

In our study, expertise represents the technical know-how of a

rm. Trustworthiness represents the extent to which the firm can

e trusted to deliver what it promises, is reliable, and benevolent.

oth of these attributes are considered critical firm characteristics

n business ecosystems.

.2. Performance of business ecosystem analysis tasks

We quantitatively evaluate the outcomes using two commonly

sed task performance measures: decision time and decision ac-

uracy ( Vessey & Galletta, 1991a ). Decision time is measured in

econds from the time when the participant began working on

he task until she submitted her final answer verbally. Deci-

ion accuracy is measured using a binary response variable (cor-

ect/incorrect). We should note that some tasks may have multiple

orrect answers (e.g. there may be multiple correct paths connect-

ng two firms); we do not distinguish correct answers by the best

ossible answer.

.3. Creation of synthetic business ecosystems

Next, we had to develop realistic synthetic business ecosystems

n which users had to perform the aforementioned tasks. At a fun-

amental level, business ecosystems consist of sets of firms ( N )

hat are connected by relationships ( R ). Each firm is associated

ith an ecosystem segment (e.g. software, hardware, etc.) and has

everal attributes (e.g. expertise, trustworthiness, etc.). Moreover,

he structure of business ecosystems is often scale-free, implying

hat the underlying relationship distribution follows a power law

Barabási, 2009 ).

Taking these design parameters into account, our synthetic

usiness ecosystems contained the following information. Readers

hould note that we purposely did not use actual firm or segment

ames as domain knowledge could have potentially contributed to

ecision bias.

• The firm name , represented by a firm ID (e.g. Firm1, Firm2,

etc.) • The segment of the firm. We used colors to represent different

segments (e.g. yellow, red, blue, etc.) • The total number of collaboration partners of a firm. This

number indicates how many firms a focal firm is connected to. • The number of collaborations with a specific partner. This

number indicates the strength of relationship between two

firms. For instance, two firms may have multiple collaborative

relationships (e.g. supply, marketing, R&D). • The expertise level of a focal firm, ranging from 0 (low) to 100

(high). • The trustworthiness level of a focal firm, ranging from 0 (low)

to 100 (high).

We generated the underlying business ecosystem structure us-

ng NetworkX, a Python language package for exploration and anal-

sis of complex networks and network algorithms ( Hagberg, Swart,

Schult, 2008 ) and used a random uniform distribution to deter-

ine expertise and trustworthiness levels for each firm. We used

wo levels of business ecosystem complexity C for each of the four

asks ( T 1 − T 4 ). Business ecosystem complexity is based on an in-

ormation theoretic based measure of entropy ( Shannon, 1948 ) and

as computed as follows:

= −n ∑

i =1

p(x i ) log 2 p(x i ) (1)

here n is the number of firms in the ecosystem and p(x i ) is the

robability of relationships that firm i is involved in. The unit of

omplexity resulting from this equation is binary digits, or bits. In-

uitively, it represents the number of binary questions one would

ave to ask and have answered to determine the state of the

cosystem ( Basole & Rouse, 2008 ). This led to the creation of two

ynthetic ecosystems, E 1 and E 2, one with low ( n = 50 ; C = 13 . 0 ),

he other with high complexity ( n = 100 ; C = 21 . 8 ).

.4. Design and development of experimental visualizations

We implemented the three visual representations — list, ma-

rix, and network — as native web-based visualizations as shown

276 R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282

Fig. 3. Sample business ecosystem analysis views: (a) list, (b) matrix, and (c) network.

W

i

w

t

c

p

q

s

t

s

B

p

3

a

c

r

c

t

s

t

e

g

t

e

b

O

w

h

t

a

t

w

c

(

i

a

t

i

3

p

a

t

a

s

q

in Fig. 3 . The list view is a simple HTML document augmented

with sorting functionality implemented using the jQuery’s table-

sorter plugin. The matrix and network visualizations were imple-

mented using D3.js, a JavaScript library for creating data-driven vi-

sualizations using HTML5, CSS3, and SVG ( Bostock, Ogievetsky, &

Heer, 2011 ). We chose D3 as our platform as it enables fast and

effective development of interactive and animated visualizations

and allowed us to implement consistent features and functional-

ities across all three visualization representations.

• The list view presents information in table (or list) form. Each

column contains a data element. If the list is long, users can

scroll down to see additional entries. Users can sort in ascend-

ing or descending order by clicking on the column label or on

the arrows. • The matrix view is a symmetric table that shows the firms

in the rows and columns. The row and column labels show

the firm ID. Hovering over the firm ID labels gives the user

detailed information on collaboration partners, expertise, and

trustworthiness level for that firm. The color of the main di-

agonal cells indicates the firm’s segment. The off-diagonal cells

show the collaborations between firms. If the color is the same

shade as the main diagonal, that means the two firms are in

the same segments. If the color is a shade of gray, that means

the two firms are connected but come from different segments.

The darker the cell shading, the more collaborations exist be-

tween the two firms. Hovering over a cell also provides infor-

mation of the number of collaborations between the two firms.

The default sorting of labels is by firm ID. Users can also sort

the matrix by the different attributes using a drop-down menu.

The sort order is from top-left down. A simple animation sorts

the rows/columns. • The network view depicts the collaboration network structure

of the business ecosystem. Nodes represent firms, edges repre-

sent collaboration between firms. Nodes have a firm ID label

and the color of the node indicates the firm segment. The edge

thickness corresponds to the number of collaboration between

the two firms. Firms that are well-connected (i.e. are collabo-

rating with many firms) are positioned more centrally; those

not are more on the periphery of the network. Hovering over

a firm node emphasizes its collaboration partners, graying out

those the firm is not collaborating with and showing detailed

information about the firm. The node size is correlated to the

level of attribute selected in the dropdown menu. The default

is total number of collaboration partners.

3.5. Pre-testing

Prior to use in the laboratory sessions, the software functional-

ity was extensively tested by four members of the research team.

e also conducted a comprehensive pilot test of the tasks, visual-

zations, and experimental system with three representative users

ith significant industry experience. These expert users reviewed

he task description and confirmed that content reflected current

ommercial practice and language. Feedback and comments were

rovided in face to face meetings. Our expert users specifically re-

uested a refinement of the tutorial, some task wording and pre-

entation, as well as minor improvements in visualization interac-

ions. We integrated their suggestions into our final experimental

ystem design. The final design was presented to the expert users.

oth form and content of our visualizations were considered com-

rehensive and realistic, suggesting a high level of face validity.

.6. Participants

Typical users of business ecosystem visualizations include an-

lysts (e.g. market, technology, policy, intelligence), corporate de-

ision makers, venture capitalists and investors, and management

esearchers ( Basole et al., 2013; Still et al., 2014 ). We therefore fo-

used our participant recruitment on these types of users. In to-

al, we recruited 14 participants (2 females, 12 males) for a 1.5 h

ession. All participants were seasoned decision makers from the

echnology sector with a minimum of 15 years executive experi-

nce; between 49 and 69 years of age; with diverse cultural back-

rounds (North/South American, Asian, and European); 50% in en-

erprise or startup businesses, with the remainder split between

ducation, government and consulting. While we strived for gender

alance, unfortunately, the majority of our participants were male.

ur sample did not include any digital natives, as our participants

ere not university student participants. However, all participants

ad prior experience and expertise in using computers. Participa-

ion was voluntary and no compensation was provided. We fully

cknowledge that our sample size may be perceived small in con-

rast to other social science experimental studies. However, prior

ork has shown that small participant samples are very much ac-

eptable in expert user interface and design evaluation settings

Virzi, 1992 ). It is also in line with guidelines provided by visual-

zation researchers ( Carpendale, 2008 ). Given that our participants

re all executives with significant experience and representative of

ypical ecosystem analysis users, we are confident that our partic-

pant population and size is appropriate.

.7. Procedure

Before the experiment, participants were asked to fill out a

re-study survey collecting basic demographic information. Next,

brief hands-on interactive tutorial of the system and each of

he three visualization techniques were provided. Participants were

sked to perform the steps the instructor illustrated on the large

creen on their computer system and encouraged to ask clarifying

uestions on visualization entities, renderings, and color encodings.

R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282 277

A

o

f

e

d

w

r

w

s

o

(

t

t

p

a

m

s

s

W

q

c

t

l

t

a

t

N

d

t

r

r

w

c

t

a

3

l

o

h

u

v

t

V

W

m

r

m

n

l

i

w

t

d

l

r

t

o

T

t

4

t

a

A

s

p

–

c

d

f

p

t

d

a

t

r

n

t

u

t

t

t

t

r

r

p

p

t

r

a

c

d

n

l

t

p

n

h

n

i

b

s

t

a

c

T

b

d

t

(

(

printed tutorial document was provided for reference. At the end

f the tutorial, we made sure that the participant understood the

eatures of each view and was ready to start the individual session

valuation. When necessary, the instructor proposed to repeat the

emonstration again. At the end of the tutorial, three instructions

ere given:

• The participant has to answer each question as quickly as pos-

sible. • The participant should state her final answer out loud. • The participant is allowed to move to the next question without

answering before the answer time elapses in case she felt she

was not able to answer.

Participants were asked to answer 24 questions in total (3 rep-

esentations × 2 complexity levels × 4 tasks). Views and questions

ere presented in random order to avoid memorization biases. The

ystem begins with the list view and then selects a the matrix

r network view at random, asking the user to answer each task

T 1 − T 4 ) for each ecosystem ( E1 − E2 ). Then, the system moves

o the second representation technique and does the same. By in-

erchanging the representation techniques, we make sure that the

articipants have the same probability to start the evaluation with

series of visualizations belonging to either technique. Further-

ore, we also varied the entity details (i.e. firm ID, attribute, and

egment) for questions across views to avoid learning bias.

Since ecosystem analysis tasks in real-world contexts are time

ensitive, we chose to limit the answer time to 60 s per question.

hen time runs out, the system automatically moves to the next

uestion and produces a visual feedback to notify the user. In this

ase, we consider that the representation is not effective for that

ask since the user was not able to provide the answer in the al-

otted time.

We concluded the experiment with a post-study survey con-

aining questions regarding usability and usefulness of views and

n assessment of decision making characteristics and perceived

ask load based on the NASA-TLX protocol ( Hart & Staveland, 1988 ).

ASA-TLX is a well-established, multi-dimensional ratings proce-

ure that includes subscales (mental demands, physical demands,

emporal demands, performance, effort and frustration) and de-

ives an overall workload score based on a weighted average of

atings on based on the six subscales. It has been used to assess

orkload in various human-machine environments such as aircraft

ockpits, command, control, and communication (C3) worksta-

ions; supervisory and process control environments; simulations

nd laboratory tests. The instrument is provided in Appendix A .

.8. Apparatus and facility

Participants performed the study using an Apple Macbook Pro-

aptop computer with a 2.2 GHz Intel Core i7 processor and 8GB

f 13333 MHz DDR3 memory. Given the likelihood that we could

ave non-Apple users, we provided each participant the option to

se a standard (PC) mouse for interaction. Canon Vixia HF11 HD

ideo recorders were placed on tripods to record frontal and sagit-

al views of the experiment for later data collection and analysis.

ideos were organized and analyzed using Canon Image software.

e also used Silverback 2.0 usability testing software to record

ouse movement, clicks, and other UI interactions as well as user’s

eactions as audio and video. The Safari web browser window was

aximized (with no tool or address bars showing) to avoid extra-

eous onscreen stimuli. No additional applications were running.

We conducted the study in the Stanford University Peter Wal-

enberg Learning Theater (PWLT). PWLT is a state-of-the-art learn-

ng and teaching facility with a 8 × 32 ft seamless curvilinear

all, composed of a 8 × 24 array of Christie MicroTiles. The sys-

em is used as one massive display or with 16 separate channels of

isplay activity. The system can be used for education, design, col-

aboration, augmented decision making, multimedia art, and telep-

esence communications, and was particularly suitable for our tu-

orial session. The facility enabled us to observe participants un-

btrusively from multiple angles, including from an elevated view.

he experimental setup and facility is shown in Fig. 4 . Sample pho-

os of the experimental sessions are shown in Fig. 5 .

. Experimental results and discussion

We used a repeated-measures MANCOVA approach to analyze

he data, using visual representation as the between-subject factor

nd task type and ecosystem complexity as within-subject factors.

ll three main effects were statistically significant – visual repre-

entation ( F = 8.525 (1,26), p = .003), task type ( F = 39.587 (1,26),

= .0 0 0), and ecosystem complexity ( F = 9.739 (1,26), p = .003)

and in the expected directions, confirming our expectations and

onsistent with prior results.

While MANCOVA assesses effects on an aggregated measure of

ecision performance, we used parameter estimates to test the ef-

ects of the between- and within-subject variables on our two de-

endent variables (decision time and decision accuracy) for each of

he tasks. Parameter estimates are the planned contrasts used to

ifferentiate among each of the dependent variables investigated

nd are reported using a t -value, transformed from univariate F -

ests ( Speier, 2006 ).

Table 2 provides a summary of our experimental findings. The

esults show that no single visual presentation technique was sig-

ificantly superior to the others across all task types and ecosys-

em complexity levels. However, the results do suggest that partic-

lar types of tasks were more suitable for certain visual represen-

ations by complexity level.

Proposition 1 states that decision accuracy is the highest when

he representation matches the task, while Proposition 2 states

hat decision time is the lowest when the representation matches

he task. Decision accuracy is indeed higher using the symbolic

epresentation (0.93) for the attribute task ( T 1) than both spatial

epresentations (matrix (0.86) and network (0.71)) in the low com-

lexity case, but the differences are significant (( t (26) = 0.372;

= 0.713); ( t (26) = 0.257; p = 0.799). Decision times for T 1 in

he low complexity case were quite similar across all three visual

epresentations (list: 25.93 s; matrix: 23.93 s; network: 24.50 s)

nd we did not observe any significant differences.

These observations did not hold true for T 1 in the high

omplexity case. The spatial representation (list) has the fastest

ecision time (11.50 s) compared to the matrix (21.71 s) and

etwork (40.00 s). All pairwise comparison are significant. The

ist representation was significantly better than both the ma-

rix ( t (26) = 2.577; p = 0.016) and the network ( t (26) = 4.751;

= 0.0 0 0). The matrix also performed significantly better than the

etwork ( t (26) = 2.240; p = 0.024). In terms of decision accuracy,

owever, the list representation is significantly different only to the

etwork view ( t (26) = 2.104; p = 0.045).

For the three spatial tasks ( T 2 − T 4 ), we observe several signif-

cant, but counter-intuitive differences between spatial and sym-

olic representation formats for both complexity contexts. Deci-

ion time for T 2 in the low complexity context is the shortest for

he list view (49.07 s), followed by the network view (49.64 s)

nd the matrix view (56.55 s). However, we only find a signifi-

ant difference between list and matrix ( t (26) = 2.129; p = 0.043).

he list view also has the highest decision accuracy (0.79) than

oth matrix (0.29) and network (0.57), but only has a significant

ifference with the matrix ( t (26) = 2.313; p = 0.029). For T 2 in

he high complexity case, we observe similar results. The list view

0.71) has a significantly higher decision accuracy than the matrix

0.14)( t (26) = 3.625; p = 0.001) and network (0.57)( t (26) = 2.290;

278 R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282

Fig. 4. Experimental setup, panorama view of the PWLT and tutorial session, and video recording of session.

Fig. 5. Experimental session.

Table 2

Summary of experimental findings.

Complexity level Representation T1 T2 T3 T4

Time Accuracy Time Accuracy Time Accuracy Time Accuracy

μ σ 2 μ σ 2 μ σ 2 μ σ 2 μ σ 2 μ σ 2 μ σ 2 μ σ 2

High List 11 .50 4 .17 0 .93 0 .26 46 .00 10 .06 0 .71 0 .45 38 .21 13 .59 0 .00 0 .00 50 .00 17 .40 0 .00 0 .00

Matrix 21 .71 13 .63 0 .79 0 .41 59 .15 1 .66 0 .14 0 .35 45 .93 13 .76 0 .143 0 .35 56 .00 13 .27 0 .00 0 .00

Network 40 .00 16 .98 0 .57 0 .49 48 .14 14 .23 0 .36 0 .48 39 .57 14 .44 0 .00 0 .00 56 .31 9 .79 0 .29 0 .45

Low List 25 .93 10 .40 0 .93 0 .26 49 .07 13 .49 0 .79 0 .41 40 .86 16 .36 0 .57 0 .49 52 .00 15 .23 0 .00 0 .00

Matrix 23 .93 14 .60 0 .86 0 .35 56 .55 5 .88 0 .296 0 .45 39 .71 7 .37 0 .36 0 .48 57 .62 7 .42 0 .00 0 .00

Network 24 .50 16 .60 0 .71 0 .45 49 .64 13 .85 0 .57 0 .49 26 .86 12 .82 0 .79 0 .41 50 .31 12 .85 0 .43 0 .49

t

t

i

r

t

(

t

t

w

o

c

d

o

e

c

a

i

t

a

p

i

c

p = 0.030). There are no significant differences between network

and matrix. In terms of decision time, the list view (46.00 s) is sig-

nificantly faster than the matrix (59.15 s)( t (26) = 4.081; p = 0.0 0 0)

but not the network (48.14 s)( t (26) = 0.439; p = 0.664). We also

find that the network view has a significantly better decision time

than the matrix ( t (26) = 2.390; p = 0.024).

For the low complexity overview task T 3, the network view

(26.86 s) has a significantly faster decision time compared

to the list (40.86 s)( t (26) = 2.227; p = 0.035) and the matrix

(39.71 s)( t (26) = 2.675; p = 0.013). In terms of decision accuracy,

we find that the network (0.79) is only significantly better than

the matrix (0.36)( t (26) = 2.586; p = 0.016). We find no significant

differences for decision time or decision accuracy in the high com-

plexity context.

For the low complexity browsing task T 4, the network

view (0.43) has a significantly higher decision accuracy

than the list (0.00)( t (26) = 2.918; p = 0.007) and the ma-

trix (0.00)( t (26) = 2.894; p = 0.008). The same results hold

also for the high complexity context, where the network

view (0.29) has a significantly higher decision accuracy than

the list (0.00)( t (26) = 2.091; p = 0.046) and the matrix

(0.00)( t (26) = 2.073; p = 0.048). We find no significant dif-

ferences for decision time in either complexity context.

Table 3 provides a summary of the significant differences in

he pairwise comparison tests between our three visual represen-

ations. Our results confirm that the symbolic representation (list)

s particularly effective for attribute-based tasks. Among the three

epresentations, the spatial-traversal representation (network) was

he least effective for attribute-based tasks. While both symbolic

list) and spatial-connectivity (matrix) representations were par-

icularly poor for browsing tasks, the spatial-traversal representa-

ion (network) in contrast was particularly effective for browsing as

ell as overview-based decision tasks. Table 4 provides a summary

f average decision performance by visual representation, task, and

omplexity level. Several things can be observed. First, the average

ecision accuracy decreases rapidly from T 1(80%) to T 4(12%). An-

ther interesting observation of our study is the impact of business

cosystem complexity on decision making performance. In a low

omplexity context, the average decision time was 41.4 s with an

ccuracy rate of 52% across all tasks. In a high complexity context,

n contrast, average decision time remained constant (42.7 s), but

he accuracy rate dropped to 33%. In particular topology, browsing

nd overview-based tasks suffered when moving to a high com-

lexity business ecosystem. One counter example, however, is the

mprovement in decision time for attribute-tasks from low to high-

omplexity business ecosystem using the symbolic representation

R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282 279

Table 3

Summary of significant differences in pairwise comparison tests (List (L); Matrix(M); Network (N)).

Low complexity High complexity

T1 T2 T3 T4 T1 T2 T3 T4

Time – L > M (p = . 043) N > L (p = . 035) – L > M (p = . 016) L > M (p = . 0 0 0) – –

N > M (p = . 013) L > N (p = . 0 0 0) N > M (p = . 024)

N > M (p = . 016)

Accuracy – L > M (p = . 029) N > M (p = . 016) N > L (p = . 046) L > N (p = . 045) L > M (p = . 001) – N > L (p = . 046)

N > M (p = . 048) L > N (p = . 030) N > M (p = . 048)

Table 4

Summary of average decision performance by representation, task, and complexity.

Decision performance Visual representation Task type Complexity

List Matrix Network T1 T2 T3 T4 Low High

Avg. time (seconds) 39 .2 45 .1 41 .9 24 .6 51 .4 38 .5 53 .7 41 .4 42 .7

Avg. accuracy (%) 49 .0 32 .0 46 .0 80 .0 48 .0 31 .0 12 .0 52 .0 33 .0

Table 5

Influence of task load index and decision tool comfort.

TLI/CDT level List Matrix Network

Time Accuracy Time Accuracy Time Accuracy

μ σ 2 μ σ 2 μ σ 2 μ σ 2 μ σ 2 μ σ 2

Low/low 45 .42 17 .25 0 .46 0 .49 47 .64 16 .15 0 .21 0 .31 46 .08 13 .89 0 .29 0 .38

Low/high 35 .88 17 .48 0 .50 0 .48 44 .56 17 .10 0 .31 0 .44 38 .19 14 .54 0 .50 0 .48

High/low 35 .63 17 .05 0 .53 0 .49 43 .20 17 .71 0 .38 0 .47 42 .25 18 .31 0 .30 0 .45

High/high 44 .03 16 .51 0 .44 0 .49 39 .18 14 .61 0 .38 0 .48 40 .13 19 .32 0 .44 0 .46

Table 6

Evaluation of usability and usefulness (5-point Likert scale, strongly disagree (1) to strongly agree (5)); n = 14.

Statement List Matrix Network

μ σ 2 μ σ 2 μ σ 2

Easy to learn 4 .00 1 .13 2 .86 0 .91 3 .79 0 .94

Easy to get to do what I wanted to do 2 .79 1 .01 2 .86 0 .00 3 .08 1 .14

Flexible to interact with 2 .62 1 .08 3 .21 0 .67 3 .79 0 .67

Easy to become skilful at using 3 .57 0 .82 3 .29 0 .96 3 .69 0 .72

Helps to accomplish business ecosystem analysis more quickly 2 .29 0 .96 3 .14 0 .99 4 .08 0 .73

Makes it easier to accomplish business ecosystem analysis 2 .31 0 .61 3 .08 1 .00 4 .14 0 .83

A useful tool for ecosystem analysis 2 .71 1 .10 3 .43 0 .82 4 .14 0 .52

(

t

4

p

e

c

m

c

g

h

v

r

t

i

m

a

s

p

w

l

c

r

4

u

e

c

a

s

H

a

(

I

t

s

u

(

t

list). This improvement may be attributable to experience with

he tool that accrues through the experiment.

.1. Impact of perceived task load and user characteristics

To further understand our results we explored the impact of

erceived task load and user characteristics. Table 5 shows our

xperimental results of decision time and accuracy by user’s per-

eived task load (TLI) and comfort with computational decision

aking tools (CDT), differentiated by the visual representation. To

onduct a reasonable comparison, we split our sample into four

roups: low TLI/low CDT, low TLI/high CDT, high TLI/low CDT, and

igh TLI/high CDT.

Several interesting observations can be made. Across almost all

isual representations and task load index levels, decision accu-

acy goes up by a significant amount with higher levels of decision

ool comfort. This confirms that participants with greater analyt-

cal and visualization decision tool experience are more likely to

ake more accurate decisions when provided with task appropri-

te visual representation. The one exception is for the list repre-

entation where accuracy actually decreases. This in fact confirms

artly that list views are most likely preferred by decision makers

ith lower decision tool comfort, irrespective of the perceived task

oad. A second important observation is that decision time also de-

reases substantially with higher level of decision tool comfort ir-

espective of perceived task load.

.2. Post-hoc usability and usefulness evaluation

Table 6 shows the results of our post-hoc questions on the

sability and usefulness of our three visual representations for

cosystem analysis. Fig. 6 shows these results visually through a

omparative radar chart. Given the general familiarity with tables

nd spreadsheets by business users, it is not surprising that the

ymbolic representation (list) was considered the easiest to learn.

owever, it was considered the least useful view for ecosystem

nalysis. On the other hand the spatial-traversal representation

network) was rated the highest in virtually all perceptual ratings.

n particular, respondents considered the network representation

he view that could help accomplish business ecosystem analy-

is more quickly and easier and overall was considered the most

seful. Interestingly, while the spatial-connectivity representation

matrix) was rated the most difficult to learn, it was also perceived

o be relatively useful for ecosystem analysis.

280 R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282

Fig. 6. Usability and usefulness ratings of visual representations.

a

s

s

a

t

p

c

e

b

v

m

t

d

s

6

s

fl

d

s

i

i

d

p

f

v

s

d

e

c

e

t

m

i

s

p

i

a

a

i

T

i

p

o

v

f

c

A

f

v

p

t

s

p

l

i

t

l

5. Implications

The design of business ecosystem intelligence tools and vi-

sual presentation of business data are emerging and important

research areas. In an experimental context, experienced decision

makers explored three visualization methods (list, matrix and net-

work), and we observed decision making performance in the con-

text of business ecosystem analysis. Drawing on cognitive fit the-

ory, we posited to see performance differentials between visualiza-

tion methods, ecosystem analysis tasks and ecosystem complexity.

The expected impact of visualization method on decision per-

formance was observed with task type differentials. For the list

view, decision performance in the attribute-based tasks was high.

Apart from the browsing task, decision makers’time and accuracy

using the matrix view matched their performance using the net-

work view. The network view allowed the best performance in

the browsing task and also enabled users to perform relatively

well on other task types. The fact that decision makers’ratings of

usability and usefulness for the network view were as high or

higher than their ratings for the list view and the matrix view

gives a strong indication of the cognitive fit between the network

view and decision-making tasks in the context of complex business

ecosystem analysis.

Neither experience with decision complexity, frequency of mak-

ing complex decisions or comfort with decision support systems

mitigated the negative impact of perceived cognitive load on qual-

ity of decision making with complex business ecosystems. Comfort

with decision tools was associated with higher decision accuracy

only for the network view, and then primarily for low complex-

ity business ecosystems. For less complex ecosystems, tool comfort

was positively associated with increased time and accuracy of de-

cisions.

These results suggest that decision quality is likely to suffer

when input information is more complex than decision makerâÇÖs

cognitive orientation, given a particular decision support tool. To

harness the positive value of the wide lens view, the complexity

of business ecosystem data must be calibrated to the mental rep-

resentation and the problem representation of the decision makers

and their visual representation tools. This implication was particu-

larly poignant in the case of the complex ecosystem. We hypoth-

esize that the requirements for such calibration are likely to vary

across complexity contexts and need to be adapted for specific de-

cision scenarios, as well as for the particular cognitive orientation

of decision makers âÇô individually and in groups.

Because many data-driven decisions about business ecosystems

re made by groups of decision makers, it is critical to under-

tand the influence of processes and tools on collaborative deci-

ion making. We recommend further study of business ecosystem

nalysis tasks, views and complexity to explore how processes and

ools can best support decision making groups to select reference

oints, identify common ground, and optimize opportunities for

o-creation.

According to our results, the network view was ranked high-

st for usability and usefulness for browsing and overview tasks in

usiness ecosystem analysis. This was somewhat surprising as the

iew is not yet widely used. It is possible that our effective tutorial

ay have contributed to this. Still, this gives a strong indication of

he cognitive fit between the spatial-traversal (network) view and

ecision-making tasks in the context of complex ecosystem analy-

is.

. Concluding remarks

This study comparatively evaluates the effectiveness of three vi-

ualization methods (list, matrix, network) and the moderating in-

uence of data complexity, task type, and user characteristics on

ecision performance in the context of business ecosystem analy-

is. We pursue this objective using an experiment with prototyp-

cal users (e.g. executives, analysts, investors, and policy makers)

n a unique laboratory setting using synthetic business ecosystem

ata. The results show that while network visualizations do sup-

ort the wide lens perspective, decision making performance suf-

ers if decision support tools are not balanced for mental load and

isual representation. Moreover, the results show that matching vi-

ual representations to task type is more critical at higher levels of

ata complexity.

Results of this experiment also suggest that visual literacy and

xperience with decision making tools are associated with de-

ision quality when using advanced visualization techniques for

cosystem decisions. Moreover, our results highlight that execu-

ives’decision making is compromised when inundated with too

uch information. A potentially trivial but rather interesting find-

ng from our research is that complex issues requiring time-

ensitive decisions require visual representations that are appro-

riate to the task and to the executivesâÇÖ tool comfort. Select-

ng and filtering data to optimally support decision making is

lso important, especially for the wide lens. Further, providing

dditional complementary capabilities, such as search, is equally

mportant.

Our study makes several theoretical and practical contributions.

heoretically, we extend cognitive fit theory and investigate the

mpact of data and task complexity. In particular, we provide sup-

ort that business ecosystem complexity has a differential impact

n how ecosystem visualization representation (the DSS tool – the

iew) has on decision accuracy, as one measure of decision per-

ormance. Matching DSS tool preference to task and cognitive load

ontributes to decision quality, as measured by time and accuracy.

t very high levels of ecosystem complexity, decision making per-

ormance suffers if DSS tools are not balanced for mental load and

isual representation.

Managerially, our study contributes to the relatively underex-

lored, but emerging area of the design of ecosystem intelligence

ools and presentation of ecosystem data for the purpose of deci-

ion making. Our results show that network visualizations do sup-

ort the wide lens perspective. Our study further stresses the chal-

enges data scientists and information visualization designers play

n constructing the DSS tools to illuminate trade off issues that fac-

or into complex ecosystem analysis decisions.

Our study certainly has limitations. The increased control of the

aboratory setting must be traded off against the limitation of the

R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282 281

a

p

r

o

e

p

o

t

a

s

u

t

A

v

B

A

i

s

e

R

A

A

A

A

B

B

B

B

B

B

B

BB

B

B

B

C

C

C

C

C

C

D

D

D

E

F

F

G

H

H

H

H

H

pproach, primarily the generalizability. A study ’in-the-wild’would

rovide substantive value. Moreover, while we ensured to involve

eal, experienced decision makers, a non-trivial effort to achieve,

ur sample size is small and may not allow us to make broad gen-

ralizations. Future studies should include a larger pool of partici-

ants. Similarly, while we strived for gender balance, the majority

f our participants were male. An interesting extension would be

o include more female participants and assess whether there are

ny gender differences. Another limitation is the deliberate use of

ynthetic data. Different insights may have been generated if we

sed an actual business ecosystem context. Each of these limita-

ions presents exciting opportunities for future research.

cknowledgements

The authors acknowledge support from mediaX at Stanford Uni-

ersity, and helpful comments from Neil Jacobstein and Michael

ernstein.

ppendix A. NASA task load index

We would like to know about the workload you experienced

n performing this task. Titles and meaning for each item are pre-

ented below.

• Mental demand: How much mental and perceptual activity

was required (e.g. thinking, deciding, calculating, remembering,

looking, searching, etc.)? Was the task easy or demanding, sim-

ple or complex, exacting or forgiving? • Physical demand: How much physical activity was required

(e.g. pushing, pulling, turning, controlling, activating, etc.)? Was

the task easy or demanding, slow or brisk, slack or strenuous,

restful or laborious? • Time demand: How much time pressure did you feel due to

the rate or pace at which the tasks occurred? Was the pace

slow and leisurely or rapid and frantic? • Performance: How successful do you think you were in accom-

plishing the goals of the task? How satisfied were you with

your performance in accomplishing these goals? • Effort: How hard did you have to work (mentally and physi-

cally) to accomplish your level of performance? • Frustration: How insecure, discouraged, irritated, stressed and

annoyed versus secure, gratified, content, relaxed and compla-

cent did you feel during the task?

Please place an “X” on each scale at the point that matches your

xperience. Consider each scale individually.

eferences

dipat, B. , Zhang, D. , & Zhou, L. (2011). The effects of tree-view based presentation

adaptation on model web browsing. MIS Quarterly, 35 (1), 99–121 .

dner, R. (2012). The Wide Lens: What Successful Innovators See That Others Miss .Portfolio Trade .

garwal, R. , Sinha, A. P. , & Tanniru, M. (1996). Cognitive fit in requirements model-ing: A study of object and process methodologies. Journal of Management Infor-

mation Systems, 13 (2), 137–162 . nderson, E. W. , Potter, K. C. , Matzen, L. E. , Shepherd, J. F. , Preston, G. , &

Silva, C. T. (2011). A user study of visualization effectiveness using EEG and cog-

nitive load. In Computer graphics forum: 30 (pp. 791–800). Wiley Online Library .arabási, A.-L. (2009). Scale-free networks: A decade and beyond. Science,

325 (5939), 412–413 . asole, R. C. (2009). Visualization of interfirm relations in a converging mobile

ecosystem. Journal of Information Technology, 24 (2), 144–159 . asole, R. C. (2014). Visual business ecosystem intelligence: Lessons from the field.

IEEE Computer Graphics and Applications, 34 (5), 26–34 . asole, R. C. , Clear, T. , Hu, M. , Mehrotra, H. , & Stasko, J. T. (2013). Understanding in-

terfirm relationships in business ecosystems with interactive visualization. IEEE

Transactions on Visualization and Computer Graphics, 19 (12), 2526–2535 . asole, R. C. , Hu, M. , Patel, P. , & Stasko, J. T. (2012). Visual analytics for converg-

ing-business-ecosystem intelligence. IEEE Computer Graphics and Applications,32 (1), 92–96 .

asole, R. C. , & Rouse, W. B. (2008). Complexity of service value networks: concep-tualization and empirical investigation. IBM Systems Journal, 47 (1), 53–70 .

asole, R. C. , Russell, M. G. , Huhtamäki, J. , & Rubens, N. (2015). Understanding mo-

bile ecosystem dynamics: A data-driven approach. ACM Transactions on Manage-ment Information Systems, 6 (2), 6 .

ertin, J. (1981). Graphics and Graphic information processing . Walter de Gruyter . lohm, I. , Riedl, C. , Füller, J. , & Leimeister, J. M. (2016). Rate or trade? identifying

winning ideas in open idea sourcing. Information Systems Research, 27 (1), 27–48 .ostock, M. , Ogievetsky, V. , & Heer, J. (2011). D 3 data-driven documents. IEEE Trans-

actions on Visualization and Computer Graphics, 17 (12), 2301–2309 .

runelle, E. (2009). The moderating role of cognitive fit in consumer channel pref-erence. Journal of Electronic Commerce Research, 10 (3), 178–195 .

urton-Jones, A. , & Straub, D. W. (2006). Reconceptualizing system usage: An ap-proach and empirical test. Information Systems Research, 17 (3), 228–246 .

ard, S. K. , & Mackinlay, J. (1997). The structure of the information visualizationdesign space. In Proceedings of the IEEE symposium on information visualization

(pp. 92–99). IEEE .

ardinaels, E. (2008). The interplay between cost accounting knowledge and pre-sentation formats in cost-based decision-making. Accounting, Organizations and

Society, 33 (6), 582–602 . arpendale, S. (2008). Evaluating information visualizations. In Information visual-

ization (pp. 19–45). Springer . han, H. C. , Goswami, S. , & Kim, H.-W. (2012). An alternative fit through problem

representation in cognitive fit theory:. Journal of Database Management, 23 (2),

22–43 . handra, A. , & Krovi, R. (1999). Representational congruence and information re-

trieval: Towards an extended model of cognitive fit. Decision Support Systems,25 (4), 271–288 .

hi, E. H. (20 0 0). A taxonomy of visualization techniques using the data state ref-erence model. In Proceedings of the ieee symposium on information visualization

(pp. 69–75). IEEE .

avis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptanceof information technology. MIS Quarterly, 13 (3), 319–340 .

ennis, A. R. , & Carte, T. A. (1998). Using geographical information systems for deci-sion making: Extending cognitive fit theory to map-Based presentations. Infor-

mation Systems Research, 9 (2), 194–203 . unn, C. , & Grabski, S. (2001). An investigation of localization as an element of cog-

nitive fit in accounting model representations. Decision Sciences, 32 (1), 55–94 . vans, P. C. , & Basole, R. C. (2016). Revealing the api ecosystem and enterprise strat-

egy via visual analytics. Communications of the ACM, 59 (2), 26–28 .

ew, S. (2009). Now you see it: Simple visualization techniques for quantitative analy-sis . Analytics Press .

rownfelter-Lohrke, C. (1998). The effects of differing information presentations ofgeneral purpose financial statements on users’decisions. Journal of Information

Systems, 12 (2), 99–107 . oswami, S. , Chan, H. C. , & Kim, H. W. (2008). The role of visualization tools in

spreadsheet error correction from a cognitive fit perspective. The Role of Visual-

ization Tools in Spreadsheet Error Correction from a Cognitive Fit Perspective, 9 (6),321–343 .

agberg, A. , Swart, P. , & Schult, D. (2008). Exploring network structure, dynamics,and function using NetworkX. Technical Report . Los Alamos National Laboratory

(LANL) . art, S. G. , & Staveland, L. E. (1988). Development of nasa-tlx (task load index):

Results of empirical and theoretical research. Human Mental Workload, 1 (3),

139–183 . eer, J. , Bostock, M. , & Ogievetsky, V. (2010). A tour through the visualization zoo.

Communications of the ACM, 53 (6), 59–67 . eer, J. , & Shneiderman, B. (2012). Interactive dynamics for visual analysis. Queue,

10 (2), 30 . ong, W. , Thong, J. Y. , & Tam, K. Y. (2004). The effects of information format and

shopping task on consumers’online shopping behavior: A cognitive fit perspec-

tive. Journal of Management Information Systems, 21 (3), 149–184 .

282 R.C. Basole et al. / Expert Systems With Applications 65 (2016) 271–282

S

S

S

S

T

T

T

T

T

T

U

V

V

V

W

W

X

Z

Huang, Z. , Chen, H. , Guo, F. , Xu, J. J. , Wu, S. , & Chen, W.-H. (2006). Expertise visu-alization: An implementation and study based on cognitive fit theory. Decision

Support Systems, 42 (3), 1539–1557 . Hubona, G. S. , Everett, S. , Marsh, E. , & Wauchope, K. (1998). Mental representa-

tions of spatial language. International Journal of Human-Computer Studies, 48 (6),705–728 .

Huhtamäki, J. , Russell, M. G. , Rubens, N. , & Still, K. (2015). Ostinato: The explo-ration-automation cycle of user-centric, process-automated data-driven visual

network analytics. In Transparency in social media (pp. 197–222). Springer .

Huhtamäki, J. , Russell, M. G. , Still, K. , & Rubens, N. (2011). A network-centric snap-shot of value co-creation in finnish innovation financing. Online Business Re-

search Review, 5 (March 2011) . Hung, S.-Y. , Ku, Y.-C. , Liang, T.-P. , & Lee, C.-J. (2007). Regret avoidance as a mea-

sure of dss success: An exploratory study. Decision Support Systems, 42 (4),2093–2106 .

Iansiti, M. , & Levien, R. (2004). The keystone advantage: What the new dynamics of

business ecosystems mean for strategy, innovation and sustainability . Boston, MA:Harvard Business School Press .

Kamis, A. , Koufaris, M. , & Stern, T. (2008). Using an attribute-based decision supportsystem for user-customized products online: An experimental investigation. MIS

Quarterly, 32 (1), 159–177 . Keim, D. A. (2002). Information visualization and visual data mining. IEEE Transac-

tions on Visualization and Computer Graphics, 8 (1), 1–8 .

Kelly, E. (2015). Introduction: Business ecosystems come of age. Khatri, V. , Vessey, I. , Ram, S. , & Ramesh, V. (2006). Cognitive fit between conceptual

schemas and internal problem representations: the case of geospatio-temporalconceptual schema comprehension. IEEE Transactions on Professional Communi-

cation, 49 (2), 109–127 . van der Land, S. , Schouten, A. P. , Feldberg, F. , van den Hooff, B. , & Huys-

man, M. (2013). Lost in space? Cognitive fit and cognitive load in 3D virtual

environments. Computers in Human Behavior, 29 (3), 1054–1064 . Lee, B. , Plaisant, C. , Parr, C. S. , Fekete, J.-D. , & Henry, N. (2006). Task taxonomy for

graph visualization. In Proceedings of the 2006 avi workshop on beyond time anderrors: Novel evaluation methods for information visualization (pp. 1–5). ACM .

Li, M. , Wei, K.-K. , Tayi, G. K. , & Tan, C.-H. (2015). The moderating role of infor-mation load on online product presentation. Information & Management, 53 (4),

467âÇô480 .

Mahoney, L. S. , Roush, P. B. , & Bandy, D. (2003). An investigation of the effects ofdecisional guidance and cognitive ability on decision-making involving uncer-

tainty data. Information and Organization, 13 (2), 85–110 . Mennecke, B. E. , Crossland, M. D. , & Killingsworth, B. L. (20 0 0). Is a map more

than a picture? the role of SDSS technology, subject characteristics, and Problemcomplexity on map reading and Problem solving. MIS Quarterly, 24 (4), 601–629 .

Moore, J. F. (1996). The death of competition: Leadership and strategy in the age of

business ecosystems . HarperBusiness New York . Morse, E. , Lewis, M. , & Olsen, K. A. (20 0 0). Evaluating visualizations: Using a taxo-

nomic guide. International Journal of Human-Computer Studies, 53 (5), 637–662 . North, C. (2006). Toward measuring visualization insight. IEEE Computer Graphics

and Applications, 26 (3), 6–9 . Novick, L. R. , & Hurley, S. M. (2001). To matrix, network, or hierarchy: That is the

question. Cognitive Psychology, 42 (2), 158–216 . Russell, M. G. , Huhtamäki, J. , Still, K. , Rubens, N. , & Basole, R. C. (2015). Relational

capital for shared vision in innovation ecosystems. Triple Helix Journal, 2 (1),

1–36 . Russell, M. G. , Still, K. , Huhtamäki, J. , Yu, C. , & Rubens, N. (2011). Transforming inno-

vation ecosystems through shared vision and network orchestration. In Proceed-ings of the Ninth International Conference of the Triple Helix Association . California,

USA: Stanford . Shaft, T. M. , & Vessey, I. (2006). The role of cognitive fit in the relationship between

software comprehension and modification. MIS Quarterly, 30 (1), 29–55 .

Shannon, C. E. (1948). A mathematical theory of communication. Bell System Tech-nical Journal, 27 (4), 623–656 .

Sharda, R. , Barr, S. H. , & MCDonnell, J. C. (1988). Decision support system effective-ness: A review and an empirical test. Management Science, 34 (2), 139–159 .

Shen, M. , Carswell, M. , Santhanam, R. , & Bailey, K. (2012). Emergency managementinformation systems: Could decision makers be supported in choosing display

formats? Decision Support Systems, 52 (2), 318–330 .

hneiderman, B. (1996). The eyes have it: A task by data type taxonomy for infor-mation visualizations. In Visual languages, 1996. proceedings., ieee symposium on

(pp. 336–343). IEEE . Shneiderman, B. , & Plaisant, C. (2006). Strategies for evaluating information visual-

ization tools: multi-dimensional in-depth long-term case studies. In Proceedingsof the 2006 avi workshop on beyond time and errors: novel evaluation methods for

information visualization (pp. 1–7). ACM . inha, A. , & Vessey, I. (1992). Cognitive fit: An empirical study of recursion and it-

eration. IEEE Transactions on Software Engineering, 18 (5), 368–379 .

Smelcer, J. B. , & Carmel, E. (1997). The effectiveness of different representationsfor managerial problem solving: Comparing tables and Maps. Decision Sciences,

28 (2), 391–420 . oukup, T. , & Davidson, I. (2002). Visual data mining: Techniques and tools for data

visualization and mining . Wiley. com . Speier, C. (2006). The influence of information presentation formats on complex

task decision-making performance. International Journal of Human-Computer

Studies, 64 (11), 1115–1131 . peier, C. , & Morris, M. G. (2003). The influence of query interface design on deci-

sion-making performance. MIS Quarterly, 27 (3), 397–423 . Still, K. , Huhtamäki, J. , Russell, M. G. , & Rubens, N. (2014). Insights for orchestrat-

ing innovation ecosystems: The case of EIT ICT labs and data–driven networkvisualisations. International Journal of Technology Management, 66 (2), 243–265 .

Teets, J. , Tegarden, D. , & Russell, R. (2010). Using cognitive fit theory to evaluate the

effectiveness of information visualizations: An example using quality assurancedata. IEEE Transactions on Visualization and Computer Graphics, 16 (5), 841–853 .

egarden, D. P. (1999). Business information visualization. Communications of the AIS,1 (1es), 4 .

homas, J. J. , & Cook, K. A. (2005). Illuminating the path: The research and develop-ment agenda for visual analytics . IEEE Computer Society Press .

odd, P. , & Benbasat, I. (1992). The use of information in decision making: An exper-

imental investigation of the impact of computer-based decision aids. MIS Quar-terly , 373–393 .

Tory, M. , & Moller, T. (2004). Rethinking visualization: A high-level taxonomy. InInformation visualization, 2004. infovis 2004. ieee symposium on (pp. 151–158).

IEEE . ory, M. , & Moller, T. (2005). Evaluating visualizations: Do expert reviews work?

IEEE Computer Graphics and Applications, 25 (5), 8–11 .

ufte, E. R. , & Graves-Morris, P. (1983). The visual display of quantitative information :2. Graphics Press .

uttle, B. M. , & Kershaw, R. (1998). Information presentation and Judgment strategyfrom a cognitive fit perspective. Journal of Information Systems, 12 (1), 1 .

Umanath, N. S. , & Vessey, I. (1994). Multiattribute data presentation and Humanjudgment: A Cognitive fit perspective. Decision Sciences, 25 (5–6), 795–824 .

rbaczewski, A. , & Koivisto, M. (2008). The importance of cognitive F it in mobile

information systems. Communications of the Association for Information Systems,22 (10) .

an Alstyne, M. W. , Parker, G. G. , & Choudary, S. P. (2016). Pipelines, platforms, andthe new rules of strategy. Harvard Business Review, 94 (4), 54–+ .

essey, I. , & Galletta, D. (1991a). Cognitive fit: An empirical study of informationacquisition. Information Systems Research, 2 (1), 63–84 .

Vessey, I. , & Galletta, D. (1991b). Cognitive fit: An empirical study of informationacquisition. Information Systems Research, 2 (1), 63–84 .

irzi, R. A. (1992). Refining the test phase of usability evaluation: How many

subjects is enough? Human Factors: The Journal of the Human Factors and Er-gonomics Society, 34 (4), 457–468 .

alsham, G. (2006). Doing interpretive research. European journal of informationsystems, 15 (3), 320–330 .

right, W. (1997). Business visualization applications. IEEE Computer Graphics andApplications, 17 (4), 66–70 .

u, P. , Chen, L. , & Santhanam, R. (2015). Will video be the next generation of e–

commerce product reviews? presentation format and the role of product type.Decision Support Systems, 73 , 85–96 .

hu, Y. (2007). Measuring effective data visualization (pp. 652–661)). Springer .