American Institute of Aeronautics and Astronautics

1

Evaluating an Immersive Virtual Environment for Organizationally Distributed Software Development

Redge Bartholomew1 Rockwell Collins, Cedar Rapids, Iowa 52498

As systems become large, they are developed by complex organizations, involving dozens of companies spread over several contractual tiers and several geographic locations. As component scales shrink, systems integrate increasing numbers of diverse applications onto common computing platforms, requiring diverse expertise from diverse locations. In this environment, common media can be inadequate for coordinating and synchronizing developer efforts. An Immersive Networked Virtual Environment emulates colocation. From their PC keyboard and monitor, participants steer representations of themselves through a 3-dimensional virtual world. In some cases they communicate with each other audibly using a headset and microphone; in others communication is via text messaging. They interact to socialize, play games, develop models, assemble and animate 3-dimensional objects, and other similar activities. In some cases they can launch standard desktop applications within embedded windows to create documents, designs, and other artifacts. This paper describes one example of such an environment and how it might improve geographically distributed software development. It also describes results-to-date of an ongoing evaluation.

I. Problems of Scale and Scope Colocation of developers optimizes development. It promotes communication, familiarity, trust, information

sharing, resource location, and problem resolution, and can mitigate culture & language differences 1-3, 32-35. Increasingly, however, systems are large and complex, involving millions of lines of source code. In this case, they are developed by dozens of organizations spread over several tiers of contracts and subcontracts, scattered across many geographic regions 4-10. Results of academic studies and industry pilot projects indicate a need for significant improvement 1-3, 10-13, 15, 36. Problems include comprehending and deploying requirements, designs, and interfaces; synchronizing actions among contractors; confirming interoperability among component systems; and providing accurate status (including risks and opportunities) and plans. Development at this scale requires remediation: collaboration is problematic.

The complement to this problem of scale is the problem of development scope. As hardware component scale shrinks, increasingly diverse functions are integrated onto common platforms 10, 14, 15. Displays, navigation, flight control, landing, data links, and so on are becoming software components sharing a common computing platform. Integration of diverse system functions forces integration of diverse expertise. Simultaneous with this increasing integration, however, cycle time is shrinking, leaving little time to develop local experts. Even for smaller-scale developments, teams must exploit available knowledge and skill regardless of location and without constant migration. This also requires remediation.

Commonly used collaboration technologies provide an inadequate level of shared understanding among developers at different regional facilities. This results in reduced productivity and greater rework compared to collocated teams. This paper describes an ongoing corporate evaluation of an immersive networked virtual environment (INVE) to determine its potential for improving organizationally and geographically distributed software development.

II. Background Engineers must be able to collaborate on system, subsystem, and component developments while maintaining a

common and accurate understanding of requirements and designs. Misunderstandings, especially at the handoff points, must be minimal. Also, developers need to calibrate each other – i.e., determine who has what knowledge and what depth of knowledge. They need to know who they can go to for what kind of help, and what weight to

1 Engineering & Technology, 400 Collins Road, M.S. 125-100, Cedar Rapids, Iowa 52498

AIAA Infotech@Aerospace Conference <br>and <br>AIAA Unmanned...Unlimited Conference 6 - 9 April 2009, Seattle, Washington

AIAA 2009-2031

Copyright © 2009 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved.

assign to contradictory opinions. In addition, where consensus is part of the dynamic of decision-making, developers need to know and trust each other enough to feel that compromise will not lead to exploitation.

Familiarity and trust have typically required working in close quarters for a prolonged period. To the extent possible, software engineers have been physically collocated and have been part of the same organization. Systems are sometimes architected so that work assignments minimize development coupling among regional sites. Sometime this is done even at the risk of sub-optimizing system performance. Over time, this produces an environment in which local communication is open, unfiltered, and unreserved. Ambiguity among collocated, collaborating engineers tends to be lower than geographically distributed teams, even where the separation is only among different hallways of the same building 36.

Colocation rarely scales well. There are physical limits – e.g., the number of people that can comfortably and productively be located within 30 meters of each other. For the prime contractor, developers working on one segment of an aircraft may never meet those working on other segments. The customer or the program office may become familiar with the prime contractor, but may be unfamiliar with subcontractors, especially those at the second, third, and subsequent tiers. Even for subcontractors developing a single system or subsystem, given an increasingly short development cycle, there may no longer be adequate time to relocate staff: beyond an initial kickoff meeting, some may never again see others.

Coordination and synchronization problems increase with the number of people, organizations, and time zones involved. System or aircraft integration can become a process of iterative discovery. So also can development and operational test. All can require significant degrees of rework as developers discover differences in interpretation of requirements, designs, architectures, and interfaces. Problems can result from misinterpretations of schedule milestones and increment content.

Experiments and observations from academia and industry indicate that telephones, video, instant messaging, email, temporary colocation, and other available methods and technologies are inadequate, that they are unable to overcome the problems that appear to be inherent in multi-site, multi-organization collaboration 27-28, 33-35. Voice or text-only media do not enable a necessary degree of familiarity and trust; scheduling and synchronizing visual media inhibits spontaneity. Questions are deferred, assumptions remain without validation.

III. A Possible Solution Literature and casual observation suggest an immersive virtual world might adequately resolve these problems 17-

32. The immersive world is displayed on the computer monitor. Participants are represented by an animated figure called an avatar. The avatar is moved through offices, labs, meeting rooms, libraries, and other real-world representations using the arrow keys of a standard desktop keyboard, or in some cases a game control pad. Participants log onto a common server (or set of linked servers) that synchronizes avatar movements across platforms, providing a common view updated in real-time. In some, communication is text only, in others it can be voice and text – e.g., Voice-Over-Internet-Protocol audio (VOIP), text via an instant messaging window. Some can provide body language, facial expression, sound and animation synchronization, data persistence, and other elements of face-to-face collaboration. Participants collaboratively build models, create designs, code, hold artifact reviews and inspections, and run tests using standard desk-top applications installed on clients or server.

Figure 1. MPK20

American Institute of Aeronautics and Astronautics

2

American Institute of Aeronautics and Astronautics

3

bjects

Within massively multi-player online games like World of Warcraft and immersive virtual social worlds like Second Life, strangers meet, form teams, strategize, plan, coordinate actions, and adjust plans as needed. Potentially, this could provide enough fidelity to real colocation to significantly improve distributed collaboration.

For a proof-of-concept evaluation, we used Sun’s MPK20 (also known as Project Wonderland) 16. It is an open source, immersive virtual world that can be installed to operate inside an enterprise firewall. The download consists of 3 primary pieces: a client, a server, and a voice bridge. Assuming the server is already running, the user enters the environment by starting the client and entering a user name and a server name. This establishes communication between the participants PC and the server. Users logged into the same server can see and hear each other (their avatars), interact, and share data and applications. Access to the environment is controlled by limiting access to the server. Each client determines and generates the user’s view (what is in view, relative location of objects); the server synchronizes the client views, sending vectors to indicate updates (e.g., avatar or object movement).

It includes VOIP audio, instant messaging, and configurable persistent avatars (name and physical characteristics are retained between sessions). A standard keyboard steers each participant’s avatar through a prefabricated virtual office building that includes a main concourse, a conference room, offices, and a lab. (Figs. 1 & 2.) Users can modify the basic geography or they can create their own.

Three dimensional ocan be assembled and animated (e.g., container exploded into circuit boards (Fig. 3.), and avatars can penetrate objects to view internal components. When a user logs-in, his or her avatar becomes visible to the other online participants. When the user exits the program, the avatar disappears. A separate navigation window identifies the names and relative locations of the other avatars currently online.

Audio communication is full duplex, so all participants can speak at the same time and interrupt each other. The audio is also stereo and tied to the avatar’s placement with respect to the others – e.g., those on the right are heard in the right ear. Audio volume is dependent on distance: as the distance between avatars decreases, voice volume increases).

Tool-based collaboration is initiated via a drop-down menu identifying available applications. Launched applications appear as 2-dimensional windows running within the virtual world – flattened against the wall of a conference room (like an overhead projector screen), or suspended in mid air. (Figs. 2, 4, 5.) All participants can manipulate the application via explicit control sharing. Users’ avatars are shown in the same window as the application window, similar to collocated engineers standing on either side of the projected image from an overhead projector.

Figure 2. The Immersive Virtual Office

IV. Evaluation The initial evaluations were all performed outside the corporate firewall, using off-the-shelf laptop computers and

a public-access wireless network. Once we were able to establish that communication could be contained within the

client and server machines and that access could be controlled via server access, the evaluation transitioned to the corporate network using enterprise-configured desktop machines.

The first four users were experienced engineers, each employed with the company for between 15 and 30 years. Two had prior experience with online games or immersive social worlds. All four had considerable experience collaborating with regional facilities via conventional media. Initially, only one of the participants knew the other three, and only two were located in the same building. One was separated from the other two by approximately 2 Km. The fourth was separated from the others by approximately 3000 Km. In some cases, all were logged directly into the corporate network; in others, access was via a corporate VPN.

In a separate but related activity, we funded a research project at the Software Engineering Research Center (SERC), a National Science Foundation Industry/University Cooperative Research Center. The goal of the project was two-fold: first, to help determine if an immersive virtual environment would satisfy our need for training and development; second, to determine approximately what it would take to acquire it, adapt it, and deploy it across a multi-site network. We experimented with inter-organizational collaboration, using the Virtual Private Network at Ball State University to communicate with the SERC team. Capability demonstrations and informal status reviews were held within the immersive virtual environment. This allowed a cursory evaluation of band-width and other network impacts.

V. Results To date, collaboration

has worked well. The greatest early-on advantages over conventional media were informal communication and integrated application sharing. The integrated audio and visual dimensions added a significant element missing from aggregations of other media, providing greater fidelity to a real, face-to-face environment. Within a small scale context, the immersive virtual environment promotes familiarity and trust to a greater extent than other media. Impromptu and immediate access to development tools and artifacts to make a point or to aid an explanation was more effective in establishing common understanding than scheduling and coordinating a video link or a webinar, or sending an email attachment. Direct access to milestone schedules appeared to improve task coordination over other media.

The audio-visual fidelity to a conference room or auditorium with broadcast sound and projected image is very high. Pair-wise programming could be implemented this way, as well as artifact reviews, project and program reviews, technical interchange meetings, status reviews, and so on. The appearance is that in-world artifact libraries could be easier to find via visual cues (e.g., a “requirements this way” sign) than server paths, and could be easier to navigate than directory trees.

Currently, however, only limited visual cuing is available. The avatar has minimal face, hand, and body gestures, making it difficult to communicate non-verbal responses. Keyboard-driven gestures are as effective as emoticons. The extent to which this can be improved and the difficulty in doing so has not yet been determined.

Also, in-world application sharing is implemented via peer-to-peer connections, which are currently only available on UNIX and Linux platforms, not Wintel. In-world application sharing among Wintel clients can be implemented in less direct ways (e.g., using rdesktop running on a UNIX machine to access applications running on a Windows server). In this case, the application window can be visible, but the window content is not – the application runs in its own, non-embedded window.

Figure 3. Three-Dimensional Animated Object.

American Institute of Aeronautics and Astronautics

4

For an INVE to be optimally effective, each team member would likely have to maintain a continuous presence in it, and use it for all aspects of collaborative development. Modeling, coding, editing, linking, debugging, and so on would have to be done in in-world application windows. In terms of incremental effort, this may be no more demanding than the current practice of logging in to the corporate network first thing in the morning. Providing degrees of privacy, however, may require accommodation in INVE building floor plans. If developers are to maintain a continuous in-world presence, in-world space may need to be available for the kinds of sandbox and prototype activities that many prefer to do privately. It could amount to retreating to a virtual office.

In-world audio/visual discussions can be recorded and played back later. This recovery of information could potentially mitigate some of the effects of time-zone separation – e.g., missed meetings, reviews, impromptu discussions, decision points, and so on. Access over a VPN was adequate (setting aside initial bandwidth and client/server logistics issues) making collaboration from home at off-hours easier.

It is unclear what impact the INVE might have on existing infrastructure with increased use. Initially, all machines that were used for the evaluation required graphics card upgrades. Attempts to use a lower-capacity card produced unacceptably long lag in command response, or an application crash. Server capacity would presumably become an issue. The MPK20 server is responsible for synchronizing the views of all logged-in users (movement, placement of objects). The practical limit on the number that can simultaneously use the same server (i.e., occupy the same environment) is unclear and will require measurement. Similarly, the impact of connecting regional facilities to common servers to synchronize client views is unclear. The impact on network capacity will require measurement and possible accommodation or remediation. Other issues must be identified and accommodated – effectiveness of access control, application licensing, export control, and so on. Determining practical, implementation-specific limits will be part of a pilot evaluation.

Inter-organizational collaboration may also require remediation. Our experience in communicating with the SERC team was that over time as general network access policies changed, explicit accommodation was required within the application or within the desktop or server support infrastructure. Occasionally there were unexpected access denials; sometimes long response times, which were especially problematic in trying to use a mouse to control avatars or manipulate objects.

More also needs to be learned about the effectiveness of the INVE as a communication medium. Among developers who will never meet, issues that could degrade its effectiveness need to be identified so that accommodation and remediation can be explored. Finding ways of improving non-verbal communication will be explored further at some point – e.g., driving avatar gestures from a digital camera focused on the user 29.

Figure 4. Application Window.

VI. Significance Geographically distributed development of large scale systems and broadly scoped systems is problematic. Use

of a networked virtual environment may mitigate some of the problems. Up to some yet-to-be-determined degree of time zone separation, development teams could be formed regardless of the geographic or organizational distribution of the available expertise.

American Institute of Aeronautics and Astronautics

5

An emerging operational concept includes a baseline world that would have a project building with a library and a system assembly area at the center, with hallways radiating outward. Hallways would contain virtual offices and labs for analysts, developers, end-users, subcontractors, the acquisition team, and so on. Virtually collocated developers would analyze, design, and code in windows visible to all authorized participants. Collaborating developers could view requirements, architectures, designs, and code, and witness testing by visiting labs and offices. Released artifacts would be kept in a virtual library – e.g., books organized on stacks in alcoves based on domain and function. Double clicking on the virtual book would launch the application that would open the file. Projects would hold artifact reviews in the library near project-specific stacks. They would hold program reviews in a central auditorium.

A prime contractor would control the assembly area, while subcontractors each control to their own hallway – including access control. Subsystem interfaces would be defined, modeled, and developed in in-world windows opened in shared-access development areas. Systems engineers could define, model, and validate requirements, architectures, and interfaces in application windows.

Software engineers would develop detailed requirements, designs, and code, create and validate models, perform static analysis and unit testing. As software components and subsystems emerge from unit testing, they would be integrated into the system build. Developers would meet daily to review the results of the latest build and system-level static analysis, and determine responsibility for error-correction.

Desktop simulation windows would show the results of ongoing dynamic testing at the interface, subsystem, and system levels. Data analysis application windows would show efforts to evaluate test results and determine error sources.

Developers could pursue any form of development and test susceptible to desktop applications. As-needed training would be provided in centrally located training rooms using view cell and other application windows as appropriate.

Figure 5. Application Sharing.

VII. Conclusion The immersive networked virtual environment appears promising enough to justify a continuing evaluation. We

are planning a pilot project to determine its effectiveness for engineering training across multiple regional facilities and multiple time zones.

It appears to accelerate the establishment of working relationships among a distributed team. It appears to be easier to achieve candor of communication using an immersive virtual medium than, say, a video link. It appears to make knowledge sharing and resource locating easier and faster, and to reduce the level of ambiguity and confusion that typically exists among distributed development teams. Still to be considered are scalability (e.g., of resources, across time zones), and extendibility (application to projects covering a broad spectrum of scope).

DoD policy revisions 37 and emerging legislation 38 may increase the focus on developmental testing in an effort to accelerate delivery of capability while also reducing the number of operational failures. In this and other similar developmental contexts, by improving communication and coordination the immersive virtual environment may enable a significant reduction in the number of errors that escape from their phase of insertion, that escape developmental testing, and may accommodate requirements volatility by reducing confusion to a greater degree than is currently available.

American Institute of Aeronautics and Astronautics

6

American Institute of Aeronautics and Astronautics

7

References 1 Fernandes, et al, Proceedings 2006 IEEE International Conference on Global Software Engineering, IEEE Computer

Society Press, October 2006 2 Ebert, Global Software Engineering: A Ready Note for Engineers, Project Managers, consultants and Suppliers Engaged in

Global Development Projects, IEEE, 2006 3 Sangwan, et al, Global Software Development Handbook, Auerbach Publications, 2007 4 Government Accountability Office, Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major

Weapon System Program Outcomes, GAO-08-619, Report to the Committee on Armed Services, U.S. Senate, July 2008 5 Government Accountability Office, Defense Acquisitions: Better Weapon Program Outcomes Require Discipline,

Accountability, and Fundamental Changes in the Acquisition Environment, Testimony Before the Committee on Armed Services, U.S. Senate, 3 June 2008

6 Francis (GAO), Defense Acquisitions: 2009 Is a Critical Juncture for the Army’s Future Combat System, Government Accountability Office, March 2008

7 Government Accountability Office, Defense Acquisitions: Cost to Deliver Zumwalt-Class Destroyers Likely to Exceed Budget, GAO-08-804, July 2008

8 Government Accountability Office, Defense Acquisitions: Significant Challenges Ahead in Developing and Demonstrating Future Combat System’s Network and Software, GAO-08-409, Government Accountability Office, March 2008

9 National Research Council, Preliminary Observations on DoD Software Research Needs and Priorities, National Academies Press, 2008

10 National Academy of Sciences, Human-System Integration in the System Development Process: A New Look, The National Academies Press, 15 June 2007

11 Government Accountability Office, Joint Strike Fighter: Recent Decisions by DoD Add to Program Risks, GAO-08-388, March 2008

12 Government Accountability Office, Defense Acquisitions: Assessments of Selected Weapon Programs, GAO-08-467SP, March 2008

13 Government Accountability Office, Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DoD’s Acquisition Environment and Weapon System Quality, GAO-08-294, February 2008

14 Srinivasan, Lundqvist, “Why is Aerospace Software Development and Sustainment Hard?”, Lean Aerospace Initiative Working Paper Series, January 2008, http://lean.mit.edu/index.php?option=com_docman&task=doc_details&gid=1745&Itemid=332

15 Boehm (U of Southern California), Turner (George Washington University), Balancing Agility and Discipline: A Guide for the Perplexed, Addison Wesley Professional, 2003

16 Project Wonderland, https://lg3d-wonderland.dev.java.net/17 Stach, et al, “Improving Recognition and Characterization in Groupware with Rich Embodiments”, Proceedings CHI 2007,

ACM, 2007, pp. 11-20 18 Hinds, McGrath, “Structures that Work: Social Structure, Work Structure and Coordination Ease in Geographically

Distributed Teams”, Proceedings Conference on Computer Supported Cooperative Work 2006, ACM, 2006, pp. 343-352 19 Moore, et al “Coordinating Joint Activity in Avatar-mediated Interaction”, Proceedings CHI 2007, ACM, 2007, pp. 21-30 20 Nardi, Harris, “Strangers and Friends: Collaborative Play in World of Warcraft”, Proceedings Conference on Computer

Supported Cooperative Work 2006, ACM, 2006, pp. 149-158 21 Bainbridge, “The Scientific Research Potential of Virtual Worlds”, AAAA Science, 27 July 2007, pp. 472-476 22 Chatham, “Games for Training”, Communications of the ACM, July 2007, pp. 37-43 23 Bowman, McMahan, “Virtual Reality: How much Immersion Is Enough?”, IEEE Computer, July 2007, pp. 36-43 24 Donath, “Virtually Trustworthy”, AAAS Science, 6 July 2007, pp. 53-54 25 Whitehead, “Collaboration in Software Engineering: A Roadmap, Proceedings International Conference on Software

Engineering, Future of Software Engineering 2007, IEEE, 2007 26 Wang, et al, “Social Computing: From Social Informatics to Social Intelligence”, IEEE Intelligent Systems, March/April

2007, pp. 79-83 27 Herbsleb, et al, “An Empirical Study of Global Software Development: Distance and Speed”, International Conference on

Software Engineering 2001, IEEE, 2001, pp. 81-90 28 Avritzer, Ostrand, Weyuker, “Experience Developing Software Using a Globally Distributed Workforce”, International

Conference on Global Software Engineering 2006, IEEE, October 2006 29 Schreer, et al, “Real-Time Vision and Speech Driven Avatars for Multimedia Applications”, IEEE Transactions on

Multimedia, Vol. 10, No. 3, April 2008, pp. 352-360 31 Kujanpaa, Manninen, “Supporting Visual Elements of Non-Verbal Communication in Computer Game Avatars”,

Proceedings Level Up – Digital Games Research Conference, 2003 32 Kring, Communication Modality And After Action Review Performance In A Distributed Immersive Virtual Environment,

PhD. Dissertation, Department of Psychology, College of Arts & Sciences, University of Central Florida, Spring Term, 2004 33 Oshri, Kotlarsky, Willcocks, “Missing Links: Building Critical Social Ties for Global Collaborative Teamwork”,

Communications of the ACM, April 2008

American Institute of Aeronautics and Astronautics

8

34 Teasley, et al, “Rapid Software Development through Team Collocation”, IEEE Transactions on Software Engineering, July 2002, pp. 671-683

35 Olson, Olson, “Distance Matters”, Human Computer Interaction, September 2000, pp. 139-178 36 Herbsleb, “Global Software Engineering: The Future of Socio-Technical Coordination”, Proceedings 2007 IEEE Future of

Software Engineering, May 2007 37 Undersecretary of Defense for Acquisition, Technology, and Logistics, Department of Defense Instruction Number

5000.02 “Operation of the Defense Acquisition System”, December 8, 2008 38 U.S. Senate, S.454: Weapon Systems Acquisition Reform Act of 2009,

http://www.govtrack.us/congress/billtext.xpd?bill=s111-454