Establishing a User-Driven, World-Establishing a User-Driven, World-Class Oceanographic Data Center by Class Oceanographic Data Center by the Right People, in the Right Place , the Right People, in the Right Place , and at the Right Timeand at the Right Time

L. Charles Sun

National Center for Ocean Research

20-24 June, 2005, Taipei, Taiwan

2

OutlineOutline

1. Time, Place, and People2. Steps in Establishing an NODC3. Mission and Role of an NODC4. QC and QA5. Products and Services6. Information Technology7. Organizational Considerations and Chart8. “Collaboratory” 9. IDARS, Argo & GTSPP: Three examples of “Collaboratories”10. Data Portal: “Gateway” to Ocean Data11. Climate Data Portal: The Proven Prototype12. Other Technologies for the Collaboratory13. The Future

3

Time, Place, and PeopleTime, Place, and People

Time: Since 1975 ~Place: The Center of the worldPeople: We are the right people

4

Steps in Establishing an NODC - ISteps in Establishing an NODC - I

1 Recruit a team of interested parties to propose a mission and organizational model for the center.

2 Construct a draft mission.3 Conduct negotiations with the potential

partners.

5

Steps in Establishing an NODC - IISteps in Establishing an NODC - II

4 Prepare a draft administrative organization.

5 Prepare a final version of the mission and information on partnerships for final approval.

6

Organization ChartOrganization Chart

Office of the DirectorDirector

Deputy DirectorAssociate Director

Ocean DynamicsChief

Data Base ManagementChief

Information Technology Chief

Staff

Data ProcessingResearch Data and

Product Development

Data ArchivalDatabase Development and

Maintenance

NetworkingOperating System Maintenance

Hardware/software purchase and Maintenance

LibraryChief

Service

7

Mission of an NODCMission of an NODC

To safeguard versions of oceanographic data and information.

To provide high quality data to a wide variety of users in a timely and useful manner.

8

Roles of an NODCRoles of an NODC

Conventional role – as a minimumContemporary role – in response to

advances in data collection and information technology

9

Conventional Role - IConventional Role - I

Receive data, perform quality control, archive and disseminate it on request.

Keep copies of all or part of its data holdings in the format in which the data were received.

Developing and protecting national archives of oceanographic data

10

Conventional Role - IIConventional Role - II

Produce and provide inventories of its holdings on request.

Referral of the users to sources of additional data and information not stored in the NODC.

Participate in international oceanographic data and information exchange.

11

Contemporary Role - IContemporary Role - I

Receive data via electronic networks on a daily basis, process the data immediately, and provide outputs to the user or to the data collectors for data in question.

Report the results of quality control directly to data collectors as part of the quality assurance module for the system.

12

Contemporary Role - IIContemporary Role - II

Process and publish data on the Internet and on CD/DVD-ROMs.

Publish statistical studies and atlases of oceanographic variables.

Performing a level of quality control on its data holdings

13

Quality Control and AssuranceQuality Control and Assurance

Data can be detected easily by a data center Obvious errors such as an impossible date and

time and location

Data cannot usually be detected by a data center Subtle errors such as an instrument may be off

calibration

14

Information Technologies - IInformation Technologies - I

Data Storage/ArchiveData ProcessingLocal Area NetworkingWide Area Networking – the Internet

(and the GTS)

15

Information Technologies - IIInformation Technologies - II

Publishing DVD/CD–ROMsGraphics Capability (Graphical

Information System)Software Development &

ImplementationHardware procurement &

Maintenance

16

Products Development - IProducts Development - I

Work with the client to determine what the real need. Examples of data products include atlases, datasets of ocean observations filtered by area, time and variables observed

17

Products Development - IIProducts Development - II

Review the world wide web sites of existing NODCs for ideas and examples of data and Information products.

18

ServicesServices

Providing directory and inventory information

Acting as a referral centerReceiving data for specific

processing followed by delivery of the processed data

19

Organizational ConsiderationsOrganizational Considerations

A centralized data center A distributed data center

Centers of Data : “Data Portals” or “Virtual Collaboratories”

Data CenterData Center

Center of Data ACenter of Data A Center of Data BCenter of Data B Center of Data CCenter of Data C

20

What is a Collaboratory?What is a Collaboratory?

The fusion of computers and electronic communications has the potential to dramatically enhance the output and productivity of researchers. A major step toward realizing that potential can come from combining the interests of the scientific community at large with those of the computer science and engineering community to create integrated, tool-oriented computing and communication systems to support scientific collaboration. Such systems can be called "collaboratories."

From "National Collaboratories - Applying Information Technology for Scientific Research," Committee on a National Collaboratory, National Research Council. National Academy Press, Washington, D. C., 1993.

21

AcknowledgementAcknowledgement

Soreide, N. N. and L. C. Sun, 1999:

Virtual Collaboratory: How Climate Research can be done Collaboratively using the Internet. U.S. – China Symposium and Workshop on Climate variability, September 21-24, 1999, Beijing, China

Presented by Len Pietrafesa, North Carolina State University.

22

Collaboratory Infrastructure Collaboratory Infrastructure Data Portal

– Computer and networking hardware and software – Increased network bandwidth/speed– Next Generation Internet (NGI) connection

Visualization– Interactive Java graphics– 3D, Virtual Reality, collaborative virtual environments– immersion technology CAVE, ImmersaDesk...

Relationships:– Observing System Project Offices– Research community, Academia...– Other Collaboratory nodes– Steering Committee

23

International Steering Committee

CollaboratoryPartner

CollaboratoryPartner

Collaboratory

Partner

Collaboratory Partners & CustomersProviders of Data & Information

Users of Data & Information

Observations&

Satellite Groups

Modeling&

ForecastingGroups

ResearchGroups

New Users Educational Administrators General Public

Structure of the Collaboratory for Ocean Research

24

IDARSIDARS** as an example... as an example...

• Real-Time Coastal Water Temperature Data• Real-Time Argo Profile Data• Real-Time Global Temperature and Salinity

Profile Data• Time Series Data• NOAA CoastWatch AVHRR SST Images

http://www.nodc.noaa.gov/idars/

*Interactive Data Access and Retrieval System

25

Argo as an example...Argo as an example...

26

GTSPPGTSPP** as an example... as an example...

** Global Temperature-Salinity Profile Program

27

Argo and GTSPP Argo and GTSPP

Argo and GTSPP set a standard in the international ocean data management community

Data dissemination in near-real time– Researcher involvement has assured data quality

Benefits of data dissemination– Wide use of Argo and GTSPP data – Traditional research, modeling, forecasting groups– Related disciplines, educational, administrative, public

With recent advances in technology, we can do much more...

28

Distributed Object TechnologyDistributed Object Technology

Data servers and datasets are objects – software packages of procedures and data that contain their own context

Solid, commercial underpinning for distributed object technology in the ocean sciences

30

The Data Portal: a “gateway” to ocean dataThe Data Portal: a “gateway” to ocean data

Why do we need a Data Portal?– Each center of data provides a highly customized Web

sites for their data• but different datasets have different navigation and interface

characteristics• so the user faces a bewildering spectrum of data access

interfaces and locations

Data Portal is single, uniform, consistent “gateway” to ocean data in a common format

• User goes to a single location and sees a consistent interface• Complements the customized data access

31

Data Portal/Visualization/CollaborationData Portal/Visualization/Collaboration

Traditional users:ModelersForecastersResearchers

New users:EducatorsStudentsGeneral Public

Data & Data & Information UsersInformation Users

Distributed data Observed data Satellite data Data and information products Model outputsVisualizationUniform network accessUniform network access

32

WebBrowser

JavaApplication

User

Network

CORBA*

Client Support

Java Servlet

Graphics

One or more Web Servers

TAO data support

CORBA*

Data

Observing System Server

Data

Common Object Request Broker Architecture (CORBA) is an industry standard Middleware. CORBA is used in the NOAAServer software from which this effort will leverage. Based on performance indicators, Java Remote Method Invocation (RMI), an alternative middleware, could easily be substituted for CORBA.

CORBA*

Network

Data ServerData Portal

33

WebBrowser

JavaApplication

User

Network

CORBA*

Client Support

Java Servlet

Graphics

One or more Web Servers

Drifter Data support

CORBA*

Data

TAO data support

CORBA*

Data

Observing System Servers

Data

Common Object Request Broker Architecture (CORBA) is an industry standard Middleware. CORBA is used in the NOAAServer software from which this effort will leverage. Based on performance indicators, Java Remote Method Invocation (RMI), an alternative middleware, could easily be substituted for CORBA.

CORBA*

Network

Data ServersData Portal

34

WebBrowser

JavaApplication

User

Network

CORBA*

Client Support

Java Servlet

Graphics

One or more Web Servers

Drifter Data support

CORBA*

Data

TAO data support

CORBA*

Data

Observing System Servers

In-Situ/Satellite data support

CORBA*

Data

In-Situ/Satellite Data Servers

Model data support

CORBA*

Data

Model Output Servers

Data

Gridded data support

CORBA*

Data

Gridded Data ServersCommon Object Request Broker Architecture (CORBA) is an industry standard Middleware. CORBA is used in the NOAAServer software from which this effort will leverage. Based on performance indicators, Java Remote Method Invocation (RMI), an alternative middleware, could easily be substituted for CORBA.

CORBA*

Network

Data ServersData Portal

35

How do we build a Data Portal?How do we build a Data Portal?

Build on a proven prototype– connects 5 geographically distributed data

servers in Silver Spring, Boulder, Seattle– CORBA for network connections– unified interactive Java graphics – data from distributed servers are co-plotted

together on the same axis on the users desktop

http://www.pmel.noaa.gov/~nns/noaaserver/nodc-coads-tao.htmlhttp://www.pmel.noaa.gov/~nns/noaaserver/coads-tao-raster.html

36

BoulderCO

Prototype Data Portal: CDPPrototype Data Portal: CDP**

SeattleWA

Silver Spring

MD

HonoluluHI

*Climate Data Portal

37

Climate Data Portal Sample PlotsClimate Data Portal Sample Plots

38

Data Selection : Web InterfaceData Selection : Web Interface

Utilizes CORBA for network connections.

Utilizes EPIC Web Technology:– Java Applets– JavaScript– Java Servlets

Searches data by keywords, location and time ranges.

39

Web Interface sWeb Interface screen Shotscreen Shots

40

Other Technologies for the Other Technologies for the Collaboratory:Collaboratory:

Networks (100 Megabits/sec today, 10 Gigabits/sec in future)– Next Generation Internet (NGI) and Internet 2

Visualization– Interactive Java graphics– 3D, Virtual reality– Immersion technology

Collaboration tools– high-speed telecommunications systems for advanced

collaboration applications– tele-immersion systems allow individuals at different locations to

share a single virtual environment– Use networks not airplanes for collaboration

41

Virtual RealityVirtual Reality

Virtual Reality lets the scientist touch the data, move into it, and see it from different viewpoints– The realism of virtual reality enables the scientist and the lay

person to understand complex ideas more easily – Scientists using virtual reality affirm this new technology

discloses features of their data and model outputs which were undiscovered with standard visualization techniques

Virtual reality can be approachable and affordable Widens audience for scientific data and information

– Government administrators and decision makers– Educators and students– General public

Some examples follow…

Courtesy of Nancy N. Soreides, PMEL

42

Why use Virtual Reality?Why use Virtual Reality?

Virtual reality modeling language (VRML) rendering of temperatures and sea surface topography along the equator in the tropical Pacific, viewed from South America, showing the dynamics of El Nino and La Nina.

Using an inexpensive PC and a web browser with a free plug-in, the images can be rotated, animated, and zoomed. Changes in the equatorial Pacific during El Nino and La Nina are clearly understood by scientist and layman. http://www.pmel.noaa.gov/toga-tao/vis/vrml/ or http://www.pmel.noaa.gov/vrml

El Nino La Nina

Courtesy of Nancy N. Soreides, PMEL

43

Stereographic Virtual RealityStereographic Virtual Reality3D, interactive virtual reality visualizations are not difficult for a scientist to create or to view, from the web or from the desktop, and the effect can be enhanced dramatically by including the capability of stereographic viewing.

With a PC and a 99-cent pair of red/green sci-fi glasses, the spheres and vectors will pop out of the page in stereo, revealing the true 3D location of the fish, the steep slopes of the bathymetry, and the vertical motions near the submarine canyon.

The images can be rotated, animated and zoomed. http://www.pmel.noaa.gov/~hermann/vrml/stereo.html

Fish larvae and velocity vectors in a submarine canyon, from a circulation model of Pribolof Canyon in the Bering Sea. Use red/green glasses to see images on the right in stereo.

Stereo

Stereo

Courtesy of Nancy N. Soreides, PMEL

44

Immersive devices provide the graphical illusion of being in a three-dimensional space by displaying visual output in 3D and stereo, and by allowing navigation through the space.

Navigating through our virtual environments and viewing the data from different vantage points greatly increases our ability to perform analysis of scientific data.

The impact of such visualizations in person is stunning, and must be experienced by the scientist to be fully comprehended .

Users of these advanced immersion technologies affirm that no other techniques provide a similar sense of presence and insight into their datasets.

Immersive Virtual RealityImmersive Virtual Reality

Courtesy of Nancy N. Soreides, PMEL

45

The CAVEThe CAVE

The CAVE is a multi-person, high resolution, 3D graphics video and audio virtual environment. The size of a small room (10x10x10 foot), it consists of rear-projected screen walls and a front-projected floor.

Using special "stereoscopic" glasses inside a CAVE, scientists are fully immersed in their data. Images appear to float in space, with the user free to "walk" around them, yet maintain a proper perspective.

The CAVE was the first virtual reality technology to allow multiple users to immerse themselves fully in the same virtual environment at the same time.

View of the CAVE

Scientist inside the CAVE

CAVES have been deployed in academia, government, and industry, including NASA, NCAR, NCSA, Argon National Laboratory, Caterpillar Corp., General Motors, among others.

http://www.pyramidsystems.com/CAVE.htmlCourtesy of Nancy N. Soreides, PMEL

46

The ImmersaDeskThe ImmersaDesk

Courtesy of Nancy N. Soreides, PMEL

47

The FutureThe Future

“The development of scientific data manipulation and visualization capabilities requires an integrated systems approach … [including] the end-to-end flow of data from generation to storage to interactive visualization, and must support data retrieval, data mining, and sophisticated interactive presentation and navigation capabilities.”

“Data Exploration of petabyte databases will required both technology development and altered work patterns for research scientists and engineers.”*

* Data and Visualization Corridors, Report on the 1998 DVC Workshop Series, Edited by Paul H. Smith and John van Rosendale, Sponsored by the Department of Energy and the National Science Foundation, 1998.

Courtesy of Nancy N. Soreides, PMEL

48

Charles.Sun@noaa.gov