The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data...
Transcript of The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data...
![Page 1: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/1.jpg)
The EOSDIS Data Server
Jeanne BehnkeEarth Science Data Information Systems/NASA GSFC
Code 423, Greenbelt MD 20771+1-301-614-5326
Presented at the THIC Meeting at the Naval Surface Warfare Presented at the THIC Meeting at the Naval Surface Warfare Presented at the THIC Meeting at the Naval Surface Warfare Presented at the THIC Meeting at the Naval Surface Warfare Center Center Center Center CarderockCarderockCarderockCarderock
Bethesda MDBethesda MDBethesda MDBethesda MD
October 3, 2000October 3, 2000October 3, 2000October 3, 2000
![Page 2: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/2.jpg)
2
A Big Mission!A Big Mission!
• EOS = Earth Observing System– Centerpiece of NASA’s Earth Science Enterprise– Collect Earth remote sensing data for a 15 year global
change research program
• EOSDIS Data Information System– Software architecture is designed to receive, process,
archive and distribute several terabytes of science data on a daily basis
– User community consists of several thousands of science and non-science users
– 7 major facilities across the US • Distributed Active Archive Centers (DAACs)
![Page 3: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/3.jpg)
3
EOS Terra in orbitEOS Terra in orbit
![Page 4: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/4.jpg)
4
Terra ImageTerra Image
![Page 5: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/5.jpg)
5
TERRA Image TERRA Image -- MODISMODIS
Cyclone Hudah in Indian Ocean taken March 29, 2000
![Page 6: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/6.jpg)
6
TERRA Image TERRA Image -- ASTERASTER
Reno, Nevada taken April 18, 2000
![Page 7: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/7.jpg)
7
TERRA Image TERRA Image -- MISRMISR
![Page 8: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/8.jpg)
8
LandsatLandsat Browse ImageBrowse Image
![Page 9: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/9.jpg)
9
EOSDIS has 3 segments:• Networks• Flight Ops• Science Data Processing System
EOSDIS is composed ofseveral geographicallydistributed elements thatwill appear as a single,integrated, logical entity
EOSDIS is working withNOAA and other agenciesto ensure long term availability of Earth science data
EOSDIS ConceptEOSDIS Concept
![Page 10: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/10.jpg)
10
JPL
ASF
EDC
LaRC
GSFC
NSIDC
ORNL
DISTRIBUTED ACTIVE DISTRIBUTED ACTIVE ARCHIVE CENTERSARCHIVE CENTERS
for EOS Datafor EOS Data
![Page 11: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/11.jpg)
11
Predicted Data Volumes for 2000Predicted Data Volumes for 2000
• Expect launch of EOS-Aqua and ADEOS satellites this year
• 260 different data products and sets of raw instrument data
• 1.6 TB of processed data stored daily by end of 2000DataCenter
ArchiveVolumesGB/Day
# ofgranulesper day
ArchiveVolumesIn TBper year
# o f Granulescumulativeper year
DistributionviaNetworkGB/day
Distributionvia tapeGB/day
EDC 522 6886 190 2,513,390 194 159GSFC 688 5545 251 2,023,925 226 226LaR C 312 2945 114 1,074,925 102 102NSIDC 22 1083 8 395,295 6 6Total 1544 16459 563 6,007,535 528 493
![Page 12: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/12.jpg)
12
SDPS System GoalsSDPS System Goals
• Flexible, Scaleable, Reliable• Use Open System standards• Support standard interface to Earth science to
enable coordinated data analysis• Maximize the use of COTS packages and respond to
technological advances and techniques• Inevitable change and new additions• Architecture to support these goals:
– EOSDIS Core System (ECS)
![Page 13: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/13.jpg)
13
ECS Context DiagramECS Context Diagram
Dat a Se rve rS u b s y s t e m
In t e ro p e ra b il i t y
Dat aMa n a g e m e n tS u b s y s t e m
Clie n t /In t e rn a l
&Ex t e rn a l
Us e rsManagementSubsystem
Pla n n in gS u b s y s t e m
Da t aPro c e s s in g
In g e s tS u b s y s t e m
Dat aCo lle c t io n Da t a
P r o v i d e r s
Us e r re g is t ra t io n ,Orde r s t at us ,Us e r p ro f ile
Dat a p ro c e s s ingr e q u e s t s
Da t aIn g e s t
P l a n s
Dat a &O n - De m a n dRe q u e s t s
Dis c o v e r y /A d v e r t is e m e n t s
Se arch andA c c e s s
A c q u i r e
Da t a /
Do c u m e n t s /A d v e r t is e m e n t s
S e r v i c e s
![Page 14: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/14.jpg)
14
Data Server SubsystemData Server Subsystem“Heart of the ECS system”“Heart of the ECS system”
• Object-oriented C++ on a multiplatform environment of SUNs and SGIs
• Three Software Configuration Items (CI)– Science DataServer CI
• DBMS, geospatial search, inventory
– Storage Management CI• Manages all peripherals including robotic silos
– Data Distribution CI• Places data in distribution location
• Ingest Subsystem CI is also significant to data archiving
![Page 15: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/15.jpg)
15
COTS PackagesCOTS Packages
• System uses ~ 75 Off-The-Shelf packages from commercial and government sources
• Principal COTS that impact design:– Sybase Relational DBMS/SQS - dbms and spatial query system– AMASS - file storage management system for robotic storage devices– Autosys - scheduling software for the processing system– Tivoli - system management tools– HP Openview- graphical tool for system management – RogueWave - libraries used to map components to objects– DCE - distributed computing environment– ClearCase - CM tool to manage completion of different builds– Remedy - trouble-ticketing software used across project
![Page 16: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/16.jpg)
16
Data Server SubsystemData Server SubsystemHardware Context DiagramHardware Context Diagram
Data Repository - STK
Diskstorage
Repository Manager - SGIs
Data DistributionManager - SUNs
Data Server Manager - SGIs
InventoryDatabase
DistributionPeripherals
Tape drives
Push/PullStaging
CD ROM
*configuration is similar at all sites
![Page 17: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/17.jpg)
17
Data RepositoryData Repository
DAAC Make/Model Qty Drive Type MediaCapacity
Total #of Mediain silos
GSFC StorageTek STKPowderhorn
4 13 D3 drives10 9840 drives
50 GB40 GB
12,000
NSIDC StorageTek STKPowderhorn
1 3 D3 drives 50 GB 600
EDC StorageTek STKPowderhorn
3 14 D3 drives8 9840 drives
50 GB40 GB
7,700
LaRC StorageTek STKPowderhorn
2 8 D3 drives8 9840 drives
50 GB40 GB
3,100
DAAC Make/Model Qty Drive Type MediaCapacity
GSFC Exabyte tape drivesCD Rom Writers
82
8mm tape 50 GB600 MB
NSIDC Exabyte tape drivesCD Rom Writers
22
8mm tape 50 GB600 MB
EDC Exabyte tape drivesCD Rom WritersD3 tape drive
821
8mm tape
D3 tape
50 GB600 MB50 GB
LaRC Exabyte tape drivesCD Rom Writers
22
8mm tape 50 GB600 MB
ArchiveRobotic Storage
DistributionSystems
![Page 18: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/18.jpg)
18
STK SiloSTK Silo
![Page 19: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/19.jpg)
19
![Page 20: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/20.jpg)
20
Drive CabinetDrive Cabinet
![Page 21: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/21.jpg)
21
Typical Cartridge Media for SiloTypical Cartridge Media for Silo
![Page 22: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/22.jpg)
22
Mass Storage I/O SystemMass Storage I/O System
• Consists of silo, RAID disk, server hosts– Capable of 40 MB/s throughput sustained at all times
(3.5 TB of data per day)
• Able to push 3 to 4 times as much data because of double buffer mechanism in the storage management system design– Minimizes stress on robotics– Creates our own persistent cache
![Page 23: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/23.jpg)
23
Mass Storage I/O SystemMass Storage I/O System
• Utilize Volume Groups (groups of tapes together)– Group tapes by ‘science data type’ (for example, all
Landsat data in a silo is grouped together on a specified collection of tapes)
– Enables load balancing– Assures minimum performance levels– Allows logical management of the archive
• Additional information in two poster papers at this conference– Fault Tolerant Design – Scalable Architecture for Maximizing Concurrency
![Page 24: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/24.jpg)
24
Archive OperationsArchive Operations
• Strive for an automated archive system– Continuous connection to the archive systems by operations
personnel– At least two operations personnel at each site:
• Principle activities include error notification; backup; monitoring; problem resolution
– Strive for lights out administration
• Support several modes at each site for system upgrade– One operational mode; 2 test modes
• Scheduled maintenance includes hardware monitoring, media monitoring, format and cleanup
![Page 25: The EOSDIS Data Server · 2000. 10. 5. · The EOSDIS Data Server Jeanne Behnke Earth Science Data Information Systems/NASA GSFC Code 423, Greenbelt MD 20771 +1-301-614-5326 jeanne.behnke@gsfc.nasa.gov](https://reader035.fdocuments.in/reader035/viewer/2022071500/611e5dd2eab8705f532fcae4/html5/thumbnails/25.jpg)
25
ConclusionConclusionSo How Are We Doing?So How Are We Doing?
DAAC Type Archive SizeEDC Landsat7/
ASTER/MODIS Land
20 TB274 GB/day
NSIDC MODIS Land(snow
products)
9 TB25 GB/ day
GSFC MODIS L1/Atmos/Ocean
50 TB300 GB/day
LaRC MISR 18 TB88 GB/day
Archive Size to
Date