Page 1LWS Teams Day JSOC Overview HMI-AIA Joint Science Operations Center Science Data Processing...

LWS Teams Day JSOC Overview

HMI-AIA Joint Science Operations Center Science Data Processing a.k.a. JSOC-SDP

Overview


HMI & AIA JSOC Concept

Science TeamForecast Centers

EPOPublic

Catalog

Primary Archive

HMI & AIAOperations

House-keeping

Database

MOCSDOGSDDS

Redundant Data

Capture System

10-DayArchive

OffsiteArchiv

eLMSA

L

OfflineArchiv

e

HMI JSOC Pipeline Processing System

DataExport& WebService

Stanford

LMSAL

High-LevelData Import

AIA AnalysisSystem

Local Archive

QuicklookViewing

GSFCWhite Sands

World


JSOC Dataflow Rates

Dataflow (GB/day)

Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture

2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)

2 processors

75

Level 1(HMI)

16 processors

Online Data

325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)

240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface

Dataflow (GB/day)

Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture

2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)

2 processors

75

Level 1(HMI)

16 processors

Online Data

325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)

240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface


JSOC-SDP Major Components


JSOC DRMS/SUMS Basic Concepts

• Each “image” is stored as a record in a data “series”.

• There will be many series: e.g. hmi_ground.lev0 is ground test data

• The image metadata is stored in a relational database – our Data Record Management System (DRMS)

• The image data is stored in SUMS (Storage Unit Management System) which itself has database tables to manage its millions of files.

• SUMS owns the disk and tape resources.

• Users interact with DRMS via a programming lauguage, e.g. c, FORTRAN, IDL.

• The “name” of a dataset is actually a query in a simplified DRMS naming language that also allows general SQL clauses.

• Efficient use of the system relies on direct use of DRMS

• Data may be exported from DRMS as FITS or other (TBD) protocols for remote users.

• Several Remote DRMS (RDRMS?) sites will be established which will “subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series.

• The JSOC may act as an RDRMS to access products made at remote sites.


Pipeline Program, “module”

Record Manage

ment

Keyword Access

Data Access

DRMS Library

Link Manage

ment

Utility LibrariesJSOC Science

Libraries

Record Cache

JSOC Pipeline Processing System Components

Database Server

SUMSStorage Unit

Management System

DRMSData Record

Management SystemSUMS Tape Farm

SUMS Disks

PUIPipeline User

Interface

Pipeline processing

plan

Processing script, “mapfile”

List of pipeline modules with needed datasets for input, output

Pipeline Operato

r

Processing History Log


Simple example – find and look at an image

P% P% show_keys "ds=hmi_ground.lev0[][2007.09.09_23:50/1m]" key=FSN,T_OBSFSN T_OBS0566684 2007.09.09_23:50:01_UTC0566685 2007.09.09_23:50:06_UTC0566686 2007.09.09_23:50:11_UTC0566687 2007.09.09_23:50:16_UTC0566688 2007.09.09_23:50:21_UTC0566689 2007.09.09_23:50:26_UTC0566690 2007.09.09_23:50:31_UTC0566691 2007.09.09_23:50:36_UTC0566692 2007.09.09_23:50:41_UTC0566693 2007.09.09_23:50:46_UTC0566694 2007.09.09_23:50:51_UTC0566695 2007.09.09_23:50:56_UTCP% P% ds9 `show_keys "ds=hmi_ground.lev0[566686]" seg=file -p -q`P%

Example of simple utility “module” called “show_”keys”

First find images in the minute starting 9 Sept at 23:50 then look atone with ds9


JSOC Export

• JSOC will support VSO access

• JSOC will also have a direct web access

• There will be remote DRMS/SUMS systems at key Co-I institutions

• ALL HMI and AIA data will be available for export at level-0 through standard products (level-1 for both and level-2 for HMI)

• It would be unwise to expect to export all of the data. It is simply not a reasonable thing to expect and would be a waste or resources.

• Our goal is to make all useful data easily accessible.

• This means “we” must develop browse and search tools to help generate efficient data export requests.


DRMS/SUMS Configuration

• DRMS and SUMS use the open-source PostgreSQL database engine.

• DRMS will run on a pair of dedicated servers likely 4-quad-core processors with up to 10TB fast disk.

• SUMS will consist of file servers with attached tape systems.

• SUMS will manage 200TB cache disk with 150TB/year of permanent archive for level-1 and higher level products.

• SUMS archive will use LTO-4 tapes in a robotic system with at least 10 drives and 2000 tapes near-line.

• The pipeline processing system will have about 50 processor cores dedicated to level-0 to level-1 processing and about 450 cores for higher level processing in the pipeline. HMI standard products will need about half of these.


Extra Info


Data Record Management Service

(DRMS)


(DRMS)

Pipeline client-server architecture

JSOC Disks

Analysis codeC/Fortran/IDL/Matlab

JSOC Library

Record Cache (Keywords+Links+Data paths)

OpenRecordsCloseRecords

GetKeyword, SetKeywordGetLink, SetLink

OpenDataSegmentCloseDataSegment

Pipeline client process

JSOC DisksJSOC DisksJSOC Disks


(DRMS)

Storage Unit Management Service

(SUMS)

Tape ArchiveService

SeriesTables

Record Catalogs

Storage UnitTables

Record CatalogsRecord

Tables

PostgreSQL DatabaseServer

AllocUnitGetUnitPutUnit

SQL queries

SQL queries

SQL queries

Storage unit transfer

Data Segment I/O

Storage unit transfer

Generic file I/O

DRMS socket protocol


Pipeline batch processing

• A pipeline batch is encapsulated in a single database transaction, “DRMS session”:

– If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session

– If failure occurs all data records are deleted and the database rolled back

– It is possible to commit data produced up to intermediate checkpoints during sessions

DRMS Service = Session Master

Input data records

Output data records

DRMS API

Registersession

DRMS API

Module 1

DRMS API

Module 2.2

…

DRMS API

Module N

DRMS API

Commit Data&

Deregister

Pipeline batch = atomic transaction

Record & SeriesDatabase SUMS

DRMS API

Module 2.1


HMI module status and MDI heritage

DopplerVelocity

HeliographicDoppler velocity

maps

Tracked TilesOf Dopplergrams

StokesI,V

ContinuumBrightness

Tracked full-disk1-hour averagedContinuum maps

Brightness featuremaps

Solar limb parameters

StokesI,Q,U,V

Full-disk 10-minAveraged maps

Tracked Tiles

Line-of-sightMagnetograms

Vector MagnetogramsFast algorithm

Vector MagnetogramsInversion algorithm

Egression andIngression maps

Time-distanceCross-covariance

function

Ring diagrams

Wave phase shift maps

Wave travel times

Local wave frequency shifts

SphericalHarmonic

Time series

Mode frequenciesAnd splitting

Brightness Images

Line-of-SightMagnetic Field Maps

Coronal magneticField Extrapolations

Coronal andSolar wind models

Far-side activity index

Deep-focus v and cs

maps (0-200Mm)

High-resolution v and cs

maps (0-30Mm)

Carrington synoptic v and cs maps (0-30Mm)

Full-disk velocity, sound speed, Maps (0-30Mm)

Internal sound speed

Internal rotation

Vector MagneticField Maps

MDI pipeline modules exist

Standalone production codes in use at Stanford

Research codes in use by team

Codes to be developed at HAO

Codes being developed in the community

Codes to be developed at Stanford

Primary observables

Intermediate and high level data products


AIA Level-2


JSOC Data Volumes from Proposal

this version modified to show the links to the hardware plan

img size channels

cadence

compress

HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49

AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59

HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93

AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004

HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0

AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0

HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0

AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0

HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0

AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0

HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0

AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0

HMI: tlm SU 553 100% 198 24

AIA: tlm SU 674 100% 241 30

HMI: Lev0, Lev-1, All Higher SU 730 412 93

AIA: Lev0, Lev1a SU 1,296 743 2,004

HMI Totals 68 71 610 118

AIA Totals 146 77 984 2,034

Combined (TB) 214 148 1,594 2,151

Tape shelf size (TB) 7,968

Tape shelf number of tapes - mixed density 11,257

Export

Near-line retain days

Near-line Cache (TB)

Tape Archive Fraction

Totals

2,026Local tape

LMSAL Link 1,193

1,227Offsite tape

Higher level 286

Level-1

Data Path Assumptions Combined (GB/day)

Fixed Disk cache (TB)

Online disk cache days

Perm disk per year (TB)

Level-01,610

Volume (GB/day)

Processed at

In from DDS

Tape per year (TB)

1,227

1,210

797


Sample of DRMS/SUMS Resource Assessment


JSOC Level-0 processing flow


JSOC-SDP Status

• Capture system complete, waiting testing with DDS at White Sands

• DRMS and SUMS virtually done

• Level-0 work progressing, to be done by instrument deliveries

• Level-1 work to start after delivery

• Getting serious about work on basic pipeline modules

• Review of status and computer hardware plans in early November


Stanford JSOC effort plan

HMI-SU Effort Distribution

0

100

200

300

400

500

600

700

800

900

1000

FY2006Q1

FY2006Q2

FY2006Q3

FY2006Q4

FY2007Q1

FY2007Q2

FY2007Q3

FY2007Q4

FY2008Q1

FY2008Q2

FY2008Q3

FY2008Q4

FY2009Q1

Pe

rce

nt

FT

E

On-Demand Support

Irradiance

Forecast - farside…

Coronal Inferences

Mag Field - Vector Field

Mag. Field - Line-of-Sight

Level 2 - Local HS Holography

Level 2 - Local HS Ring Diagram

Level 2 - Local HS Time Distance

Level 2 - Global HS

Level 2 - Quick Look

HK & FDS

Level-0

JSOC Verification & Test

Data Quality and Proc. Metadata

General Env.: cvs,os,oracle,compile

Archive h/w arch.

Processing Hardware Arch.

User Tools: API, Data Export, & Browsing

pui

Data Capture

DRMS

SUMS

Support for AIA

Page 1LWS Teams Day JSOC Overview HMI-AIA Joint Science Operations Center Science Data Processing...

Documents

Transcript of Page 1LWS Teams Day JSOC Overview HMI-AIA Joint Science Operations Center Science Data Processing...