Page 1JSOC Overview August 2007 HMI Status HMI is virtually done. –Virtually similar to but not...

JSOC Overview August 2007

HMI Status

• HMI is virtually done.

– Virtually similar to but not in fact

– Front window issue resolved, flight window now at LMSAL will be installed next week

– Light leaks to be fixed next week.

– Some electronics boards need conformal coating

– Some software issues remain

• Vibration and EMI/EMC testing done

• Thermal Balance and Thermal Vacuum testing starting week after next.

• In Air and Vacuum Calibration next few weeks.

• Pre-Ship Review set for 25 Sept.

• Ship date estimated 5 October.

• Launch date slip to probably 15 Jan 2009 pending review at HQ.

• Some serious implications from delay


JSOC-SDP Status

• Capture system complete, waiting testing with DDS at White Sands

• DRMS and SUMS virtually done

• Level-0 work progressing, to be done by delivery

• Level-1 work to start after delivery

• Getting serious about work on basic pipeline modules


HMI & AIA JSOC Architecture

Science TeamForecast Centers

EPOPublic

Catalog

Primary Archive

HMI & AIAOperations

House-keeping

Database

MOCDDS

Redundant Data

Capture System

30-DayArchive

OffsiteArchiv

e

OfflineArchiv

e

HMI JSOC Pipeline Processing System

DataExport& WebService

Stanford

LMSAL

High-LevelData Import

AIA AnalysisSystem

Local Archive

QuicklookViewing

housekeeping GSFCWhite Sands

World


JSOC Dataflow Rates

Dataflow (GB/day)

Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture

2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)

2 processors

75

Level 1(HMI)

16 processors

Online Data

325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)

240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface

Dataflow (GB/day)

Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture

2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)

2 processors

75

Level 1(HMI)

16 processors

Online Data

325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)

240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface


JSOC-SDP Major Components


JSOC Pipeline Processing System Components

Database Server

SUMSStorage Unit

Management System

DRMSData Record

Management SystemSUMS Tape Farm

SUMS Disks

Pipeline Program, “module”

Record Manage

ment

Keyword Access

Data Access

DRMS Library

Link Manage

ment

Utility LibrariesJSOC Science

Libraries

Record Cache

PUIPipeline User

Interface

Pipeline processing

plan

Processing script, “mapfile”

List of pipeline modules with needed datasets for input, output

Pipeline Operato

r

Processing History Log


Data Record Management Service

(DRMS)


(DRMS)

Pipeline client-server architecture

JSOC Disks

Analysis codeC/Fortran/IDL/Matlab

JSOC Library

Record Cache (Keywords+Links+Data paths)

OpenRecordsCloseRecords

GetKeyword, SetKeywordGetLink, SetLink

OpenDataSegmentCloseDataSegment

Pipeline client process

JSOC DisksJSOC DisksJSOC Disks


(DRMS)

Storage Unit Management Service

(SUMS)

Tape ArchiveService

SeriesTables

Record Catalogs

Storage UnitTables

Record CatalogsRecord

Tables

PostgreSQL DatabaseServer

AllocUnitGetUnitPutUnit

SQL queries

SQL queries

SQL queries

Storage unit transfer

Data Segment I/O

Storage unit transfer

Generic file I/O

DRMS socket protocol


Pipeline batch processing

• A pipeline batch is encapsulated in a single database transaction, “DRMS session”:

– If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session

– If failure occurs all data records are deleted and the database rolled back

– It is possible to commit data produced up to intermediate checkpoints during sessions

DRMS Service = Session Master

Input data records

Output data records

DRMS API

Registersession

DRMS API

Module 1

DRMS API

Module 2.2

…

DRMS API

Module N

DRMS API

Commit Data&

Deregister

Pipeline batch = atomic transaction

Record & SeriesDatabase SUMS

DRMS API

Module 2.1


HMI module status and MDI heritage

DopplerVelocity

HeliographicDoppler velocity

maps

Tracked TilesOf Dopplergrams

StokesI,V

ContinuumBrightness

Tracked full-disk1-hour averagedContinuum maps

Brightness featuremaps

Solar limb parameters

StokesI,Q,U,V

Full-disk 10-minAveraged maps

Tracked Tiles

Line-of-sightMagnetograms

Vector MagnetogramsFast algorithm

Vector MagnetogramsInversion algorithm

Egression andIngression maps

Time-distanceCross-covariance

function

Ring diagrams

Wave phase shift maps

Wave travel times

Local wave frequency shifts

SphericalHarmonic

Time series

Mode frequenciesAnd splitting

Brightness Images

Line-of-SightMagnetic Field Maps

Coronal magneticField Extrapolations

Coronal andSolar wind models

Far-side activity index

Deep-focus v and cs

maps (0-200Mm)

High-resolution v and cs

maps (0-30Mm)

Carrington synoptic v and cs maps (0-30Mm)

Full-disk velocity, sound speed, Maps (0-30Mm)

Internal sound speed

Internal rotation

Vector MagneticField Maps

MDI pipeline modules exist

Standalone production codes in use at Stanford

Research codes in use by team

Codes to be developed at HAO

Codes being developed in the community

Codes to be developed at Stanford

Primary observables

Intermediate and high level data products


AIA Level-2


JSOC Data Volumes from Proposal

this version modified to show the links to the hardware plan

img size channels

cadence

compress

HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49

AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59

HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93

AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004

HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0

AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0

HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0

AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0

HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0

AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0

HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0

AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0

HMI: tlm SU 553 100% 198 24

AIA: tlm SU 674 100% 241 30

HMI: Lev0, Lev-1, All Higher SU 730 412 93

AIA: Lev0, Lev1a SU 1,296 743 2,004

HMI Totals 68 71 610 118

AIA Totals 146 77 984 2,034

Combined (TB) 214 148 1,594 2,151

Tape shelf size (TB) 7,968

Tape shelf number of tapes - mixed density 11,257

Export

Near-line retain days

Near-line Cache (TB)

Tape Archive Fraction

Totals

2,026Local tape

LMSAL Link 1,193

1,227Offsite tape

Higher level 286

Level-1

Data Path Assumptions Combined (GB/day)

Fixed Disk cache (TB)

Online disk cache days

Perm disk per year (TB)

Level-01,610

Volume (GB/day)

Processed at

In from DDS

Tape per year (TB)

1,227

1,210

797


Stanford JSOC effort plan

HMI-SU Effort Distribution

0

100

200

300

400

500

600

700

800

900

1000

FY2006Q1

FY2006Q2

FY2006Q3

FY2006Q4

FY2007Q1

FY2007Q2

FY2007Q3

FY2007Q4

FY2008Q1

FY2008Q2

FY2008Q3

FY2008Q4

FY2009Q1

Pe

rce

nt

FT

E

On-Demand Support

Irradiance

Forecast - farside…

Coronal Inferences

Mag Field - Vector Field

Mag. Field - Line-of-Sight

Level 2 - Local HS Holography

Level 2 - Local HS Ring Diagram

Level 2 - Local HS Time Distance

Level 2 - Global HS

Level 2 - Quick Look

HK & FDS

Level-0

JSOC Verification & Test

Data Quality and Proc. Metadata

General Env.: cvs,os,oracle,compile

Archive h/w arch.

Processing Hardware Arch.

User Tools: API, Data Export, & Browsing

pui

Data Capture

DRMS

SUMS

Support for AIA

Page 1JSOC Overview August 2007 HMI Status HMI is virtually done. –Virtually similar to but not...

Documents

Transcript of Page 1JSOC Overview August 2007 HMI Status HMI is virtually done. –Virtually similar to but not...