Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting -...
-
Upload
dale-cooper -
Category
Documents
-
view
217 -
download
0
Transcript of Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting -...
![Page 1: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/1.jpg)
Digital Packaging Processor - Overview
Gordon Hurford
Nov 7, 2011
EOVSA Technical Design Meeting - NJIT
![Page 2: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/2.jpg)
Digital Packaging Processor
• Role of DPP
• Overall assumptions and priorities
• Interface Overview
• Tasks and algorithms
• Hardware/software Implementation
-----------------------------------------------------------
• DPP Interface Details
![Page 3: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/3.jpg)
Data System Approach Fundamental Drivers
• Very limited software resources• Non-trivial data rate and volume • Automated analysis pipeline for efficient observing• Must have science-useable system in place by
September 2013• Data products to be readily useable by broader solar
community– Data products with preset parameters– Data products with user-selected parameters– Tools and support for experienced users
![Page 4: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/4.jpg)
Data System Approach Implications
• Monomode observing• Calibrated data archived in application-specific databases• Reliance on existing software packages
– Miriad package for calibration & mapping– RHESSI SolarSoft package for user interface and data product
display– RHESSI database model
• Err on side of over-rejection of data • Limited initial support for ‘nice-to-have’ options• Limited initial support for calibration refinements• Limited support for non-solar applications
![Page 5: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/5.jpg)
Data System Assumptions
• All information required for data analysis is written by the DPP to the Interim Data Base
• Engineering data acquisition, archiving and display is the responsibility of the ACC, and is “largely” decoupled from science data.
![Page 6: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/6.jpg)
Nomenclature
• Data frame = Interval representing data from one correlator cycle (20 ms, ~4000 channels with 500 MHz range)
• Spectral frame = Data corresponding to a complete frequency-agile cycle (nominal 1 second, 10s to 100’s of ‘science channels, 18 GHz rang) Corresponds to a state frame.
• Scan: Observing interval within which target and frequency cycling pattern is unchanged
![Page 7: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/7.jpg)
Role of Digital Packaging Processor
• To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base
• Real time, irreversible processing
![Page 8: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/8.jpg)
DPP Interface Overview
DPP
State Frame
ACC
Correlator
Start / End Scan Commands
Scan-independent Calibration Parameters
Scan Parameters
Frame parameters
Frame status report
<P>, <P2>,
Correlations
Interim Data Base
Miriad
format
Internal RFI
Database1 s timing tick
0.02 s timing tick
RFI
results
![Page 9: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/9.jpg)
DPP Task Timing
• Occasional – non operational– Accept, store and preprocess calibration parameters
• Scan initiation– Accept, store and preprocess scan-specific parameters
• Data frame (20 ms) – filter, and frequency-average correlator output
• Spectral frame (1 s) – Assemble, pre-calibrate, reformat and write data to Interim
database
• TBD – Format results and write to RFI database
![Page 10: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/10.jpg)
DPP – Stage 1 ProcessingEvery data frame (20ms)
• Evaluate kurtosis data to identify RFI-affected subbands as a function of frequency only.
• Save RFI statistics• Combine with pre-flagged subbands to
generate a “destination vector” for each subband
• Apply complex gains at subband level ???• Average subband data into spectral channels • Save 1st 3 moments of averages ???
![Page 11: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/11.jpg)
DPP Stage 2 ProcessingEvery spectral frame (1s)
– Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags
– Apply time-independent complex gains if available– Apply baseline corrections– Apply non-linearity corrections– Correct for attenuator settings– Correct for spectral simultaneity
• Miriad format this is no longer optional
– Convert visibility, uv and analysis-relevant state-frame data to Miriad-compatible format
– Write spectral frame to IDB– Report DPP status to state frame
![Page 12: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/12.jpg)
DPP - Implementation
• Original concept was to follow FASR plan for a cluster-based DPP
• Estimate processing requirements for EOVSA DPP at ~100 MIPS = 1/60 of FASR requirements
• Implementation will be based on a single multi-core machine
• Software organization will be compatible with migration to a cluster if necessary
![Page 13: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/13.jpg)
DPP Software Architecture
ACC State Frame Correlator IDB
Coordination Task
I/O, data assembly, no processing per se
Header Processing
Stage 1 Processing
Stage 2 Processing
Pointers within shared memory
Conventional, time-independent processing tasks
C1
C2 C3, C4 C2
Cn = core within a quad core processor or nodes in a cluster
DPP
RFI database
ParameterProcessing
C2
![Page 14: Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.](https://reader035.fdocuments.in/reader035/viewer/2022062409/5697bfe21a28abf838cb43a0/html5/thumbnails/14.jpg)
DPP Status• Software architecture and tasks identified
• Detailed definition of interfaces is underway
• EOVSA to Miriad format conversion being tested with FST data– (Fortran 77 for Miriad compatibility)
• Next:1. Complete definition of interfaces2. Code Stage 1 tasks (GH)
• Evaluate timing requirements3 Code Coordination task (JM). 3. Detailed definition of processing algorithms 4. Code of Stage 2 tasks
5. Machine selection and purchase • Development platform?
• Goal: Functional DPP to support prototype testing