OVERVIEW OF SERVER-SIDE PROCESSING Doug Clough SYNERGETICS Engineered Systems 12 January 2009 .
-
Upload
dominic-west -
Category
Documents
-
view
214 -
download
0
Transcript of OVERVIEW OF SERVER-SIDE PROCESSING Doug Clough SYNERGETICS Engineered Systems 12 January 2009 .
OVERVIEW OF SERVER-SIDE PROCESSING
Doug Clough
SYNERGETICS Engineered Systems
12 January 2009
www.ptagis.org
12 January 2009
Pacific States Marine Fisheries Commission
Server-Side Processing
• Data Basics (15 min)– PTAGIS Specification Document– Tagging & Interrogation Files– Core Database Tables
• PTTP – Client and Server Implementations (15 min)– Request processing– Request transformation
• FDVL – Field Data Validating Loader (30 min)– Validation and Loading– User Feedback: PTTP and Email– Loader performance report
• IDL – Interrogation Data Loader (30 min)– Validation and Loading– Loader performance report– Interrogation System performance reports
12 January 2009
Pacific States Marine Fisheries Commission
Server-Side Processing
• Data Basics (15 min)– PTAGIS Specification Document– Tagging & Interrogation Files– Core Database Tables
• PTTP – Client and Server Implementations (15 min)– Request processing– Request transformation
• FDVL – Field Data Validating Loader (30 min)– Validation and Loading– User Feedback: PTTP and Email– Loader performance report
• IDL – Interrogation Data Loader (30 min)– Validation and Loading– Loader performance report– Interrogation System performance reports
12 January 2009
Pacific States Marine Fisheries Commission
Data Basics
• PIT Tag Specification Document
• Tagging & Interrogation Files
• Core Database Tables
12 January 2009
Pacific States Marine Fisheries Commission
Data Basics
• PIT Tag Specification Document• Tagging & Interrogation Files
• Core Database Tables
12 January 2009
Pacific States Marine Fisheries Commission
PIT Tag Specification Document
• PIT Tag Steering Committee (PTSC)
• Tagging & Interrogation Files– Tagging File pg 14– MiniMon Interrogation File pg 27– MULTIMON File pg 30
• Core Database Tables– Appendix F– Excludes ‘process-specific tables’
A subcommittee of the Fish Passage Advisory Committee of the Columbia Basin Fish and Wildlife Authority. The Committee is made up of technical representatives of the Basin’s fisheries agencies and tribes. The Committee's function is to provide guidance in the development of tagging methodologies and data standards and to serve as a technical forum for the operation of PTAGIS and the PIT tag data collection and data distribution system.
V. Glossary of Terms – pg 71
12 January 2009
Pacific States Marine Fisheries Commission
Data Basics
• PIT Tag Specification Document
• Tagging & Interrogation Files• Core Database Tables
12 January 2009
Pacific States Marine Fisheries Commission
Tagging Files
• Multiple versions are anticipated– All versions maintained on server
– FDVL provides insert / update / delete capability
– Latest version is in database
• FILE TITLE Header Record– Auto-generated by P3
– xxxYYDDD.zzz• xxx Valid
Coord ID• YYDDD Day of
Year• zzz
Arbitrary Extension
• External file name must match internal FILE TITLE
• When choosing zzz extension, user must be sure not to create a conflict with an existing file
12 January 2009
Pacific States Marine Fisheries Commission
MiniMon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records YES
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.an• sss Site Code
• YYDDD Day of Year
• a Sequence character
• n 1 or 2 (Primary/Backup)
• External file name must match internal FILE TITLE
12 January 2009
Pacific States Marine Fisheries Commission
MiniMon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records YES
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.an• sss Site Code
• YYDDD Day of Year
• a Sequence character
• n 1 or 2 (Primary/Backup)
• External file name must match internal FILE TITLE
FishTag
Interrogation
12 January 2009
Pacific States Marine Fisheries Commission
MiniMon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records YES
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.an• sss Site Code
• YYDDD Day of Year
• a Sequence character
• n 1 or 2 (Primary/Backup)
• External file name must match internal FILE TITLE
TimerTag
Interrogation
12 January 2009
Pacific States Marine Fisheries Commission
MiniMon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records YES
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.an• sss Site Code
• YYDDD Day of Year
• a Sequence character
• n 1 or 2 (Primary/Backup)
• External file name must match internal FILE TITLE
TASSTransceiverDiagnostics
12 January 2009
Pacific States Marine Fisheries Commission
Multimon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records NO
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.nnn• sss Site
Code• YYDDD Day of
Year• nnn
Sequence number100’s primary, 200’s backup
• External file name must match internal FILE TITLE
12 January 2009
Pacific States Marine Fisheries Commission
Multimon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records NO
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.nnn• sss Site
Code• YYDDD Day of
Year• nnn
Sequence number100’s primary, 200’s backup
• External file name must match internal FILE TITLE
DGEControlState
Events
12 January 2009
Pacific States Marine Fisheries Commission
Multimon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records NO
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.nnn• sss Site
Code• YYDDD Day of
Year• nnn
Sequence number100’s primary, 200’s backup
• External file name must match internal FILE TITLE
DGEInterrogation
Events
12 January 2009
Pacific States Marine Fisheries Commission
Multimon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records NO
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.nnn• sss Site
Code• YYDDD Day of
Year• nnn
Sequence number100’s primary, 200’s backup
• External file name must match internal FILE TITLE
TASSTransceiverDiagnostics
12 January 2009
Pacific States Marine Fisheries Commission
Multimon Interrogation Files
• Multiple versions not supported– Files are maintained on server
– IDL provides insert capability only
• Format defined in SpecDoc?– Interrogation records NO
– Transceiver diagnosticsNO
• Varies by model / firmware
• FILE TITLE Record– Auto-generated by MiniMon
– sssYYDDD.nnn• sss Site
Code• YYDDD Day of
Year• nnn
Sequence number100’s primary, 200’s backup
• External file name must match internal FILE TITLE
TimerTag
Interrogation
12 January 2009
Pacific States Marine Fisheries Commission
Data Basics
• PIT Tag Specification Document
• Tagging & Interrogation Files
• Core Database Tables
12 January 2009
Pacific States Marine Fisheries Commission
Core Database Tables
• Have existed from ‘day one’
• All contain data of interest to ‘typical user’ except …
– TEST_TAGS– TEST_TAG_DATA
• Many non-core tables exist to support PTAGIS processes
12 January 2009
Pacific States Marine Fisheries Commission
Server-Side Processing
• Data Basics (15 min)– PTAGIS Specification Document– Tagging & Interrogation Files– Core Database Tables
• PTTP – Client and Server Implementations (15 min)– Request processing– Request transformation
• FDVL – Field Data Validating Loader (30 min)– Validation and Loading– User Feedback: PTTP and Email– Loader performance report
• IDL – Interrogation Data Loader (30 min)– Validation and Loading– Loader performance report– Interrogation System performance reports
12 January 2009
Pacific States Marine Fisheries Commission
PTTP – Client and Server Implementations
• Request processing– PTTP has processing, reporting, and messaging capabilities– What happens to ‘data payload’ depends on request and type of data
12 January 2009
Pacific States Marine Fisheries Commission
PTTP – Client and Server Implementations
• Request processing– PTTP has processing, reporting, and messaging capabilities– What happens to ‘data payload’ depends on request and type of data
Primary job
12 January 2009
Pacific States Marine Fisheries Commission
PTTP – Client and Server Implementations
• Request processing– PTTP has processing, reporting, and messaging capabilities– What happens to ‘data payload’ depends on request and type of data
• Request transformation– PTTP deployment began in Jan 2001
• Previously used upload or email for tagging files• Previously used upload or email for interrogation files• Deployment took time; backward compatibility was required
– Interrogation files emailed to [email protected] • Turned into PTTP transactions by PTTP_EmailClient• PTTP stages files for processing by IDL
– Tagging files transmitted by PTTP • Original email interface to FDVL was retained• PTTP_Dispatcher invokes PTTP_FDVL via PTTP_Handler • Sends email to [email protected] to invoke FDVL
Important secondary job
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Overview
PTTP ‘Transaction’ Zip-Files
• Name: TX<UUID>.zip
• Transmitted by FTP
• Unpacked in per-transaction directory– One serialized PTTP_Req (XML)– Zero or more data files
• Results available in PTTP_Ack (XML)
Remote PTTP Clients
• P3 - Tagging files
• Minimon, Multimon – Interrogation files
Local PTTP Clients
• PTTP_EmailClient– Generates PTTP transaction for
interrogation data sent by email to [email protected]
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
PTTP_Req examines data payload. If interrogation files are included, the transaction identifier is posted to input queues of auxiliary analytical processes:
•TASS_Monitor
•DGE_Monitor
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
For ‘Invocation’ event PTTP_Dispatcher …
• Invokes handler
• Handler reads possible Input file
• Handler performs action and queues‘Completion’ event
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
For ‘Invocation’ event PTTP_Dispatcher …
• Invokes handler
• Handler reads possible Input file
• Handler performs action and queues‘Completion’ event
File Type Handler
INTERROGATION PTTP_intIDL
MULTIMON PTTP_mulIDL
TAGGING PTTP_FDVL
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
For ‘Invocation’ event PTTP_Dispatcher …
• Invokes handler
• Handler reads possible Input file
• Handler performs action and queues‘Completion’ event
For ‘Completion’ event PTTP_Dispatcher
• Instantiates PTTP_Ack to log completion
• Calls genNextEvent() to queue possible follow-on ‘Invocation’ event
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
For ‘Invocation’ event PTTP_Dispatcher …
• Invokes handler
• Handler reads possible Input file
• Handler performs action and queues‘Completion’ event
For ‘Completion’ event PTTP_Dispatcher
• Instantiates PTTP_Ack to log completion
• Calls genNextEvent() to queue possible follow-on ‘Invocation’ event
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Detail
Process flow …
1) watchFtp launches processPittagFtp
2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req
3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event
4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type
For ‘Invocation’ event PTTP_Dispatcher …
• Invokes handler
• Handler reads possible Input file
• Handler performs action and queues‘Completion’ event
For ‘Completion’ event PTTP_Dispatcher
• Instantiates PTTP_Ack to log completion
• Calls genNextEvent() to queue possible follow-on ‘Invocation’ event
12 January 2009
Pacific States Marine Fisheries Commission
Server-Side Processing
• Data Basics (15 min)– PTAGIS Specification Document– Tagging & Interrogation Files– Core Database Tables
• PTTP – Client and Server Implementations (15 min)– Request processing– Request transformation
• FDVL – Field Data Validating Loader (30 min)– Validation and Loading– User Feedback: PTTP and Email– Loader performance report
• IDL – Interrogation Data Loader (30 min)– Validation and Loading– Loader performance report– Interrogation System performance reports
12 January 2009
Pacific States Marine Fisheries Commission
FDVL – Field Data Validating Loader
• Validation and Loading
• User Feedback– PTTP– Email
• Loader performance report
12 January 2009
Pacific States Marine Fisheries Commission
FDVL – Field Data Validating Loader
• Validation and Loading• User Feedback
– PTTP– Email
• Loader performance report
12 January 2009
Pacific States Marine Fisheries Commission
FDVL – Recording ‘Field Events’ in Database Tables
Original File Types
• Tagging
• ReleaseInformation
• MonitoredRelease
• Mortality
12 January 2009
Pacific States Marine Fisheries Commission
FDVL – Recording ‘Field Events’ in Database Tables
Since 2001
• Tagging
• ReleaseInformation
• MonitoredRelease
• Mortality
Since 2001
Per-fish flag codes are usedto indicate recaptureand mortality eventswithin a Tagging file
12 January 2009
Pacific States Marine Fisheries Commission
FDVL Overview
Staging eliminated with
fd_quickLoad
in August 2002
12 January 2009
Pacific States Marine Fisheries Commission
FDVL - Meta-Data
• Stored and maintained in MS Access database
– spec_doc.mdb
• Exported to CSV-files for use– FD_FileMetaData.txt
– FD_FileTableMap.txt
• CSV-files populate in-memory data structures in fdc_receiveEmail.pl
– SpecDoc
– FileMap
12 January 2009
Pacific States Marine Fisheries Commission
Specification Document: FDVL Model of PTAGIS File Types
12 January 2009
Pacific States Marine Fisheries Commission
FDVL: Mapping of Record Fields to Columns in Database Tables
12 January 2009
Pacific States Marine Fisheries Commission
User’s Tagging File
• Must contain all mandatory records
• Values in optional fields must be valid
• As Validator parses file it – Assembles FileData tree
– Tallies ‘leaves’ on the tree
– Assesses their validity
– Errors exist if valid count is less than item count
• Processing stops with user message if errors exist
• Otherwise Loader is told to run
– May be running already
– May have been halted
12 January 2009
Pacific States Marine Fisheries Commission
Validator Log File
For each record group• item_count
– Tally of required records
• valid_count– Tally of valid records
If errors are encountered• Specifics are posted to log• Log entries are incorporated
into user message
12 January 2009
Pacific States Marine Fisheries Commission
Loader Log File
LOAD• Each body record loaded via
insert/update DB procedure• Record context determines
which proc is called– TAG_TAG
– TAG_REC
– TAG_MRT
• Duplicates encountered produce log message
– SelfDup or OtherDup
PURGE• Records unmarked by DB
proc are deleted
AGGREGATION• Record disposition by table
is tallied and logged
12 January 2009
Pacific States Marine Fisheries Commission
Loader Log File
LOAD• Each body record loaded via
insert/update DB procedure• Record context determines
which proc is called– TAG_TAG
– TAG_REC
– TAG_MRT
• Duplicates encountered produce log message
– SelfDup or OtherDup
PURGE• Records unmarked by DB
proc are deleted
AGGREGATION• Record disposition by table
is tallied and logged
Determined by analysis of
flag codes in TAGGING file
BODY records
12 January 2009
Pacific States Marine Fisheries Commission
FDVL Performance
• Performance report built to– Understand usage patterns
– Inform spreading of work away from heavy usage times if necessary
• Allows tracking performance week by week and year by year
– FDVL has always been able to outpace demand
– Loading rates have grown due to faster hardware
– Current average roughly 4000 tagging file ‘body’ records per minute
12 January 2009
Pacific States Marine Fisheries Commission
FDVL Performance
• Performance report built to– Understand usage patterns
– Inform spreading of work away from heavy usage times if necessary
• Allows tracking performance week by week and year by year
– FDVL has always been able to outpace demand
– Loading rates have grown due to faster hardware
– Current average roughly 4000 tagging file ‘body’ records per minute
Design has scaled well
12 January 2009
Pacific States Marine Fisheries Commission
FDVL – Versioning of Submitted Files
• Database table: fd_file– Date and time of most recent submittal
– Submittal sequence number
– File archival directory (logical path name)
• Database table: fd_submit– Timestamps of arrival, validation, and loading
for each submittal
– Counts of body records by type
• item_count
• recap_count
• mort_count
• dot_out_count
• dup_count
12 January 2009
Pacific States Marine Fisheries Commission
FDVL – Versioning of Submitted Files
• Database table: fd_file– Date and time of most recent submittal
– Submittal sequence number
– File archival directory (logical path name)
• Database table: fd_submit– Timestamps of arrival, validation, and loading
for each submittal
– Counts of body records by type
• item_count
• recap_count
• mort_count
• dot_out_count
• dup_count
Originally used for TAGGING files only
Now used for TAGGING and INTERROGATION files
12 January 2009
Pacific States Marine Fisheries Commission
FDVL – Versioning of Submitted Files
• Database table: fd_file– Date and time of most recent submittal
– Submittal sequence number
– File archival directory (logical path name)
• Database table: fd_submit– Timestamps of arrival, validation, and loading
for each submittal
– Counts of body records by type
• item_count
• recap_count
• mort_count
• dot_out_count
• dup_count
Originally used for TAGGING files only
Now used for TAGGING and INTERROGATION files
Not applicable for INTERROGATION files
12 January 2009
Pacific States Marine Fisheries Commission
Server-Side Processing
• Data Basics (15 min)– PTAGIS Specification Document– Tagging & Interrogation Files– Core Database Tables
• PTTP – Client and Server Implementations (15 min)– Request processing– Request transformation
• FDVL – Field Data Validating Loader (30 min)– Validation and Loading– User Feedback: PTTP and Email– Loader performance report
• IDL – Interrogation Data Loader (30 min)– Validation and Loading– Loader performance report– Interrogation System performance reports
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Loader Invocation
IDL is launched by
UNIX cron
‘every 3 hours’
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Loader Invocation
IDL is launched by
UNIX cron
‘every 3 hours’
We haven’t done thatfor a long time!
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Loader Invocation
PTTP
IDL is launched by
UNIX cron
‘every 3 hours’
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Loader Invocation
PTTP
IDL is launched by
UNIX cron
‘every 3 hours’
Files are pre-processed byidl_validateFile.pl
one-by-one as IDL runs
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Operation in Serial Mode
Output from idl_validateFile.pl
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Operation in Serial Mode
Output from idl_validateFile.pl
Logic is contained in load_int_data_cdbp
‘called’ database procedure
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Motivation for Parallel Loading Capability
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Primary Storage Structure For Parallel Operation
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Operation in Parallel Mode
Partition the Data Stream
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Operation in Parallel Mode
Drop Indexes
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Operation in Parallel Mode
Load Data
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Operation in Parallel Mode
Build Indexes
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report• Interrogation System performance reports
– TimerTag Report– TASS– CEA– DGE
12 January 2009
Pacific States Marine Fisheries Commission
IDL Performance
• Performance report allows year-to-year comparisons
– Loading rates have grown due to faster hardware
– Overall IDL run times have decreased even though data volumes have increased
• With small data sets – Per-file overhead
dominates
• With large data sets– Most time actually spent
loading data– Average loading rates
exceed 1500 rows/min
• Haven’t resorted to Parallel mode for many years
– It’s our ‘ace in the hole’ if we need to scale further
12 January 2009
Pacific States Marine Fisheries Commission
IDL Performance
• Performance report allows year-to-year comparisons
– Loading rates have grown due to faster hardware
– Overall IDL run times have decreased even though data volumes have increased
• With small data sets – Per-file overhead
dominates
• With large data sets– Most time actually spent
loading data– Average loading rates
exceed 1500 rows/min
• Haven’t resorted to Parallel mode for many years
– It’s our ‘ace in the hole’ if we need to scale further
Design has scaled well
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
12 January 2009
Pacific States Marine Fisheries Commission
Type of timer-tag depends on transceiver …
FS1001 Physical timer-tag at each coil FS1001A Virtual timer-tag in transceiver
FS1001M Virtual timer-tag in transceiver
Timer Tag
- Known PIT-code emitted periodically to signal that coil and transceiver are working properly
Transceiver
Coil
Physical Timer Tag- Actual tag mounted close to coil- Normally shunted to ground- Transceiver un-grounds to test- More likely detected than actual fish
Virtual Timer Tag- Code generated by transceiver circuitry- Transceiver contains antenna emulator- Code is presented to emulator to test- Likelihood of detection similar to actual fish
12 January 2009
Pacific States Marine Fisheries Commission
TimerTag Performance Report
• Launched by UNIX cron after IDL runs
• Red cells indicate missing TimerTag detections
• Black vertical bars (none in this example) indicate inter-file ‘data gaps’
• HTML links in Extension row display associated interrogation file in browser
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
12 January 2009
Pacific States Marine Fisheries Commission
TASSTransceiver Annunciator / Statistical Summarizer
• Alarms displayed as “red light” – Site annunciator - SLC500 comms failure– Coil annunciator - Timer-tag failure
• Corrective actions posted to event log
• Transceiver diagnostics reported– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time
12 January 2009
Pacific States Marine Fisheries Commission
- ReportingPolicy determines what is displayed- Established in configuration file- Any number of Annunciator Panels can be defined
12 January 2009
Pacific States Marine Fisheries Commission
TASSTransceiver Annunciator / Statistical Summarizer
• Alarms displayed as “red light” – Site annunciator - SLC500 comms failure– Coil annunciator - Timer-tag failure
• Corrective actions posted to event log• Transceiver diagnostics reported
– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time
12 January 2009
Pacific States Marine Fisheries Commission
Text entered here is posted to the site event-log forum and text file.
Sets lights green
Displays report
Displays file
12 January 2009
Pacific States Marine Fisheries Commission
TASSTransceiver Annunciator / Statistical Summarizer
• Alarms displayed as “red light” – Site annunciator - SLC500 comms failure– Coil annunciator - Timer-tag failure
• Corrective actions posted to event log
• Transceiver diagnostics reported– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time
12 January 2009
Pacific States Marine Fisheries Commission
Three types of transceiver currently in use
FS1001 Alarm & parameter reporting FS1001A Enhanced diagnostics (noise)
FS1001M Multiple “coils”
Transceiver
- Transmitter / Receiver- Decodes Tag-ID from digital signal- Periodically energizes TimerTag
Transceiver - “Unique” ON or OFF
Coil
Single-Coil
Multiplexed
12 January 2009
Pacific States Marine Fisheries Commission
Three types of transceiver currently in use
FS1001 Alarm & parameter reporting FS1001A Enhanced diagnostics (noise)
FS1001M Multiple “coils”
Transceiver
- Transmitter / Receiver- Decodes Tag-ID from digital signal- Periodically energizes TimerTag
Transceiver - “Unique” ON or OFF
Coil
Single-Coil
Multiplexed
Content and format of diagnostic messages written to data stream
depend on transceiver model
12 January 2009
Pacific States Marine Fisheries Commission
TASSTransceiver Annunciator / Statistical Summarizer
• Alarms displayed as “red light” – Site annunciator - SLC500 comms failure– Coil annunciator - Timer-tag failure
• Corrective actions posted to event log
• Transceiver diagnostics reported
– Basic: Alarms and parameter values– Enhanced: Noise and temperature plots vs. time
12 January 2009
Pacific States Marine Fisheries Commission
Actual time-stamp value
“Almost-same” time-stamp; actual times differ
12 January 2009
Pacific States Marine Fisheries Commission
TASSTransceiver Annunciator / Statistical Summarizer
• Alarms displayed as “red light” – Site annunciator - SLC500 comms failure– Coil annunciator - Timer-tag failure
• Corrective actions posted to event log
• Transceiver diagnostics reported– Basic: Alarms and parameter values
– Enhanced: Noise and temperature plots vs. time
12 January 2009
Pacific States Marine Fisheries Commission
Min MaxMean
+/- 3-Sigma
Click the light for a particular coil
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
Parsing & Statistics
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
DataStructures & Algorithms
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
DataStructures & Algorithms
Hydraulic Topology
Specifically: “How fish move through the facility”
12 January 2009
Pacific States Marine Fisheries Commission
“OK, it’s an interrogation site …”
JUVENILE?
ADULT?
… but there are many differences!!
Transceiver Model
FS1001FS1001AFS1001M
Adult and Juvenile sites have this in
common
How Could We Describe An INTERROGATION Site?
12 January 2009
Pacific States Marine Fisheries Commission
“OK, it’s an interrogation site …”
JUVENILE?
ADULT?
… but there are many differences!!
Transceiver Model
FS1001FS1001AFS1001M
Adult and Juvenile sites have this in
common
How Could We Describe An INTERROGATION Site?
What Else?
12 January 2009
Pacific States Marine Fisheries Commission
Attributes in Common: Adult & Juvenile Facilities
ADULT JUVENILE
Same technology is used for detecting fish: transceiver_id One or more transceivers transceiver_mfgr Same transceiver makes transceiver_model Same range of transceiver models antenna_id One or more antennas per transceiver
Water flows downstream through both kinds of site: flow_node_id Water enters and leaves site through one or more “nodes” node_type For example: Weir, SlideGate flow_path_id Water follows “paths” between “nodes” rf_energy_where “Where the fish are!”: WeirOrifice, ClosedPipe, Flume
12 January 2009
Pacific States Marine Fisheries Commission
“OK, it’s an interrogation site …”
JUVENILE?
ADULT?
… but there are many differences!!
Transceiver Model
FS1001FS1001AFS1001M
Adult and Juvenile sites have this in
common
How Could We Describe An INTERROGATION Site?
What Differences?
12 January 2009
Pacific States Marine Fisheries Commission
Attributes: Juvenile Fish Bypass Facility
Only juvenile fish, all travelling downstream: node_type For example: FishSeparator, SlideGate
Fish are entrained in flow - Movement through “monitors” is “continuous”: continuous_flow_y_n “Y” - Yes, what goes in must come out!
12 January 2009
Pacific States Marine Fisheries Commission
Attributes: Adult Fish Ladder
Mix of adult and juvenile fish, travelling both upstream and downstream: node_type For example: OverflowWeir, SlottedWeir, CountingWindow
Fish are not entrained in flow - Movement through “monitors” is “discontinuous”: continuous_flow_y_n “N” - No, a fish can turn around and retrace its path.
12 January 2009
Pacific States Marine Fisheries Commission
Attributes: Adult Fish Ladder
Mix of adult and juvenile fish, travelling both upstream and downstream: node_type For example: OverflowWeir, SlottedWeir, CountingWindow
Fish are not entrained in flow - Movement through “monitors” is “discontinuous”: continuous_flow_y_n “N” - No, a fish can turn around and retrace its path.
Or ‘in-stream’ site
12 January 2009
Pacific States Marine Fisheries Commission
Existing SiteConfigurationManagement Schema
What if: Activity-TechnicalConfiguration Schema
antenna_group monitor
antenna site_coilATC Schema (strawman)Activity-TechnicalConfiguration Schema
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
Cumulative Detection Efficiency Analyzer
12 January 2009
Pacific States Marine Fisheries Commission
Terminology - Monitor
Interrogation Monitor
- Two or more “coils” (antennas)- Flow continuity- High probability of detecting fish
Transceiver Energizes the coil
CoilEnergizes PIT-tags as
they pass through
12 January 2009
Pacific States Marine Fisheries Commission
CEA Probability of Missing Fish is valid only where …
Interrogation Monitor
- Two or more “coils” (antennas)- Flow continuity- High probability of detecting fish
Transceiver Energizes the coil
CoilEnergizes PIT-tags as
they pass through
Fish are entrained in flow
12 January 2009
Pacific States Marine Fisheries Commission
CEA Textual Report
• Launched by UNIX cron– After each IDL run
– Once per week for aggregation
• Provides quick overview of interrogation activity at a site
• Count of fish seen and missed by coil at each monitor
• Probability of missing fish at each monitor
• Daily and by-week aggregations – Stored in database
– Accessible from web site
12 January 2009
Pacific States Marine Fisheries Commission
CEA Textual Report
• Launched by UNIX cron– after each IDL run
– once per week for aggregation
• Provides quick overview of interrogation activity at a site
• Count of fish seen and missed by coil at each monitor
• Probability of missing fish at each monitor
• Daily and by-week aggregations – Stored in database
– Accessible from web site
Depends on flow continuity
12 January 2009
Pacific States Marine Fisheries Commission
CEA Web Report
• Web interface gives easy access to current and historical results
• Report is useful as a simple indicator of fish activity
• For this reason, results are produced even for sites where flow continuity does not exist
• FishSeen link explains …
12 January 2009
Pacific States Marine Fisheries Commission
Visio file • UML ComponentDiagram (shown)• UML StaticStructure• UML UseCase
Textual documentation on ‘sockeye’ • /home/ptagdev/ptagis3/source/cum_det_eff
«executable»cea_processYesterdayWeekly.sh
«executable»cea_processYesterdayDaily.sh
«executable»p3_daily_4.sh
«call» «executable»env_common_defs.sh
«executable»env_ptagis_master.sh
The ‘YESTERDAY’ environment variable is defined here
«executable»CEA_processHistoricalData.pl
«executable»p3_daily_3.sh
«executable»p3_weekly_3.sh
«executable»CEA_Direct_weekly.sh
«executable»CEA_Convert_Weekly.sh
«executable»CEA_Direct_daily.sh
«executable»CEA_Convert_Daily.sh
«call»«call»
«import»
$CEA_REP_DIR/
«executable»run_cea_direct.pl
«executable»cea_direct.pl
ALL_weekly.txt ALL_daily.txt
Produce one output file for each active interrogation site.
Concatenate per-site filesto produce the “ALL” file
«executable»CEA_convertA2S.pl
«executable»CEA_reportHistoricalData.pl
«executable»CEA_reportYTDData.pl
«table»PTAGIS3_InterrogationData
«table»PTAGIS3_CumulativeEffData
«executable»CEA_generateReport.pl
$CEA_ARCH_DIR/
$CEA_PUB_FTP_DIR/
NOTE: ‘weekly’ and ‘2-week’ invocations are no-op unless YESTERDAY is a Sunday
CEA_AST-SQL: Component DiagramDoug Clough, SYNERGETICSVer. 1.0 21 Nov 2003
CEA_AST-SQL: Component Diagram
ReportWriter files and outputs
CSV output files
CEA DocumentationCumulative Efficiency Analyzer
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
12 January 2009
Pacific States Marine Fisheries Commission
DGEDiversion Gate Efficiency Analyzer
• DGE Configuration File basics• How DGE replays real-world events
• Last-known PLC values– Sample rates (Data > Reports > SystemGen > DGE PLC)
– Switch settings (Farther down the same page)
• Last 24-hours & YTD efficiencies– Action summary by control context– Gate actions - success & failure - by hour
12 January 2009
Pacific States Marine Fisheries Commission
DGE Configuration File Basics
Multimon’s SxC implementation
• Needs to know what gate is controlled by a given monitor– This information derives from facility ‘port and transceiver’ layout
• Does not need to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to
success or failure of diversion attempt
DGE
• Needs to know what gate is controlled by a given monitor
• Also needs to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to
success or failure of diversion attempt– This information derives from facility ‘concrete and steel’ layout
which is not represented in the database– DGE’s site configuration files provide this information
12 January 2009
Pacific States Marine Fisheries Commission
DGE Configuration File Basics
Multimon’s SxC implementation
• Needs to know what gate is controlled by a given monitor– This information derives from facility ‘port and transceiver’ layout
• Does not need to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to
success or failure of diversion attempt
DGE
• Needs to know what gate is controlled by a given monitor
• Also needs to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to
success or failure of diversion attempt– This information derives from facility ‘concrete and steel’ layout
which is not represented in the database– DGE’s site configuration files provide this information
Network Topology
Hydraulic Topology
12 January 2009
Pacific States Marine Fisheries Commission
DGE Configuration File Basics
Multimon’s SxC implementation
• Needs to know what gate is controlled by a given monitor– This information derives from facility ‘port and transceiver’ layout
• Does not need to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to
success or failure of diversion attempt
DGE
• Needs to know what gate is controlled by a given monitor
• Also needs to know– Where a diverted fish is seen subsequent to diversion– What detections subsequent to diversion mean with respect to
success or failure of diversion attempt– This information derives from facility ‘concrete and steel’ layout
which is not represented in the database– DGE’s site configuration files provide this information
Network Topology
M4 – If it’s to provide this functionality …
Hydraulic Topology
Both Are Needed
12 January 2009
Pacific States Marine Fisheries Commission
Little Goose Smolt Bypass and Collection (GOJ)
12 January 2009
Pacific States Marine Fisheries Commission
http://www.psmfc.org/pittag/DGE/persist/2004/GOJ/GOJ_Config.txt
12 January 2009
Pacific States Marine Fisheries Commission
AST-formatted ASCII file:
- Indentation captures hierarchical relationships- Designed as an “outline” to be read by people- Think of it as XML without tags
persistenceHashAssemblage
file
12 January 2009
Pacific States Marine Fisheries Commission http://www.psmfc.org/pittag/DGE/persist/2004/GOJ/GOJ_Config.txt
12 January 2009
Pacific States Marine Fisheries Commission http://www.psmfc.org/pittag/DGE/persist/2004/GOJ/GOJ_Config.txt
Variable names
Site topology
List of Gates controlled by this Monitor
12 January 2009
Pacific States Marine Fisheries Commission
Variable names
Variable-name Mapping
User-friendly
Internal global
Site topology
List of Gates controlled by this Monitor
12 January 2009
Pacific States Marine Fisheries Commission
Internal globalInternal
local
Site topology
Gate behavior
ControlList PLC switch settings that affect this gate
12 January 2009
Pacific States Marine Fisheries Commission
Site topology
Gate behavior
What to do in a given
ControlContext
ControlContext Set of ControlList values at any given time
12 January 2009
Pacific States Marine Fisheries Commission
DGEDiversion Gate Efficiency Analyzer
• DGE Configuration File basics
• How DGE replays real-world events• Last-known PLC values
– Sample rates (Data > Reports > SystemGen > DGE PLC)
– Switch settings (Farther down the same page)
• Last 24-hours & YTD efficiencies– Action summary by control context– Gate actions - success & failure - by hour
12 January 2009
Pacific States Marine Fisheries Commission
PTTP Drives DGE
• $DGE_LOG serves as input queue for DGE_Monitor
• Arrival of $TX_ID in queue wakes up DGE_Monitor
• If $TX_ID contains files from sites in SiteList, launch DGE_ProcessFile
• DGE Process Overview1) Instantiate site model
2) Process available files
3) When queue is empty, persist component objects
4) Launch DGE_Driver_Reports
$PTTP_SERVER_PERSIST_DIR
Several 'Mini/Multimon' files from one Site transmitted in a single PTTP 'zip-file'
DGE_Monitor.pl
DGE_processFile
.pl
DGE Archives for Site
"volatilize" "persist"
InterrogationSite PTTP: <TX_ID>.zip
Per-$TX_ID Directory
$DGE_LOG
PTTP_Req
$TX_ID
"tail -f"
"Un-zip"
SiteList
$MULTIMON_DIR
Store each file
File listINT or MUL
DGE_Driver_Reports.pl
R_AST
1
2
12 January 2009
Pacific States Marine Fisheries Commission
DGE Process
1) Instantiate site model from configuration file and per-component persistence files
2) Retrieve ‘events’ from interrogation file
3) Delegate ‘control state’ events to PLC object
4) Propagate ‘detection events’ throughout gate-monitor ‘flow paths’
5) Register diversion successes and failures
6) Persist until next time
DGE_Monitor
DGE_processFile.pl
$MULTIMON_DATA
$DGE_PERSIST_DIR
.../<SiteCode>
DGE_Detection
Facility
FileList
<SiteCode>YYJJJ.nnn
DGE_Component
<SiteCode>_Config.txt
DGE_MultiCoilDetector
DGE_Gate
DGE_Prog
Controller
DGE_SlideGate
DGE_2WayGate
DGE_3WayGate
As dictated by Config file
<SiteCode>
Loop through the file
DGE_Event
DGE_Detection
Event
DGE_ControlState
Event
<SiteCode>.archive
Instantiate
Persist
1
2
3
12 January 2009
Pacific States Marine Fisheries Commission
Read config file and instantiate site model
Set PLC variables from persistence file
Establish relationships between site components
12 January 2009
Pacific States Marine Fisheries Commission
Sets a PLC variable
Read interrogation file and generate DGE_Events
ControlState_Event
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor downstream from DiversionGate
Tag-ID Coil-ID
[DiversionRequest]
12 January 2009
Pacific States Marine Fisheries Commission
DGE_Component instances propagate Interrogation_Event
upstream
12 January 2009
Pacific States Marine Fisheries Commission
DGE_Component instances propagate Interrogation_Event
upstream
12 January 2009
Pacific States Marine Fisheries Commission
DGE_Component instances propagate Interrogation_Event
upstream
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor that controls a DiversionGate
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor that controls a DiversionGate
DGE_Component instances propagate Interrogation_Event
upstream
DGE_Monitor instance forwards Interrogation_Event to
associated DiversionGate(s)
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor that controls a DiversionGate
DGE_Component instances propagate Interrogation_Event
upstream
DGE_Monitor instance forwards Interrogation_Event to
associated DiversionGate(s)
12 January 2009
Pacific States Marine Fisheries Commission
Diversion_Gate retrieves ControlContext from PLC
Interrogation_Event at Monitor that controls a DiversionGate
DGE_Component instances propagate Interrogation_Event
upstream
DGE_Monitor instance forwards Interrogation_Event to
associated DiversionGate(s)
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor that controls a DiversionGate
DGE_Component instances propagate Interrogation_Event
upstream
DGE_Monitor instance forwards Interrogation_Event to
associated DiversionGate(s)
ControlContext and DiversionRequest determine required GateAction
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor downstream from DiversionGate
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor downstream from DiversionGate
DGE_Component instances propagate Interrogation_Event
upstream
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor downstream from DiversionGate
DGE_Component instances propagate Interrogation_Event
upstream
Since the Tag-ID was seen here, the gate
was CLOSED
DiversionGate remembers requested GateAction; compares with actual
12 January 2009
Pacific States Marine Fisheries Commission
Interrogation_Event at Monitor downstream from DiversionGate
DGE_Component instances propagate Interrogation_Event
upstream
DiversionGate tallies actions by ControlContext:Request:Actual
Since the Tag-ID was seen here, the gate
was CLOSED
12 January 2009
Pacific States Marine Fisheries Commission
DGEDiversion Gate Efficiency Analyzer
• DGE Configuration File basics
• How DGE replays real-world events
• Last-known PLC values– Sample rates (Data > Reports > SystemGen > DGE PLC)
– Switch settings (Farther down the same page)
• Last 24-hours & YTD efficiencies– Action summary by control context– Gate actions - success & failure - by hour
12 January 2009
Pacific States Marine Fisheries Commission
DGEDiversion Gate Efficiency Analyzer
• DGE Configuration File basics
• How DGE replays real-world events
• Last-known PLC values– Sample rates (Data > Reports > SystemGen > DGE PLC)
– Switch settings (Farther down the same page)
• Last 24-hours & YTD efficiencies– Action summary by control context– Gate actions - success & failure - by hour
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
12 January 2009
Pacific States Marine Fisheries Commission
IDL – Interrogation Data Loader
• Validation and Loading
• Loader performance report
• Interrogation System performance reports– TimerTag Report– TASS– CEA– DGE
There have been others!!
12 January 2009
Pacific States Marine Fisheries Commission
DEA – 1995-?Instantaneous Detection Efficiency Analyzer
• Developed when 400khz tags were in use– Transceivers provided no diagnostic messages– Statistical analysis gave detection efficiency estimate
• Ring-buffer data structure maintained interrogation context– Multiple interrogations of a single tag still counted as one fish– Computed probability based on ‘tally groups’ of 20 fish
• Not used much since advent of ‘advanced’ transceivers– Self-configuring; costs nothing to let it run
12 January 2009
Pacific States Marine Fisheries Commission
• Fish not entrained in water flow– Adult fish can jump over weirs– A single fish can traverse weir upstream and downstream
• Ring-buffer data structure maintained interrogation context– Multiple interrogations of a single tag still counted as one fish– Computed probability based on ‘tally groups’ of 20 fish
• Retired when counting windows were instrumented– Provided ‘close enough’ approximation of flow continuity
ADE – 2001-2008Adult Detection Efficiency Analyzer
12 January 2009
Pacific States Marine Fisheries Commission
Server-Side Processing
• Data Basics (15 min)– PTAGIS Specification Document– Tagging & Interrogation Files– Core Database Tables
• PTTP – Client and Server Implementations (15 min)– Request processing– Request transformation
• FDVL – Field Data Validating Loader (30 min)– Validation and Loading– User Feedback: PTTP and Email– Loader performance report
• IDL – Interrogation Data Loader (30 min)– Validation and Loading– Loader performance report– Interrogation System performance reports
12 January 2009
Pacific States Marine Fisheries Commission
Concluding Remarks
• PTAGIS continually evolving
• Development of server-side tools driven by evolution of – PIT tag technology– Transceiver capabilities
• Stable processing environment has maximized time available for addressing needs of
– Researchers– PTOC O&M Staff
• Major factors providing stability– UNIX (POSIX) standards– Language and API ‘interface stability’ of Perl and CPAN modules– Ingres RDBMS
• Practically ‘hands-off’ day by day operation
• Continued evolution of VLDB features
• Good Tech Support