Post on 27-Jan-2016
description
MARFC Operational BackupMARFC Operational BackupA Case StudyA Case Study
January 26, 2006January 26, 2006
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 22
OutlineOutline
• ProblemProblem
• Proposed SolutionProposed Solution
• ResultsResults
• IssuesIssues
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 33
ProblemProblem• MARFC moving to new facilityMARFC moving to new facility• AWIPS unavailable for up to 6 days during AWIPS unavailable for up to 6 days during
movemove• How to conduct operations during AWIPS How to conduct operations during AWIPS
outageoutage– Maintain full operational outputMaintain full operational output
• Official ProductsOfficial Products• All web informationAll web information
– Worst case planning?Worst case planning?
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 44
Proposed SolutionProposed Solution
• Utilize in-house Linux serverUtilize in-house Linux server
• X-client LaptopsX-client Laptops
• ER WAN ConnectionER WAN Connection
• Data feed via LDMData feed via LDM
• Transmission via LDADTransmission via LDAD
• Additional seats for flooding and supportAdditional seats for flooding and support
• AWIPS OB4 BasisAWIPS OB4 Basis
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 55
Proposed Solution - ServerProposed Solution - Server
• Linux ServerLinux Server– Dell 4600 PowerEdge (late 2002)Dell 4600 PowerEdge (late 2002)– Dual 2 GHz Xeon CPUsDual 2 GHz Xeon CPUs– 1 GB Memory1 GB Memory– Raid 1 SCSI 73GB HDsRaid 1 SCSI 73GB HDs– Dual Power SuppliesDual Power Supplies– RH 7.3RH 7.3
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 66
Proposed Solution - AccessProposed Solution - Access• 4 Operational seats4 Operational seats• X access via 3 client laptopsX access via 3 client laptops
– Part of ER RFC Backup ProjectPart of ER RFC Backup Project– 1600x1050 resolution1600x1050 resolution
• 1 ½ screens1 ½ screens
– External mouse and keyboardExternal mouse and keyboard– Additional monitor for “non-AWIPS” displayAdditional monitor for “non-AWIPS” display
• Server console for HASServer console for HAS• 2 add’l clients for support and flooding2 add’l clients for support and flooding
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 77
Proposed Solution - CommsProposed Solution - Comms
• ER WAN ConnectionER WAN Connection
• Data feed via LDMData feed via LDM– Redundancy via ERH and SRHRedundancy via ERH and SRH
• Transmission via LDADTransmission via LDAD– Redundancy via PBZ with ERH backupRedundancy via PBZ with ERH backup
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 88
Proposed Solution - AppsProposed Solution - Apps• Based on AWIPS OB4Based on AWIPS OB4• Synchronize local apps changes Synchronize local apps changes
between AWIPS and serverbetween AWIPS and server– Crons as wellCrons as well
• Allow all auto processes to continueAllow all auto processes to continue– Shut off delivery via tokensShut off delivery via tokens
• Identify files to sync to go liveIdentify files to sync to go live– OFS, DB, controlOFS, DB, control
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 99
ResultsResults
• Difference in capabilitiesDifference in capabilities– No D2DNo D2D– No ArcView useNo ArcView use
• FOPFOP• Inundation mappingInundation mapping
– No 12Planet (within AWIPS only)No 12Planet (within AWIPS only)– 1 ½ vs. 3 screens1 ½ vs. 3 screens
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1010
ResultsResults• OperationsOperations
– Started with HAS AM shift 12/5/2005Started with HAS AM shift 12/5/2005– Product delivery easily shut off on AWIPS and Product delivery easily shut off on AWIPS and
initiated on backup via tokensinitiated on backup via tokens– Continued in backup mode until Thursday, 12/8, Continued in backup mode until Thursday, 12/8,
morning hydro shiftmorning hydro shift• AWIPS available for testing Wednesday night, 12/7AWIPS available for testing Wednesday night, 12/7
– Token reset shut off delivery from backup and Token reset shut off delivery from backup and initiated on AWIPSinitiated on AWIPS
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1111
ResultsResults
• PerformancePerformance– No forecaster perceived slowness in any No forecaster perceived slowness in any
operational applicationoperational application– No missed delivery of any text or graphical No missed delivery of any text or graphical
productproduct
• SUCCESS!SUCCESS!
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1212
IssuesIssues• Keeping in-step with AWIPSKeeping in-step with AWIPS
– Upgraded to OB5 on AWIPS prior to moveUpgraded to OB5 on AWIPS prior to move– Stayed on OB4 due to DB table changesStayed on OB4 due to DB table changes
• Not sure of best approach to incorporate changes of this Not sure of best approach to incorporate changes of this naturenature
• OB6?OB6?
• Help with problemsHelp with problems• Maintenance overhead – 2 systemsMaintenance overhead – 2 systems• Overall RFC solution?Overall RFC solution?
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1313
IssuesIssues
• Redundancy is critical!Redundancy is critical!
• 1 hour into backup operations one HD 1 hour into backup operations one HD on server failedon server failed– Continued with one HD with no problemsContinued with one HD with no problems
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1414
IssuesIssues
• Off-site computing capabilitiesOff-site computing capabilities
• If incapable of full operations support, If incapable of full operations support, what do you cut outwhat do you cut out– Customers expecting informationCustomers expecting information
• HD failure forced laptop server HD failure forced laptop server implementationimplementation
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1515
IssuesIssues
• Off-site computing capabilitiesOff-site computing capabilities– Part of ER RFC Backup projectPart of ER RFC Backup project– Dell 5150 “desktop replacement” laptopDell 5150 “desktop replacement” laptop– 1 year old1 year old– Pentium 4 3.0 GHzPentium 4 3.0 GHz– 1 GB Memory1 GB Memory– 1 100GB HD; 7200 RPM ATA 1 100GB HD; 7200 RPM ATA
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1616
IssuesIssues
• Off-site computing capabilitiesOff-site computing capabilities– Operations testOperations test
• 3 client laptops3 client laptops• Laptop displayLaptop display• Full cron loadFull cron load• Morning forecast operationsMorning forecast operations
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1717
IssuesIssues
• Off-site computing capabilitiesOff-site computing capabilities– Operations test – resultsOperations test – results
• SlowdownsSlowdowns– NMAP initiationNMAP initiation– Heavy disk access – e.g. Informix extractionsHeavy disk access – e.g. Informix extractions
• Slower than in-house serverSlower than in-house server• Acceptable by forecast staff knowing limitationsAcceptable by forecast staff knowing limitations
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1818
Performance ComparisonPerformance Comparison
• Identical cron instructionsIdentical cron instructions
• Handoff and execution of OFS jobs from Handoff and execution of OFS jobs from AWIPSAWIPS
• Laptop slowdown with heavy disk Laptop slowdown with heavy disk access activityaccess activity
• More analysis neededMore analysis needed
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1919
updatedb,db_purge,sys_clean
db_purge
???
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 2020
Off-site CapabilitiesOff-site Capabilities• ““Shoebox” server, aka ShuttlePCShoebox” server, aka ShuttlePC• ““Server-like”; luggableServer-like”; luggable• Dual CPUsDual CPUs• Multi-GB memory capabilitiesMulti-GB memory capabilities• 7200 RPM SATA II7200 RPM SATA II
– SCSI-like performance?SCSI-like performance?– Larger, cheaper than SCSILarger, cheaper than SCSI– RAID 1RAID 1
1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 2121
Off-site CapabilitiesOff-site Capabilities
• Less than $3000Less than $3000– Dual 64-bit AMD Opteron, 2.4 GHzDual 64-bit AMD Opteron, 2.4 GHz– 2 GB memory2 GB memory– Dual 120 GB 7200 RPM SATA IIDual 120 GB 7200 RPM SATA II
• RAID 1RAID 1
– GB ethernetGB ethernet– 19” LCD19” LCD– External keyboard and mouseExternal keyboard and mouse
The EndThe End