Post on 02-Jan-2016
Eyes Off Glass
Dinesh GodeSr. Technical Specialist
Oct 9, 2007
Eyes Off Glass3
Project’s Goal
> Eliminate watching multiple monitors in Command Center
> Provide automated recovery related to incidents
Eyes Off Glass4
Phase 1 - Enhance Health Dashboards • Product
• Infrastructure
• Customer
• Software Deployment
Phase 2 - Expand drill down capabilities from the Dashboards
• Incident root cause
• Infrastructure view (example Pass & e-Ticketing)
• Scheduled Changes
Phase 3 - Automate recovery and notification• Eyes Off Glass
Three Phases / Targets
Eyes Off Glass5
> Exploit current automation tools
> Consolidate scheduling tool across all platforms• Expand use of current deployment tools
– ADS (Application Deployment System) is now used for e-Pricing, e-certified & Nexus
– Scheduler is now interfaced with ADS which is our homegrown Application Deployment System, Source Integrity & Integrity Manager
– Scheduler interfaced with TOS and does over 95% of all TPF utilities
> Enhance alerting and self recovery automation• Automated self recovery of production e-Pricing Query Servers
• Automated removal of staging query servers from load balancer
• Automated recovery and inserting staging query servers
> CMDB (Configuration Management Database)
How do we get there
Eyes Off Glass6
Three areas of focus for Automation
Routine Activities
Recovery &
Notification
Software Deploy
Eyes Off Glass7
Enterprise Monitoring
IBM Compatible
TEC
TOS, Scheduler,
z/OS
ServiceCenter
(Ticketing)
Network & Server
Monitoring
Application/Database
Monitoring
End User Experience
Business Matrix
Host Monitoring
SDEIA 232
1 2 3 4 5 6 7 8 9 10 11 12Link
100 10
13 14 15 16 17 18 19 20 21 22 23 24Link
100 10
Utilization%1 5 15 25 30 65
1x 2x 3x 4x 5x 6x
13x 14x 15x 16x 17x 18x
7x 8x 9x 10x 11x 12x
19x 20x 21x 22x 23x 24xMDI
IBM Compatible
TEC Console
Microsoft Operations Manager (MOM)
Service Manager
IBM Compatible
Eyes Off Glass8
Enterprise Automation
Scheduler
Incident Management Service Center (Ticketing)
z/OS
Tivoli
TEC
Changeman
TOSADSSource
Integrity
Integrity Manager
AutomationPoint
TPFGDS (Worldspan)
OSS (Delta Operation System
Support )
Deltamatic (Delta Res)
PreProd
Servers
Eyes Off Glass9
Examples of Dashboard
> Server / Network monitoring
> Tivoli Enterprise Console (TEC)
> Service Manager
Eyes Off Glass10
Server / Network Monitoring
Eyes Off Glass11
Server / Network Monitoring
Eyes Off Glass12
Server / Network Monitoring
Eyes Off Glass13
Server / Network Monitoring
Eyes Off Glass14
Server / Network Monitoring
Eyes Off Glass15
Tivoli Enterprise Console
Eyes Off Glass16
Tivoli Enterprise Console
Eyes Off Glass17
Service Manager
> Dashboard provides real-time consolidated view • Top down business view with drill
downs
– Consolidated technical & end user experience
– Customizable dashboards for individual views
• Standardize views for Command Center
– Business view of the health of major products
– Technical view of the current infrastructure health
– Real-time status view of software deployment
Services Health
Application
Strategic Customer
Suppliers
Critical Infrastructure
Service Manager
Eyes Off Glass18
Top 30 Airline Top 10 Hotels
SuppliersTop 10 Cars Top 5 Rail Top 5 Cruise
GDS Deltamatic
Services HealthOSS LTCUST Go Res PASS e Certified Trip Mgr MTAM WebVantagee-Pricing
Application Health
Credit Cards Other Suppliers
Critical Infrastructure HealthGDS Deltamatic OSS LTCUST Go Res PASS e Certified Trip Mgr MTAM WebVantagee-Pricing
VPN Firewalls MKS/SI Indicative Automation ??? ??? ??? ??? ???T-3’s
Fares E-Ticketing Availability Bookings MailSecurity Alert Auto Loads
Automation
Cust 1 Cust 2
Strategic Customer Business ViewCust 3 Cust 6 Cust …Cust 4 Cust 5
Service Manager (drill down)
Eyes Off Glass19
Benefits
> Automated recovery for TPF • Automated scheduled & recovery IPLs of GDS & Pre-Production, saving
precious down time
• Eliminates manual intervention for recovery
• Eliminates human error
> Production Scheduling• Worldspan scheduling system, handles about 700 TPF jobs per day (1 every
2 minutes)
• Worldspan scheduling system handles a grand total of about 23,000 jobs per day
• Interfaces exist to a hosted major airline’s z/OS system – jobs can be run on demand on either system and hosting information is automatically shared
Eyes Off Glass20
Benefits> Automated Loads of Middle Tier Servers
• Eliminates hours of manual tasks for loads
• Eliminates human error
• Moves Worldspan closer to a common deployment system
> Other Automation• Automated ticketing from all platforms
• TPF production, pre-production, and native testing logs made available throughout Worldspan in file form for in-depth analysis, and real-time form for activity tracking and real-time troubleshooting.
• TPF database changes have been automated.
> Eyes off Glass• Audible alerts
• Automated Recovery on all platforms
• Service Manager – An enterprise wide business oriented Monitoring System
Eyes Off Glass21
>Questions?Contact:
Dinesh Gode
dinesh.gode@travelport.com
Ernie South
ernie.south@travelport.com