How Do You Test an IT Service? Pete Green Tom Frey.
-
Upload
keagan-siddons -
Category
Documents
-
view
213 -
download
0
Transcript of How Do You Test an IT Service? Pete Green Tom Frey.
How Do You Test an IT How Do You Test an IT Service?Service?
Pete GreenPete Green
Tom FreyTom Frey
PurposePurpose
Explore evolving influence of IT Explore evolving influence of IT systems and services on weapons systems and services on weapons systemssystems
Increasing complexity of testingIncreasing complexity of testing Our experience foundation is in NMCIOur experience foundation is in NMCI Foci are the test processesFoci are the test processes
How Did We Get Here?How Did We Get Here?
JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN
TESP InputDelivered
Veridian sole sourcedBy EDS to conduct
NMCI CT&E
BSA conductedat 4 test sites
Master TestPlan Development
CT&E 1 TestingConducted
CT&E 3Testing
CT&E 2Testing
Master Test PlanCompleted & Delivered
Interoperability Testing
& Monitoring Plan Delivery
Contract SLAs mapped to COIs
NMCI ContractAwarded to EDS
NMCI DesignatedDOT&E Oversight
Program
DASN DirectsTESP for NMCI
2001 20022000
Rolling InRolling In
Compressed ScheduleCompressed Schedule No Traditional Program DocumentationNo Traditional Program Documentation Our Response Based on 4-step Test ProcessOur Response Based on 4-step Test Process Build Up (Crawl, Walk, Run)Build Up (Crawl, Walk, Run) Concentric TestingConcentric Testing Familiar FundamentalsFamiliar Fundamentals
FacesFaces ProcessesProcesses
Test ApproachTest Approach
BSABSA Phased TestingPhased Testing
CTE1 - DT (lab, bench)CTE1 - DT (lab, bench) CTE2 - DT (networks)CTE2 - DT (networks) CTE3 - DT/OT (operational vignettes)CTE3 - DT/OT (operational vignettes) OPEVAL - OTOPEVAL - OT
SLA based - derived CTPs, COIs,SLA based - derived CTPs, COIs,
Test ApproachTest Approach
SECURITY
Electronic Survivability
Average Availability
Configuration Integrity
Data Integrity
Network Delays
Application Delays Interoperability Reconfigurability Customer
Satisfaction Help Desk
Support Technology
Insertion Latency
Recovery Latency
Security Report Latency
1 Desktop
Hardware and Operating System
2 3 1 3 2
3 E-mail Services 1 3 4 2
4 Directory Services 1 2,3 4,5,6
10 NMCI Intranet Performance 1 2 3 4
12 Internet Access 1 2 3
15 Moves, Adds, and
Changes 1,2,4 5
23 Basic Help Desk Services
1,2,3,4 ,5,6,7
25 BAN/LAN
Communications Services
1,3 2 6 4
33 NMCI Security
Operational Services General
4,5,6 9 7,8
36 NMCI Security
Planning Services 1,2 , 4
NMCI SERVICE LEVEL AGREEMENTS
USER UPGRADES
END USER SERVICES
MAINTENANCE & HELP DESK SERVICES
COMMUNICATIONS SERVIC ES
INFORMATION ASSURANCE
NMCI OPERATIONAL
REQUIREMENTS INTEROPERABILITY & RELIABILITY NETWORK OPERATIONS & MAINTENANCE
NMCI MEASURES OF
EFFECTIVENESS
Test ApproachTest Approach
IT TestingIT Testing
Similar to Early Aviation TestingSimilar to Early Aviation Testing How fast can you go?How fast can you go? How high can you go?How high can you go?
OSI layer 1-3 centricOSI layer 1-3 centric Parameter focusParameter focus
IT T&E and the OSI 7-Layer IT T&E and the OSI 7-Layer ModelModel
WhereWe
need to be
MissionBased
PresentCommercial
ITTesting
Parametric
7654321
“Is This SystemEffective andSuitable?”
“How Fast Can ThisCar Go?”
Physical
Data Link
Network
Transport
Session
Presentation
Application
Product Line MaturityProduct Line Maturity
1950 1960 1970 1980 1990 2000
Sys
tem
s M
atu
rity
Aircraft
Avionics
Software
Network
s
C4I TestingC4I Testing
Success of testing tied to maturity of program Success of testing tied to maturity of program executionexecution
OTRROTRR Schedule ConstraintsSchedule Constraints IP systems generate lots of dataIP systems generate lots of data Program Execution curve is fractalProgram Execution curve is fractal
Processes on critical path provide 1st order Processes on critical path provide 1st order approximationapproximation
Use of surveysUse of surveys AAoo
Testing and Program Testing and Program ExecutionExecution
% Complete
Time
Full Rate Production
OPEVAL
OTRR
Idealized Program Execution Curve
Test IssuesTest Issues
NMCI equal to or better than legacy NMCI equal to or better than legacy networksnetworks
Critical path processesCritical path processes Seat cutoverSeat cutover Legacy applicationsLegacy applications
SLAs to COIs not a reversible functionSLAs to COIs not a reversible function Operationalizing the metricsOperationalizing the metrics Early and combined CT/OT to minimize riskEarly and combined CT/OT to minimize risk
Baseline System Baseline System AssessmentAssessment
Innovative contract vehicleInnovative contract vehicle Lack of traditional documentationLack of traditional documentation Needed to measure the performance Needed to measure the performance
of the legacy systems for comparison of the legacy systems for comparison with NMCIwith NMCI
BSA ToolsBSA Tools AMPAMP NetIQNetIQ MIB scriptsMIB scripts Help Desk LogsHelp Desk Logs BelManageBelManage SurveysSurveys
End userEnd user CIO and IT professionalsCIO and IT professionals Information AssuranceInformation Assurance
Use same tools and processes for OPEVALUse same tools and processes for OPEVAL
BSA Lessons LearnedBSA Lessons Learned
Configuration management is Configuration management is historicalhistorical
Good management makes an Good management makes an immediate impactimmediate impact Firewalls and IDSFirewalls and IDS RoutersRouters ConnectivityConnectivity
Legacy applicationsLegacy applications
TESP ProcessTESP Process
Similar to a TEMPSimilar to a TEMP More responsiveMore responsive If it was not necessary, it was not includedIf it was not necessary, it was not included If it was necessary, it was tailored to NMCIIf it was necessary, it was tailored to NMCI
From blank sheet to approval in under From blank sheet to approval in under 11 months11 months
Represented a strategy for success for Represented a strategy for success for NMCINMCI
The State of IT Testing The State of IT Testing TodayToday
Operational assessments of interoperability Operational assessments of interoperability and information assurance in fielded systemsand information assurance in fielded systems
Inverse correlation of interoperability and Inverse correlation of interoperability and information assurance information assurance
Mission Thread AnalysisMission Thread Analysis Blue, Green, and Red TeamsBlue, Green, and Red Teams Common ToolkitsCommon Toolkits Information Operations Threat Capabilities Information Operations Threat Capabilities
SummariesSummaries Feedback and feed forwardFeedback and feed forward
Interoperability vs. Information Interoperability vs. Information Assurance:Assurance:
A Complex ChallengeA Complex Challenge
Interoperability (Iop)
InformationAssurance (IA)
Iop(ax) = by
IA
An Enterprise System Needs to Balance Iop and IA
Testing FrameworksTesting Frameworks
Device TestingDevice TestingSystem TestingSystem Testing
System of Systems TestingSystem of Systems Testing
Work Flow ThreadsWork Flow Threads
Mission-Focused TestingMission-Focused Testing
A
Z
Example Mission ThreadExample Mission Thread(ATO Distribution)(ATO Distribution)
BCDPLNS WS
GCC Prioritized
Target List
KCOIC
1
TWM &CTB-AMServer
KCOICCTAPSWS
HTACC
CTAPSAPSServer
CTAPSADSServer
HTACCCTAPSWS
RELROK WAN
Integrated
Tasking Order7
KCOICCTAPSWS
KCOIC
GCCS-KC2/COPWS
CTAPSITO DBServer
RELROK WAN
ROK AF (Suwon)
CTAPSTGTWS
5
KUNSAN
CTAPSTGTWS
6
Training Facility
GCCS-KWebServer
RELROK WAN
CP TANGO
GCCS-KC2/COPWS
8
RELROK WAN
RELROK WAN
RELROK WAN
2 3
4
2 ACCTargetNomination
ITOSent toTL # 1,6
Level of PerfSituation
Conditions
Joint Mission Areas UJTL
Assigned CoCom
MissionsCoCom JMETs
Lessons Learned CCTI
Strategies
Commanders Training
Guidance
Exercise Objectives
Joint DoctrineTT&Ps
Training ObjectivesTraining
Objectives
BLUFOR OPLAN
OPFOR Campaign
Control Plan
Scenario
MSEL
Collection Management
Plan
Integrated Architectures
C4ISP
Exercise Comm/Sys
Architecture
Mission Functional
Threads
Mission Functional
Threads
Execution
Assessment Plan
Assessment Plan
Performance Data
Performance Analysis
After Action Review
System SMEs
Mission SMEs
Operational Testers
MISSION ISSUESMISSION ISSUES
EXERCISE ISSUESEXERCISE ISSUES
SYSTEM ISSUESSYSTEM ISSUES
PE
RF
OR
MA
NC
E
AS
SE
SS
ME
NT
PE
RF
OR
MA
NC
E
AS
SE
SS
ME
NT
Standards
Assessment Methodology Assessment Methodology FlowFlow
JTS
QUESTIONS?QUESTIONS?