Dr. Steven J. Hutchison Principal Deputy Developmental Test and...
Transcript of Dr. Steven J. Hutchison Principal Deputy Developmental Test and...
Shift Left Nov 2012 Page-1
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Dr. Steven J. Hutchison Principal Deputy
Developmental Test and Evaluation November 2012
Shift Left Nov 2012 Page-2
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Persistent, rapidly composable, secure representation of the Joint Information Environment
Test & Evaluation
Operations
Performance Reliability
DT&E for Complex Systems
System Integration Labs
Training
Experimentation
Modeling & Simulation
Cyber Range
JIOR
JMETC
Interoperability Information Security
Shift Left Nov 2012 Page-3
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
The DoD Acquisition Model
Shift Left Nov 2012 Page-4
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Test, Evaluation, Certification
Late to Need!
DIACAP Security T&E
Shift Left Nov 2012 Page-5
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Hindsight is 20-20
What did we know?
What did we test?
To reduce discovery late in the acquisition lifecycle, • test in mission context, • against realistic threat,
and….. Shift Left!
DOT&E COCOM/Service
Interop & IA Assessments
Fielded systems: • Interoperability issues • IA Vulnerabilities
Compliance with IA Controls and
Interoperability Standards and Profiles
are necessary but not sufficient
Shift Left Nov 2012 Page-6
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Net Ready KPP New Role for DASD(DT&E)
New Language • “DISA will ensure JITC leverages
previous, planned and executed DT&E and OT&E tests and results to support joint interoperability test certification and eliminate test duplication.”
• “DASD(DT&E) shall approve Developmental Test and Evaluation plans in support of Joint Interoperability Test Certification as documented in the TEMP. JITC shall advise DASD (DT&E) regarding the adequacy of test planning in support of Joint Interoperability Test Certification.”
DASD(DT&E) approves adequacy of Interoperability test planning
CJCSI 6212
Shift Left Nov 2012 Page-7
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Information Assurance Policy
Information Assurance compliance activities need to be integrated into DT&E and included in the TEMP
Shift Left Nov 2012 Page-8
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Information Assurance What’s Changing?
• Implements Risk Management Framework (RMF) instead of Mission Assurance Category/Confidentiality Level (MAC/CL)
• Adopts new guidance from the National Institute of Standards and Technology (NIST) and Committee on National Security Systems Instruction (CNSSI) documents on Cybersecurity
• Goes beyond IA and adopts the term: “Cybersecurity”
• Lexicon Changes – “Certification and Accreditation” becomes “Assessment and Authorization” – “Designated Approving Authority (DAA)” becomes “Authorizing Official (AO)” – “Certifying Authority” becomes “Security Control Assessor”
Threat = Any event with potential to cause harm to the network Vulnerability = Absence/weakness of safeguards to protect the network
Risk = Likelihood that a threat will realize or exploit a vulnerability
Shift Left Nov 2012 Page-9
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Implementing Cybersecurity What’s Being Proposed?
DASD(DT&E):
• Oversight of test planning in support of Cybersecurity C&A(A&A)
• Establish procedures to ensure that DT&E authorities for acquisition programs verify that adequate DT&E is planned and resourced to address Cybersecurity
• Confirm DT&E can be executed in a timely manner prior to approval of program Test and Evaluation Master Plans (TEMPs)
DASD(DT&E) will ensure adequate Cybersecurity test planning
Shift Left Nov 2012 Page-10
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
DT&E in the Cyberspace Domain
An Integrated T&E Enterprise Capable of Creating a Realistic Cyberspace Test Environment at All Required
Security Levels
Cyberspace Threat Representations
Systems Under Test
Test Tools
Instrumentation BAF
JPRIMES
ACETEF
CDS
IO Range
SDREN
TSMO
Desired Federated Cyberspace T&E Capability
Process
Methodology Infrastructure
Workforce
Persistent, rapidly composable, secure representation of the Joint Information Environment
Shift Left Nov 2012 Page-11
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
DT&E Cybersecurity Process Summary
Step 1 Cybersecurity Test
Requirements Evaluation
Focus on initiating an approach to Cybersecurity DT&E at Milestone A or B, with update at Milestone C.
Step 4 Cybersecurity Test in
Realistic Cyber Environment
Focus is on Cybersecurity readiness in an operational mission environment to understand capabilities and limitations of the SUT and interconnections against a cyber threat using Red Team testing.
Step 3 Cyber Kill Chain
Evaluation
Focus is assessment of Cybersecurity of the system under test, in a realistic mission and cyber environment, using exploitation testing techniques, post-CDR.
Step 2 Cybersecurity
System Integration Evaluation
Focus is assessment of Cybersecurity in component and system integration vulnerability testing, between MS B and C.
Shift Left Nov 2012 Page-12
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Cybersecurity Testing in the Acquisition Lifecycle
AS TDS
JCIDS Process
Full Rate Production
Decision Review
CBA Joint
Concepts (COCOMs)
MS C MS B
Strategic Guidance (OSD/JCS)
MS A
ICD Technology Development
CDD Engineering & Manufacturing Development
Production and Deployment O&S MDD
Materiel Solution Analysis
AoA
CPD
TEMP
SEP
SRR SFR PDR CDR TRR SVR
*TEMP
*SEP
ASR
PPP
TRA *PPP STAR OTR
TEMP
IOT&E
* STAR * STAR
* SRD
AOTR
Cyber Test Step 1
Cyber Test Step 1 Step 2
Cyber Test Step 1 Step 2 Step 3
Cyber Test Step 1 Step 2 Step 3 Step 4
Reduce the Cyber Attack Surface
Shift Left Nov 2012 Page-13
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Conclusion
• DT&E in mission context
• Improve Interoperability
• Improve Cybersecurity
• Reduce discovery in IOT&E
• Improve Acquisition Outcomes
To ensure rapid fielding of enhanced capabilities to the Warfighter …
Shift Left!
Shift Left Nov 2012 Page-14
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Questions?
Shift Left Nov 2012 Page-15
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
T&E Plan – Test – Report cycle can exceed six months!
•Multiple Test Orgs – DT, OT, Iop, IA
•Multiple Decision Makers – MDA, CIO, DAA
Pilot Record OTRR
60 days
OTRR Full Deployment Decision Review
60 days
Eval Report
DIACAP
Interop Testing
OT&E
Operational Test Plan
Test Concept Brief
60 days
Test Plan Approved
User Training Support Implemented
Interop Cert
IAC&A
Tester Training DT&E
14 days
DoD Test, Evaluation, & Certification
AOTR