Test & Evaluation Andy Lee Division Director

Airport Consultants Council July 21, 2016

Innovating the Future of Aviation Security

Standards & Security | Updates to Detection Standards

Passenger Screening | Life Cycle Cost Estimates Revision Updates; Passenger Screening Experience

Test & Evaluation | New Qualification Process; Third Party Testing Update Deployment & Logistics |

Planning Guidelines and Design; Checkpoint Design Guide

Checked Baggage | Recap and Acquisition Plans

Cybersecurity | Cybersecurity Requirements; Technical Solutions

Innovation | Operational Improvements; New Technologies; Innovation Task Force

System Architecture | Update on Implementation of OSC System Architecture; Priorities for Fiscal Year 17; Transportation Security Equipment (TSE) Connectivity

1 Enhancing Core Mission Delivery by Focusing on System of Systems

2 Integrating Principles of Risk-Based Security in Capabilities, Processes, and Technologies

4 Increasing Transparency in Engagement with Stakeholders to Enable Innovation

3 Streamlining Acquisitions, Requirements, and Test and Evaluation Processes

Alignment to Strategic Five-Year Technology Investment Plan Themes:

The Office of Security Capabilities (OSC)

safeguards our Nation’s transportation systems

through the qualification and delivery

of innovative security capabilities and solutions.

Innovation

System Architecture Standards

& Security

Test & Evaluation

Deployment and Logistics

Checked Baggage

Passenger Screening

Cybersecurity

3 4

2 3 2 3

1 2 1 2 3 4

3 4

3 4 1 2 3

1

1 2

2

2

System Qualification Process Enhancement

Current Checked

Baggage Process

Enhanced Process

1. Announce a Qualified Product

List (QPL)

2. Release Request For

Qualification Data Package (QDP)

3. Certification Data Package (CDP)

Submission

4. DT 1: TSL

Certification testing

5. QDP Submission

6. Qualification Testing (TSIF)

7. Conduct OT&E 8. Add Technology to QPL

1. Announce a

QPL

2. Release Request For Qualification Verification

Package (QVP)

3. QVP Submission

4. DT 1: System/Subsystem Testing

5. QDP Submission

6. DT 2: Integration and Implementation Testing

7. Conduct Initial OT&E

8. Add Technology

to QPL 4A. TSL Cert Testing

4B. System Test OEM

6A. DT Qual Test (TSIF)

6B. DT Field Test Op Site

Certification Full Rate Decision

Qualification

Certification Full Rate Decision

Qualification

Establishes early involvement with industry that facilitates a more mature system design to meet TSA needs

Alignment of all TSA testing (TSA Systems Integration Facility (TSIF), TSL, OT) under a single unified structure to ensure consistent messaging to vendors

Benefits of QPL Enhancement Institutes a more collaborative system design and testing approach with industry

Generates a consistent Acquisition approach through requirements management, ensuring compliance with DHS AD-102 instruction, and implementing standardized industry engagement across TSA programs

To streamline acquisitions, requirements, and Test and Evaluation Processes, the Transportation Security Administration (TSA) is enhancing the system qualification process by increasing interaction with the Original Equipment Manufacturer (OEM) through earlier involvement in the acquisition process. This increased interaction will encourage more mature TSE through greater and more transparent engagement with the OEMs on system architecture and testing. Below is the enhanced qualification process and key highlights:

• Unified QPL Process across programs through an integrated continuum to support a more expedient performance assessment process from entry to deployment • Each test phase provides a specific insight that when combined defines the system capabilities, limitations, mission performance and TSA utility

• Developmental Testing (DT): Verifies status of technical progress, that design risks are minimized, substantiates achievement of technical performance, and certifies readiness for Initial Operational Test and Evaluation (IOT&E)

• Operational Testing (OT): Validates that system and operations risks are minimized, and certifies system is effective and suitable in its operational environment and intended uses

• All pieces combined demonstrate the end-to-end system and system of systems capabilities to meet today’s and tomorrow’s threats; ensure travel and Transportation Security Officer (TSO) safety before the procurement decision is made

3

Qualification Process Overview

The following is a high-level overview of the enhanced qualification process steps:

4

STEP 1 – Announce QPL Functional Category (FC)/Track/Window

The Department of Homeland Security (DHS) will announce various Trace Detection (TD) requirements by utilizing Functional Categories, Tracks, and Windows. Windows will be opened under a Functional

Category/Track to request QVP submissions specifically through a Request for QVP (RFQVP). All info posted as an amendment to the FedBizOpps (FBO) posting.

QPL Structure

Track 2

Qualified Products List (QPL)

TSA Desktop ETD

TSA Portable ETD

HID-ETD

Track 3

Track 1

Track 2

Track 3

Track 1

Track 1

Track 2

Track 3

Functional Categories:

Functional Categories Tracks: Window

Window

Window

Window

Window

Window

Window

Window

Window

5

STEP 2 – Release Qualification Management Plan (QMP) & Requirement Documents

Documents above will be either Sensitive Security Information (SSI), For Official Use Only (FOUO), and/or Classified. To release SSI and FOUO, TSA will complete vetting for each individual anticipated to access the information. For classified information, individuals must possess PCL from DSS to view in a secured facility

and must possess DD Form 254 and FCL to store.

Qualification Management Plan

Instructions for QVP and QDP Submission

QVP and QDP Evaluation Process Overview

Overview for the Qualification Process

Applicable Requirements Documents

Functional Requirements Document (FRD)

Functional Requirements Matrix (FRM)

Detection Standard (DS)

The RFQVP will be posted on FBO and will provide all info vendors will need to submit a QVP that will detail the testing qualification process for that Functional Category. The RFQVP will also provide instructions to

request and access applicable documents that cannot be released on FBO due to sensitivity of the information.

Request For Qualification Verification Package (RFQVP)

6

Overview of Qualification Process

There are three levels of qualification that can be required for a Functional Category Level 1 – Certification; Level 2 – Integration; and Level 3 – Implementation. The Government will determine which one of the levels will be required for a FC.

There will be restrictions to the number of testing rounds for each step.

Levels of Qualification

3. QVP Submission

4. DT-1: System/Subsystem Testing

4a. DT-1E1Cert Test

TSL

4b. DT-1E2System Test

OEM

5. Qualification Data Package

(QDP) Submission

6. DT-2: Integration/Implementation Testing

6a. DT-2E1: Qual Test

TSIF

6b. DT-2E2Field Test

Op Site

7. Conduct Initial Operational Test and Evaluation (IOT&E)

8. Add Technology to

QPL

Certification Full Rate Decision Qualification

321

Legend:

Government Action

OEM Action

Decision Point

Qualification Level

7

STEP 3 – QVP Submission

The initial vendor submission into the qualification process and establishes the system baseline for qualification testing.

QVP

QVP Submission

The Vendor shall provide in the QVP • Certification Data Package (CDP) for the

detection chain • Vendor System Design, Architecture, and

Performance Requirements • Life Cycle Support Concept • Major Configuration Item List (MCIL) • Vendor Test Pans and Procedures traced to the

Verification Requirements Tractability Matrix Government acceptance of the QVP

• Establishes the system baseline that will enter the Certification Process

• Approve the test plans and procedures to be used to assess system/subsystem performance

QVPs can be submitted any time during an open Window period.

QVPs will be evaluated as noted within the RFQVP. Anticipated to take four (4) weeks. .

Vendors have the ability to make corrections or additions to QVP up until the closure of the Window.

After Window closes, vendors will be removed from the process and will have to wait until the next Window is opened.

8

STEP 4 – Developmental Test & Evaluation Phase 1 (DT-1)

Step 4A – DT-1 Event 1 – Certification Testing

Government Test and Evaluation (T&E) of explosives and narcotics detection equipment

Includes Probability of Detection (Pd) and Probability of False Alarm (Pfa) in comparison to the DS

Will be conducted at the Transportation Security Laboratory. Vendors should anticipate providing approximately four (~4) systems for testing.

Step 4B – DT-1 Event 2 – System/Subsystem Testing

Testing conducted to verify that the system meets all technical requirements.

Will determine the maturity of the system design and will be used to determine system readiness to proceed into Qualification Testing.

Testing will be conducted at the vendor facilities and/or a third party test Organization facility at no cost to the Government.

DT-1 is the type of testing which verifies that the System Under Test (SUP) meets all technical requirements. DT-1 may be comprised of the two following test events and if both events are required are intended to be

conducted in parallel.

DT-1: System/Subsystem Testing

9

STEP 5 – QDP Submission

The second vendor submission into the qualification process and establishes the system baseline for DT-2 based on the results and finding from Step 4.

QDP

QDPs will be evaluated as noted within the RFQVP and QMP. Anticipated to take four weeks. .

Vendors will have the ability to make corrections or additions, but the Government may limit the number of submissions.

QDP Submission

The Vendor shall provide in the QDP • Updated System Design, Architecture, and

Performance Requirements • Updated Life Cycle Support Concept • Updated MCIL • Test Reports from DT-1E2 Government acceptance of the QDP • Establishes the system baseline that will enter

DT-2 at TSIF • System approved for entry into independent Gov

T&E Baseline shall be locked for the remainder of the Qualification Process

10

STEP 6 – Developmental Test & Evaluation Phase 2 (DT-2)

Goals of DT-2: 1) ensure the system is ready to proceed into the field for live operations, 2) assess and reduce the risk of an unsuccessful OT&E, and 3) confirm that the system will not pose any operations or security risks.

DT-2 may be comprised of the two following test events and, if both events are required, are intended to be conducted in series.

DT-2: Integration/Implementation Testing

Step 6A – DT-2 Event 1 – Qualification Testing (QT)

Confirms the integrity of the system design over the operational and environmental range in the specification using production hardware fabricated to the proposed production design specifications and drawings

Conducted at the unit, subsystem, and system level on production items and is completed before the production decision.

Will be conducted at TSA’s TSIF.

Step 6B – DT-2 Event 2 – Field Testing

Conducted in the operationally relevant environment prior to Operational T&E (OT&E) and is used to ensure the SUT is installed, integrated and ready to proceed to OT&E.

Allows for one last Developmental Test checkpoint prior to OT&E to identify performance risk and inform the Operational Test Readiness Review.

Will be conducted at the operation field site identified by the Government.

11

STEP 7 – Operational Test & Evaluation (OT&E)

Initial Operational Test and Evaluation (IOT&E)

Evaluation of production representative test articles and uses typical operational scenarios that are as realistic as possible to ensure the system meets the operational need defined in the Operational Requirements Documents.

Will be conducted in the Components’ operational setting. Vendors should anticipate having to provide four to five systems for IOT&E.

OT&E is the type of testing which validates that the SUT meets all operational requirements and intended uses. OT&E is conducted using properly trained user operators in an uncontrolled operational environment.

OT&E

12

STEP 8 – Add Technology to QPL & Competitions

Once a system passes testing as required (Level 1, Level 2, or Level 3) and the system receives a favorable production decision, as defined in the DHS Instruction 102-01-001 Rev 01 Acquisition Management Instruction,

dated March 9, 2016, the vendor’s systems will be added to the TD QPL.

QPL

Example of QPL No contract will be awarded for the qualification of a system.

Once a need is identified, a Contracting Officer from DHS Component may solicit, compete, and award their own contracts.

Solicitations may include all products/services that support the deployment of a TD system.

The Government is not required to delay a proposed award to allow a potential vendor to enter or complete the qualification process.

Functional Category 1 - HID-ETD system • Track 1: DHS HID-ETD FRM v1.2; DHS Trace

Illicit Drugs Detection Performance Requirements v1.1; Qualification Level 1

Functional Category 2 – TSA Desktop ETD • Track 1: TSA FRD vX.X; TSA DS vX.X;

Qualification Level 3

Vendor X Model Y100 Software vZ.1

Vendor A Inc. Model B3000 Software vC.1

Vendor X Model Y800 Software vZ.2

Vendor SB Inc. Model P8000 Software vC.4

13

Qualification Requirements

Deliverables

SSI Training Completion Deliverables

SSI Compliance Deliverable

MCIL

Configuration Change Management

Once a system passes testing as required (Level 1, Level 2, or Level 3) and the system receives a favorable production decision, as defined in the DHS Instruction 102-01-001 Rev 01 Acquisition Management Instruction,

dated March 9, 2016, the vendor’s systems will be added to the TD QPL.

QPL

Cost/Savings Deliverable

During Testing • Vendor shall follow instructions in QMP

After Qualification • Vendor shall follow process defined in the CMP • Will be provided upon being added to the QPL

After Contract Award/Deployment • Vendor shall follow DHS Component definition

14

Questions?

Questions?