4006 1002 0658.ppt [Read-Only] · 2017. 5. 19. · • Problem Reporting, Analysis, and Tracking...

Advanced Systems & Supportability Engineering Technology and ToolsSSETT

Systems Engineers provide a Key Contribution and Role in System Integration and Test

National Defense Industrial Association (NDIA)9th Annual Systems Engineering Conference October 23-26/2006

Test & Evaluation Track, Tuesday September 24

William Lyders, ASSETT Inc

24 October 2006 Slide 2

Advanced Systems & Supportability Engineering Technology and ToolsSSETT Agenda

The Systems Engineering Process Blends with the Test & Evaluation Process over the System Life Cycle

System Integration Planning and Execution requires both SE and T&E Collaboration

The Use of COTS challenges SE and T&E and requires planning for Technology Refresh

Field and Production Testing Offer Different Challenges

T&E Lessons Learned at ASSETT

Summary and Conclusions

Q&A


Advanced Systems & Supportability Engineering Technology and ToolsSSETT The SE and T&E Message

Successful Test & Evaluation (T&E) starts at the beginning of the Systems Engineering Process with the SE discipline generating Use Cases and Testable requirements …

…it gains momentum with the collaboration of SE and T&E transitioning into System T&E integration planning…

…and peaks with the T&E discipline during Acceptance and Certification testing…

But it doesn’t end there…

…it continues with the SE discipline as an on-going force as Technical Evaluations and Operational Evaluations of the system continue in the life cycle phase of Production Testing

Determine whether the program will use traditional Requirements Based Contracting or Performance Based Contracting…Use Cases -> Requirements

Acceptance criteria should be defined early in a project…validation of Use cases or Requirements



System Integration Planning and Execution includes many SE and T&E Activities

C u s to m e r G o a ls R e q u ire m e n ts D e f in it io n

/A rc h ite c tu re D e ve lo p m e n t

H /W – S /W D e s ig n

D e ve lo p m e n t T e s tin g

P ro d u c tio n T e s tin g

T im e to M a rk e t

C o s t

U p g ra d e P la n s

B u s in e s s P ro c e s s e s

R e q u ire m e n ts D e c o m p o s itio n

In te g ra tio n S p e c if ic

R e q u ire m e n ts

S y s te m In te g ra tio n P la n

• A p p ro a c h• S R V M• In itia l B a s e lin e fo r:

• In te rfa c e s• L in k s• M e s s a g e s

• T e s t S c e n a rio s• T e s t C a s e s

• D a ta R e q u ire m e n ts

• In te g ra tio n S c h e d u le

• C M• R is k s• R e s o u rc e s

• E q u ip m e n ts• S IM /S T IM

C o n ce p t D e fin it io n

H ig h L e ve l D e s ig n

D e ta ile d D e s ig n

S ys te m In te g ra tio n P la n

R e f in e m e n t• F in a l In te rfa c e

D e fin itio n s• C o m p le te T e s t

S c e n a r io s• C o m p le te T e s t

C a s e s• F in a l D a ta

R e q u ire m e n ts a n d M ig ra tio n

• C o m p le te In te g ra tio n S c h e d u le

• R e s o u rc e s In -P la c e

• In it ia te T e s tin g• U n it T e s t• S e g m e n t T e s t• E 2 E T e s tin g• C u s to m e r

A c c e p ta n c e• P ro b le m R e p o rtin g ,

A n a lys is , a n d T ra ck in g

• C M o f th e T e s t E n v iro n m e n t

• M T B F V a lid a tio n Id e n tif ic a tio n o f H ig h F a ilu re R a te C o m p o n e n ts

• T ra in in g V a lid a t io n• M T T R V a lid a tio n• P ro b le m R e s o lu tio n• R e te s t

U s e C a s e s

• T a ilo r T e s t S c e n a r io s a n d T e s t C a s e s

• C re a te T e s t E n v iro n m e n t fo r F ie ld Id e n tif ie d P ro b le m s

• C M fo r u p d a te s a n d n e w re le a s e s

R e q u ire m e n tsA ss um p tio n s




C u s to m e r G o a ls R e q u ire m e n ts D e f in it io n

/A rc h ite c tu re D e ve lo p m e n t

H /W – S /W D e s ig n

D e ve lo p m e n t T e s tin g

P ro d u c tio n T e s tin g

T im e to M a rk e t

C o s t

U p g ra d e P la n s

B u s in e s s P ro c e s s e s

R e q u ire m e n ts D e c o m p o s itio n

In te g ra tio n S p e c if ic

R e q u ire m e n ts

S y s te m In te g ra tio n P la n

• A p p ro a c h• S R V M• In itia l B a s e lin e fo r:

• In te rfa c e s• L in k s• M e s s a g e s

• T e s t S c e n a rio s• T e s t C a s e s

• D a ta R e q u ire m e n ts

• In te g ra tio n S c h e d u le

• C M• R is k s• R e s o u rc e s

• E q u ip m e n ts• S IM /S T IM

C o n ce p t D e fin it io n

H ig h L e ve l D e s ig n

D e ta ile d D e s ig n

S ys te m In te g ra tio n P la n

R e f in e m e n t• F in a l In te rfa c e

D e fin itio n s• C o m p le te T e s t

S c e n a r io s• C o m p le te T e s t

C a s e s• F in a l D a ta

R e q u ire m e n ts a n d M ig ra tio n

• C o m p le te In te g ra tio n S c h e d u le

• R e s o u rc e s In -P la c e

• In it ia te T e s tin g• U n it T e s t• S e g m e n t T e s t• E 2 E T e s tin g• C u s to m e r

A c c e p ta n c e• P ro b le m R e p o rtin g ,

A n a lys is , a n d T ra ck in g

• C M o f th e T e s t E n v iro n m e n t

• M T B F V a lid a tio n Id e n tif ic a tio n o f H ig h F a ilu re R a te C o m p o n e n ts

• T ra in in g V a lid a t io n• M T T R V a lid a tio n• P ro b le m R e s o lu tio n• R e te s t

U s e C a s e s

• T a ilo r T e s t S c e n a r io s a n d T e s t C a s e s

• C re a te T e s t E n v iro n m e n t fo r F ie ld Id e n tif ie d P ro b le m s

• C M fo r u p d a te s a n d n e w re le a s e s





Information for System Integration and Planning is developed in each of the SE phases

SE works closely in teams with the customer to get clear measurable requirements.

SE Tools, e.g. Rationale Tool Suite, can be used to automate both Use Cases & Requirements



SE Vee Process blends into the T&E Integration (Verification and Validation)

Business Objectives /

Requirements

Business Requirements Review

System Requirements /

Design

System Requirements Review

Component Requirements

& Design

Preliminary Design Review / Critical Design Review

Product Development

Component Test- Functional- Non-Functional

System Integration Test - Functional - Non-Functional

System Acceptance /

Validation

System Verification

Stakeholder Requirements &

Needs

Stakeholder Requirements &

NeedsDeployed Capabilities

Test Readiness Review

Production Readiness Review

System Validation

Component Verification

Satisfaction of Customer Needs (Use Cases)

System Analysis,

decomposition, and design synthesis

Component verification, system integration, system

verification, validation & acceptance

T&E:

Inte

grat

ion

& Ve

rifica

tion

SE: Use Cases and Testable Requirements

SE: Production Testing/Feedback


Advanced Systems & Supportability Engineering Technology and ToolsSSETT SE Process – Baselines & Reviews

Business Requirements

Baseline

Preliminary Design

Baseline

Critical DesignBaseline

TestReadiness Baseline

Production Readiness Baseline

Systems Engineering Technical Management

Solution Outline Macro Design Micro Design

Execute and Control/Solution

CloseDeployBuildSolution

Design

System Requirements

Baseline

BRR SRR PDR CDR TRR PRR

Systems Engineering is: A rigorous discipline for successful system development and integration. It elaborates business needs derived from Use Cases into traceable and testable requirements, establishes system baselines, and integrates and delivers the full system solution.

Many an SE will move into T&E roles for continuity of understanding the system and understanding the customer requirements.



SE and T&E Products Build on Each Other

Need / Opportunity Identification

New Production

SystemDevelopment

ConceptualSystem

Specification

BusinessRequirementsSpecification

System Requirements Specification

System Architecture

Data Architecture


System Architecture

Data Architecture

Test & Production

System Update


Component Architecture

Data Migration

Test Strategies

Test Architecture



Data Migration

Test Strategies

Test Architecture

Component Level Design

Component Test Plan


Component Test Plan

Test Plans

Test Cases

Test Data

Test Plans

Test Cases

Test Data

Release Content

Release Content

Move to Production

Plan

Move to Production

Plan

Deployment & Migration

Plans


Plans

Customer Baseline

System Baseline

Component / Application

Baseline

Design Baseline Test Baseline Production

Baseline


Review(BRR)

Preliminary Design Review(PDR)

Critical Design Review(CDR)

Test Readiness

Review(TRR)

System Requirements

Review (SRR)

Production Readiness

Review(PRR)

Design Need /

Opportunity Identification

New Production

SystemDevelopment

ConceptualSystem

Specification

BusinessRequirementsSpecification


System Architecture

Data Architecture


System Architecture

Data Architecture

Test & Production

System Update



Data Migration

Test Strategies

Test Architecture



Data Migration

TestStrategies

Test Architecture


Component Test Plan


Component Test Plan

Test Plans

Test Cases

Test Data

Test Plans

Test Cases

Test Data

Release Content

Release Content

Move to Production

Plan

Move to Production

Plan


Plans


Plans

Customer Baseline

System Baseline

Component / Application

Baseline

Design Baseline Test Baseline Production

Baseline


Review(BRR)



Test Readiness

Review(TRR)

System Requirements

Review (SRR)


Review(PRR)


Review(BRR)



Test Readiness

Review(TRR)

System Requirements

Review (SRR)


Review(PRR)

Design

Requirements Traceability Verification Matrix

Test Traceability Matrix

SET&E

Collaborative products


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Responsibilities of SE and T&E to each Other

Responsibilities of Systems Engineering regarding T&ERecommend best practices in system engineering, assist in system & test architecture, design, and developmentMonitor technical baseline along the project life cycleWork with Test team to support test case definitions, test planning and coordination, end to end testingWork with Test team for system, performance, and user acceptancetesting activities. Coordinate defect resolution with Chief Architect, Delivery Manager, and Test Manager

Responsibilities of Test & Evaluation regarding SEInfluence prioritization and testability of requirementsVerify requirements implementation at many levelsIndependent review of risks and input of T&E issuesCollaborate on Integration planning Execute tests – feedback to systems engineeringProvides an independent verification and validation of systemSupports SE with recommendations for next acquisition phase

This makes the SE a better designer as they know which requirements and capabilities really do work or how hard it was to get them to work – as they return to proposal & system definition phases on new systems.


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Levels of Systems Integration Planning

Levels of PlanningPlatformSystemSubsystemUnit/ProgramConfiguration ItemComponent

The System Integration Planning Must Address All Levels of Test and Leverage the Coverage at Each Level to Eliminate Duplication

Results in Multiple Test Plans

Planning for Regression Testing (Automatic & Manual)

Integration Planning Platform

System 1

System N

Subsystem 1

Subsystem N

Unit 1

Unit 2

Unit N

HW CI 1

CSCI 1

CSCI N

Component 1

Component N

Component 1

Component NNCO => System of Systems

System Test StrategyAcceptance Criteria for each system is identified. Test Strategy for each system is established.



System Integration Planning – Key: Master Test Plan

Test PhasesModelingPrototypingCSCI TestingConfiguration Item Testing

Software Integration and Test (SWIT)Hardware Integration and Test

Unit TestSubsystem TestSystem Integration and Test (SIT)System Design Certification Testing (SDCT)User Acceptance Testing (UAT)

Test Planning and Execution Is Initiated in the Design Cycle to Reduce Risk and Validate Requirements Early

Modeling CSCI & CIT

Testing

Unit & Subsystem

TestingPrototyping

SITSDCTUAT

Concept Design Development Testing ProductionLife Cycle Support

SRR PDR CDR TRR PRRBRR/IBR

Systems Engineer provides traceability to Test Manager.

Test Manager completes Verification Matrix.

(Key Outputs of Systems Engineering)• Requirements Traceability and Verification Matrix (RTVM)• Test Plan, Test Cases, Test Scripts in Micro Design, • Weekly Defect Report


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Requirements Verification Matrix

Requirements Verification Matrix

Project: Enter name of project here

Customer Business RequirementsSystem Requirements


CI LevelUnit

LevelSystem Level SDCT

Customer Acceptance Production

R1 Customer business requirement with multiple system requirementsR1 S1.1 System requirement with multiple component requirementsR1 S1.1 C1.1.1 Component requirementR1 S1.1 C1.1.2 Component requirementR1 S1.2 System requirement with multiple component requirementsR1 S1.2 C1.2.1 Component requirementR1 S1.2 C1.2.2 Component requirementR1 S1.2 C1.2.3 Component requirementR2 Customer business requirement with multiple system requirementsR2 S2.1 System requirement with one component requirementR2 S2.1 C2.1.1 Component requirementR2 S2.2 System requirement with one component requirementR2 S2.2 C2.2.1 Component requirementR3 Customer business requirement with one system requirementR3 S3.1 System requirement with no component requirementsR4 Customer business requirement with one system requirementR4 S4.1 System requirement with no component requirements

Requirements

T = TestD = DemonstrateA = AnalyzeI = Inspect

T

TT

A

T D D

AA

I

I

T

T

T

T

T

Complete Traceability MatrixEnumerated System Requirements trace back to enumerated Business Requirements.

Complete Verification MatrixVerification Strategy is determined for each System Requirement.

Systems Engineering Responsibility T&E Responsibility (SE Collaboration)



Up to this point we have been discussing the Traditional Approach for Development….

What do you believe would happen if commercial products were incorporated in your system?.....

We will now briefly discuss the impacts to Systems engineering and T&E efforts

Finally, during the Test Execution Phase, the Systems Engineer supports the T&E team with FFR - problem find (identification), problem fix, and problem resolution



The Use of COTS Impacts the SE and T&E Planning Process

AdvantagesTime to Market

Vendor TestingProduct AvailabilityReduced RiskLevel of Integration

DisadvantagesSystem Compliance TestingConfiguration Control

Short Product Life CycleUpgradesStandards Compliance

Built In Test (BIT) Test Coverage

Systems Engineering and T&E have different challenges with COTS –some are advantages and some disadvantages

Products come quicker … but more test cycles (regression testing) may be required as product upgrades occur


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Impact of COTS on T&E Planning - Requirements

Successful Test and Integration of COTS Technologies Starts with the Concept Phase – SE: new challenges defining Requirements

Physical – Write Requirements that Will Allow COTS to Meet the Environmental Conditions

Adapt the System to COTS not COTS to the System

Functional – Perform Functional Decomposition and Generate Derived Requirements to Align Required Capabilities with the Capabilities of Widely Available COTS Products

Target COTS Products/Capabilities During the Functional Decomposition Process

Operational – Leverage the Widely Used Commercial Graphical User Interface to Simplify/Reduce Training Requirements

e.g. Interactive Maintenance

Make Standards a Part of the System RequirementsRequire Compliance





Impact of COTS on T&E Planning-Advantage: Vendor Testing

Vendor TestingVendor Supplied Products Are Tested through Configuration Item Testing Prior to Delivery

Eliminates the Need for the System Integrator to Establish the Test Environment and Test Process for these Levels of Testing

Systems Integrator Must Validate Vendor Test ProcessReview of Vendor Test ProceduresWitness Testing at the Vendors SiteIncoming Inspection and Validation


Configuration Item Integration SystemModule

TestingYour Testing reduced



Impact of COTS on T&E Planning –Advantage Product Availability

Product AvailabilityCOTS Products Available 12-21 Months Before a Product Can Be Designed, Built, Tested and Ready for Production

COTS Products Generally Available within 90 Days of Placing the OrderDesign and Build of a Product (Hardware or Software) Requires Approximately 24 Months

Testing Can Begin When Products Are AvailableFunctional Definition and therefore the Development of Application Code Must Support the Delivery of COTS Products into Test

2004 2005 2006 2007 20081 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

Design and Build ProductDesignDiagnosticsBuildTestProcure

Buy Product

Task2004 2005 2006 2007 2008

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

Design and Build ProductDesignDiagnosticsBuildTestProcure

Buy Product

Task

COTS Products Are Available in Quantity within 90 Days after

Placing the Order

Products available earlier – testing earlier



Impact of COTS on T&E Planning –Disadvantage: Shorter Product Life Cycle

System Life Cycles Are Significantly Longer than the Life Cycle of Electronics

Driven By the Rapid Evolution of Technologies

Open Standards Are the Key to Ensuring a “Plug and Play” EnvironmentEstablish a Plan for Upgrading Technology Over the System Life Cycle

Frequency of RefreshmentDuring the Development CycleDuring Production and Life Cycle Support

Ensure System Partitioning Facilitates Technology RefreshmentProduct upgrades will necessitate regression testing

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Months

General Purpose

Processor

Signal ProcessorStandards

Product Life Cycles

Products need to be replaced earlier – more testing


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Technology Refresh Objectives

Obsolescence AvoidanceReplace Obsolete Hardware and Software System Elements With Currently Supported Vendor Products

In Some Cases Will Require Limited Redesign

Avoid Product End-of-life Issues

Maintain Infrastructure CurrencyEnsure Compliance With Current Industry State of the PracticeGradual Migration With Industry Trends and Mainstream Standards and Products

Reduce System Total Life Cycle CostAvoid Large Investments in Spare Parts InventoriesLeverage Emerging Technologies and Products to Lower Costs

Support Cost Effective Technology InsertionAnticipate Requirements Pull and Technology PushPlan TR and TI Evolution Together

Upgrading Technology: SE Oversees


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Technology Refresh – Strategy & Planning

Technology Refreshment Should be an Integral Part of a COTS-based Systems Design Strategy

Exploits Exponentially-improving TechnologyFacilitates Introduction of Advanced System CapabilitiesAllows for Cost-Effective Post-Deployment SupportReduces Total Life-Cycle Cost

Effective Planning for Technology Refreshment Based onImplementation-independent DesignComprehensive Technology StrategyUpdated Fleet Support Strategy

Cost

Time ->

->

New Technology Obsolete

Stay here with Technology Refresh

State of the Art



Research & Production Testing –Different Challenges

Full Production TestingFull SE processAll levels of T&E testingFormal documentationProduction Quality system and products for production platforms

Research Testing (e.g. SBIR & ABP programs)

Streamlined SE processMix of experimental technology and/or commercial test equipment integrationHigh level design and test documentationBreadboard and prototype system and productsUsually involves Field testing of prototype to collect data for later production design


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Some T&E Lessons Learned at ASSETT

Get testable requirements and acceptance criteria defined early and agreed upon with the CustomerCreate a Test Strategy and Master Test Plan

It should be a much shorter but do so for research also

Get buy in by all parties involved

Prepare Test Plans for each of the different levels of integration and conduct peer reviews and customer reviews as necessary – don’t want surprises at acceptanceCreate a SRVM and get it reviewed/approved Design as many auto regression tests as possible to simplify testing for upgrades and product release certificationsFully dry run all test procedures – do exactly as plan to do in formal testing – whether it be research or productionHold regular test progress/results meetings internallyDocument all test findings and share them with both Customer and own teams


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Summary & Conclusions

Concept: Solution FeasibilityInitial Requirements Defined

Design: Functional/Physical/Data ArchitecturesRequirements Finalized

Development: Detailed Design/ImplementationTesting: Verification and Validation

Validate Implementation Against Requirements

Production: Manufacturing/AssemblyBuild to PrintTest

Life Cycle Support: Deployment to Retirement



Systems Engineering provides a structured approach to managing the technical solution over the full life cycle from concept to deployment to retirement…

…Test and Evaluation complements this approach with support for defining requirements and integration planning…and conducting many levels of integration tests with systems engineering support to achieve customer acceptance of a system…

…later production testing on platforms provides feedback for new requirements.


Advanced Systems & Supportability Engineering Technology and ToolsSSETT Q&A

4006 1002 0658.ppt [Read-Only] · 2017. 5. 19. · • Problem Reporting, Analysis, and Tracking...

Documents

Transcript of 4006 1002 0658.ppt [Read-Only] · 2017. 5. 19. · • Problem Reporting, Analysis, and Tracking...