V&V of COTS RTOS for Space Flight Projects

The 1st Annual NASA Office of Safety and Mission Assurance (OSMA) Software Assurance

Symposium (SAS)

Michael Rahmatipour MSFC

Contact Information:MSFCAvionics DepartmentFlight Software Group (ED14)256-544-5115Michael.H.Rahmatipour@msfc.nasa.gov

2

• The main goal is to perform V&V of the RTOSs used in critical flight applications in the following steps:

• Evaluate the amount of software testing performed by vendors of CTOS RTOS.

• Develop a set of generic test requirements for embedded flight real-time operating systems.

• Develop a test plan and approach to perform V&V on a candidate operating systems against the above generic requirements.

• Develop test software and test procedures to implement the verification plan.

Goal of V&V of COTS RTOS Center Initiative

3

Purpose of the V&V of COTS RTOS Center Initiative

• Multiple, space-related programs currently use commercial off-the-shelf (COTS) real-time operating systems (RTOSs) as part of the NASA goal to reduce software development time and cost.

• Recent, high-profile NASA mission failures have underscored the need for highly reliable software.

• An internal NASA audit showed a need for verification and validation (V&V)/certification tests of the COTS RTOSs used.

• These operating systems should be tested to the same confidence level as flight critical software applications.

4

Current Embedded OS Usage

EE Times conducted a series of surveys over the years on the operating systems used by its readers.

0% 10% 20% 30% 40% 50% 60%

Proprietary

VxWorks

pSOS

C Executive

QNX

VRTX

LynxOS

OS-9

Nucleus

RTEK

RTXC

iRMX

Windows CE

1999199819971996

Source: EETimes 1996-1999Source: EETimes 1996-1999Embedded Systems StudyEmbedded Systems Study

5

• Completed deliverables– White paper on current practices for vendor V&V of

commercial operating systems.– Software Requirements Specification for a Reusable

Realtime Operating System for Embedded Systems.– Software Requirements Specification for the VxWorks

RTOS for Embedded Systems.– Plan and Approach for V&V of the VxWorks Realtime

Operating System.– Plan for Verification of the VxWorks Realtime

Operating System.

V&V of COTS RTOS Center Initiative Products and Status

6

V&V of COTS RTOS Center Initiative Products and Status

• Products in development– VxWorks RTOS Test procedures and test reports

• MSFC is developing the test software and procedures to verify the Software Requirements Specification for the VxWorks RTOS for Embedded Systems

• There are 157 requirements that are organized in 28 cases.

– Final Software Requirements Specification for the VxWorks RTOS for Embedded Systems

7

V&V of COTS RTOS Center Initiative Test Case Status

Case # Case Title # of Requirements Test Code Test Procedure Completed

1 Serial Communication Driver 20 Completed Completed Yes2 Logging 4 Completed Completed Yes3 Semaphores 14 Task Scheduling 25 Task Services 4 In Work In Work No6 Timers 47 Critical Sections 18 Interrupts 5 Completed Completed Yes9 Memory Pool 310 Shared Memory 111 Message Queues 212 Error Reporting 113 Watchdog Timer 114 Priority Inversion 415 RAM File System 116 Raw File System 117 DOS File System 318 Patching 119 Background Memory Scrubbing/Health Monitoring 120 Clock Configuration 221 Child Task Crashes 122 UDP/IP Sockets 323 TCP/IP Sockets 324 Add/Replace System Services 225 Load/Unload Resource Allocation 7 Completed Completed Yes26 VxWorks ROM Image 127 VxWorks Restart 228 VxWorks User Restart Services 1

VxWorks Version

V&V of VxWorks OS 5.3.1Marshall Center -1 (MC-1) Propulsion System Controller(PSC) 5.3.1Urine Processor Assembly (UPA) 5.4Material Science Research Rack (MSRR) 5.3.1X-37 5.1

Programs Currently Using VxWorks Operating System at MSFC

VxWorks Verification Test Cases

Verification by Software Execution

6000 Radstone

PPC2-603e RadstonePPC2-603e Radstone

Board/Proc. Type Manufacturer

1.1

1.0

BSP Version

1.11.1

8

V&V of COTS RTOS Center Initiative Summary

• There are many good packages available as real-time kernels. • The level of effort required by end users to verify/validate or

certify a RTOS is highly dependent on the processor board, contents of the software executable, development and source code documentation (including requirements), and tools available for verification.

• Any V&V/certification effort will be valid for the tested configuration only. Any update to the processor board, peripherals, RTOS, etc. will necessitate a re-verification.

• An important consideration in choosing a COTS RTOS for a safety critical application is the degree of support the vendor will provide for a verification effort.