1Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Challenges in the Verification ofPre-Existing Aerospace Systems

Jean-Baptiste Jeannin

CPS V&V I&F workshop, December 11th, 2014

2Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Verifying Pre-Existing Systems

!

Verified idealized system

System that actually runs on the airplane

3Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Next-Generation Airborne Collision Avoidance System (ACAS X)

• Industrial system developed by the FAA replacing TCAS• Designed to prevent collisions between aircraft• Based on optimizing a Markov Decision Process to

create a big table (several millions of entries) that is then interpolated to make decisions at runtime

4Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

COCDNC

DND

DES1500

CL1500

COC

Next-Generation Airborne Collision Avoidance System (ACAS X)

• Only vertical advisoriesare allowed

• Separation propertybased on a puck

• Table in 7 dimensionswith millions of entries

How do we verify such a huge table?

5Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

COCDNC

DND

DES1500

CL1500

COC

ACAS X Verification with KeYmaera

① For each action, identify a region where it is safe

② Formally prove in KeYmaera that the safe regions are correct

③ Compare the safe regions with the ACAS X decision table

safe CL1500

CL1500

6Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Computing the Safe Region: for a Climbing RA

1

2

3

4

parabola at acceleration

straight up at target vertical velocity

half parabola

half parabola

horizontal of width

straight up at target vertical velocity

7Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Comparison: ACAS X issues CL1500

Initial advisory begins to induce NMAC

8Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Challenge: Verifying Pre-Existing Systems

• Solution 1: Verify the system directlyProblem: its design is often ill-suited for verification

• Solution 2: Show that the system is subsumed by a more general, verified systemProblem: we need to identify this more general system

• Solution 3: …

9Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Challenge: Modeling Uncertainties

• Uncertainty due to uncertain parameters or unpredictable events: wind, component faults…

• Sensor uncertainty: sensors are never perfect, they only give values within a certain margin of error

10Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Challenge: Human in the Loop

• Airplanes have pilots who follow precise procedures:in theory their behavior is easy to model

• However it is difficult to quantify the behavior of a human (reaction times, minimum performance,…)

• What about modeling reaction to unusual or stressful events?

11Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Challenge: Numerical issues

• A computer cannot effectively perform real number computations

• Instead, computers use floats

• How do we transfer a proof using exact-precision real numbers to a system using limited-precision floats?

12Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Challenge: Scalability and Automation

• Aerospace systems are big systems

• Natural approach is to verify a simplified system

• How do we make sure a proof on a simplified system still applies to the complete system?

• At some point, systems are too big and intractable for manual proofs: need proof automation

13Challenges in the Verification of Pre-Existing Aerospace SystemsJean-Baptiste Jeannin

Challenges and Conclusion

• To bridge the gap between verified systems and implemented systems, we need to be able to:– Verify complex systems– Verify pre-existing systems

• To make our proofs more applicable, we need to take into account:– Uncertainties of parameters and sensors– Humans in the Loop– Numerical Issues– Scalability and Automation