1 © Nokia 2016

Using Safety-Critical Concepts in Privacy Engineering

Public

Dr. Ian Oliver

Bell Labs, Finland

2 November 2016

A Lecture Given to CRiM’16, Oulu, Finland

2 © Nokia 2016

Public

Auditing mobile services and associated infrastructure from an engineering perspective...

3 © Nokia 2016

Public

Auditing mobile services and associated infrastructure from an engineering perspective...

PrivacyOfficers & Lawyers

Privacy Engineers

4 © Nokia 2016

Public

The problem was (and still is, and will be) the lack of good (any?) techniques for reasoning about privacy in an engineering capacity

5 © Nokia 2016

So what is this privacy thing anyway?

Public

• vs Security...

• [[privacy]]

• Information wants to be free!• Freedom to do/be...?• Freedom from...?• Price, value

•Ownership

• You are the product

• ”Anti-privacy”

• Advertising, surveillance, hacking, oversharing

• Personal responsibility vs Technilogical complexity

• PII, personal data, pseudo-anonymous, anonymisation

•variability, entropy, Navier-Stokes

• f(p1...pn) -> R

•The Privacy Singularity•Unification of disciplines•Mathematical Foundations of Privacy•The Fundamental Theorem of Privacy

6 © Nokia 2016

Privacy as...

Public

• A legal construct •“The Right to Privacy” (Warren and Brandeis, 1890)• EU Data Protection Laws• Human Rights•...

7 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct

• morals, ethics etc• political science? Kant etc.

8 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct

• brand/shareholder value• customer relatonships• business• innovation

9 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct• A guiding principle

10 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct• A guiding principle• A sociological construct

11 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct• A guiding principle• A sociological construct• A game theoretic construct

12 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct• A guiding principle• A sociological construct• A game theoretic construct• A systems engineering construct

char collectDataFlag = 'Y'; // Future proofed boolean// Y for yes, N for no

void collectDataFunction(){//collect IMEI, IMSI, MSISDN, TimeStamp and location//and send to the hardcoded IP address...

}

void checkDataCollection(){switch(collectDataFlag){

case 'N' :// don't do anything

case 'Y' :// ok to collect everythingcollectDataFunction();

}}

compliance

13 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct• A guiding principle• A sociological construct• A game theoretic construct• A systems engineering construct (pt.2)

• Ontological structures• Metrics / Risk Analysis• Modelling• Privacy Engineering• Compliance• Culture & Safety Critical

Systems• (Aviation, Medicine)

14 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct• A guiding principle• A sociological construct• A game theoretic construct• A systems engineering construct• An optimisation construct

15 © Nokia 2016

Privacy as...

Public

• A legal construct• A philisophical construct• An economic construct• A guiding principle• A sociological construct• A game theoretic construct• A systems engineering construct• An optimisation construct• A mathematical construct

𝑡0

𝑡1

𝐷1 ×⋯ × 𝐷𝑛 < 𝜀𝑈

• metrics• topology• ontology• anonymisation & variability• turbulence, chaos theory• link back to economics & game theory• deanonymisation• information entropy

16 © Nokia 2016

Public

The problem was (and still is, and will be) the lack of good (any?) techniques for reasoning about privacy in an engineering capacity...

...so what did we do?

17 © Nokia 2016

We developed:

• Epics and Use cases for Privacy

• Checklists

• Software Development Process Integration

• Audit Procedures (non-functional aspects)

- privacy

- secuity

- performance

- continuity (resiliance)

and the result was...

18 © Nokia 2016

Failure

19 © Nokia 2016

Why didn’t it work?

• Despite highly trained personel

- Cessna Single Engine Failure

• FLY THE AIRCRAFT

- Air France AF447

• To much adherence to process

- Processes tell everyone the order of what to do

- Difficulty in handling exceptions and experts

- Aviation Checklists are status checks used to assist in due dilligence in preparation for the next and future phases of flight.

- Engineers aren’t stupid

• Checklist replaced responsibility and expertise

- For both the auditor and develoment teams

• Tick-box oriented

- Ask questions, Accept answers, TICK!

- Limited understanding and context of naswers

• Limited time-scale

- One-off review

20 © Nokia 2016

We developed:

• Simpler ”Checklists”

• Training Courses

• Realised that no-one understood each other

• Tried to ban (unsuccessfully) the term ”PII”

• Tried to formulate requirements

• Introduced more risk management ideas, eg: RCA, FMEA

and the result was...

21 © Nokia 2016

Failure2

22 © Nokia 2016

What’s the problem now?

• Communication

• Emphasis on process over method

• Lack of understanding of role

• Lack of legal and engineering techniques

• Lack of integration of legal and engineering

• The privacy organisation itself

• Humans

23 © Nokia 2016

What’s the problem now?

Actually it was much worse:

24 © Nokia 2016

What’s the problem now?

Actually it was much worse:

Total emphasis on ”compliance”

25 © Nokia 2016

What’s the problem now?

Actually it was much worse:

Total emphasis on ”compliance”

Whatever ”compliance” meant...

26 © Nokia 2016

Compliance

is fragile

Public

char collectDataFlag = 'Y'; // Future proofed boolean// Y for yes, N for no

void collectDataFunction(){//collect IMEI, IMSI, MSISDN, TimeStamp and location//and send to the hardcoded IP address...

}

void checkDataCollection(){switch(collectDataFlag){

case 'N' :// don't do anything

case 'Y' :// ok to collect everythingcollectDataFunction();

}}

27 © Nokia 2016

????!!!

28 © Nokia 2016

Help!

Not invented here!

Were there any industries or disciplines from which we could learn?

Or are software engineering and legal ’special’?

29 © Nokia 2016

29

Serendipity

© 2013 HERE | Title | Author | Company confidential

30 © Nokia 2016

Help!

Consider information to be a dangerous item

This has various meanings in aviation, medicine, civil engineering etc.

31 © Nokia 2016

31

A quick introduction to surgical infection control

© 2013 HERE | Title | Author | Company confidential

32 © Nokia 2016

32

A quick introduction to surgical infection control

© 2013 HERE | Title | Author | Company confidential

seriously!

33 © Nokia 2016

33

The Sterile Field

© 2013 HERE | Title | Author | Company confidential

Key:

• Sterile

• Non-sterile

34 © Nokia 2016

34

The Sterile Field

© 2013 HERE | Title | Author | Company confidential

Key:

• Sterile

• Non-sterile

Movement of materials from one area to the other must be controlled to prevent contamination of the sterile field with non-sterile items

Strict protocols prevent contamination

35 © Nokia 2016

Some things...

• Communication

• Culture

36 © Nokia 2016

Some things...

• Communication, Structure and Semantics

• Culture

Already solved...in other fields

37 © Nokia 2016

Standardised Communication

Public

38 © Nokia 2016

Standardised Communication

Probably not personal data/ Probably personal data

39 © Nokia 2016

Standardised Communication

Forget process, just get the information about what’s going on...

40 © Nokia 2016

Nokia Internal

41 © Nokia 2016

Nokia Internal

42 © Nokia 2016

Nokia Internal

43 © Nokia 2016

Public

Checklists

44 © Nokia 2016

Public

Checklists

45 © Nokia 2016

Public

Morbidity and MortalityAccident InvestigationReporting

46 © Nokia 2016

Public

Roles and Role Integration

R&D Team Checklist

(before review)

R&D Team Checklist

(post-review)

Audit Team Checklist(sign-in)

Audit Team Checklist

(time-out)

Audit Team Checklist(sign-out)

Project development & processes (time)

System

under

auditPrivacy

Officer

Legal

Security

Architects

47 © Nokia 2016

Public

Experience

DataCollection

CellID->Location

DataStorage

OperatorPrivacy

Preprocessing

Extraction Hashing

FileStorage

RawData

Processing &Enrichment

External Data

ExternalCross-

referencing

Atomic Data

Aggregation/Report

Generation

CustomerReception

ReportStorage

<<data subject>>Customer

48 © Nokia 2016

Public

Conclusions...

49 © Nokia 2016

Public

No heroes

50 © Nokia 2016

Public

Treat privacy as a safety-critical aspect