Performance DecisionsPerformance Decisions

Civil Engineering Systems Civil Engineering Systems

University of BristolUniversity of Bristol

Recognise these questions?Recognise these questions? Have we really taken into account all the Have we really taken into account all the

factors affecting our decision?factors affecting our decision? Tell me again about the assumptions?Tell me again about the assumptions? Just what are our levels of risk?Just what are our levels of risk? But where did these numbers come from?But where did these numbers come from? Have we got any evidence to prove that?Have we got any evidence to prove that? Where are we vulnerable?Where are we vulnerable? Are we really ready to make that decision Are we really ready to make that decision

yet?yet? What is the most important factor?What is the most important factor?

59%

53%

47%

44%

24%

15%

3%

0% 20% 40% 60% 80% 100%

Percentage of respondents

The quantitative analysis of direct drivers in decisions:

Specialist analysis hard to understand / interrogate / test Not enough analysis Clash of personal / departmental cultures Input numbers are shaky Assumptions not clearly identified / tested Decision factors / influences poorly / incompletely framed Uncertainty, risks and unknowns hidden or forgotten

Yet why?Yet why? Breadth depth & balance of issues not fully Breadth depth & balance of issues not fully

graspedgrasped Undue trust in quantitative analysisUndue trust in quantitative analysis Different influencers see only part of the Different influencers see only part of the

big picturebig picture Delays due to lack of consensusDelays due to lack of consensus Poor knowledge management or corporate Poor knowledge management or corporate

learninglearning Qualitative judgement and quantitative Qualitative judgement and quantitative

analysis done in isolation of each otheranalysis done in isolation of each other Lack of frameworkLack of framework

What is the PeriMeta What is the PeriMeta Approach?Approach?

A means to:A means to: Enhance decision making in the context of Enhance decision making in the context of

incomplete, sparse and conflicting informationincomplete, sparse and conflicting information Communicate complex systems simplyCommunicate complex systems simply Manage uncertainty explicitlyManage uncertainty explicitly Integrate ‘hard’ and ‘soft’ influences Integrate ‘hard’ and ‘soft’ influences

By:By: Modelling systems as hierarchies of processesModelling systems as hierarchies of processes Recording the attributes of the processesRecording the attributes of the processes Embedding a rich uncertainty calculusEmbedding a rich uncertainty calculus Using all available evidence in whatever formUsing all available evidence in whatever form

Added ValueAdded Value Shared understanding of the state of the Shared understanding of the state of the

asset asset across teams across teams up and down the organisationup and down the organisation with all stakeholders ..simplywith all stakeholders ..simply

Meta level system - health overview Meta level system - health overview Identify success,failure and vulnerabilityIdentify success,failure and vulnerability Sensitivity and value of informationSensitivity and value of information Vehicle for testing intervention strategiesVehicle for testing intervention strategies Decision recording Decision recording

corporate memory, transparency, auditabilitycorporate memory, transparency, auditability

Programme Development Programme Development

Civil Engineering SystemsBristol

Juniper

Oil Industry

Texaco

CSIRO

TMX & HLSPeriMeta

FaberMaunsell

Highways Agency

Halcrow

New Applications

Design Information Group

Bristol

CMAM Industrial Partners

EPSRC

Political risk & City vuln 3.02 & CMAM

At its heart………At its heart……… A way to communicate uncertainty A way to communicate uncertainty

and its converse dependability:and its converse dependability:

Sn(A) Sn(A) = Evidence that A is successful = Evidence that A is successful1 - Sp(A)1 - Sp(A) = Evidence that A is not successful = Evidence that A is not successfulSp(A) - Sn(A) = Lack of evidenceSp(A) - Sn(A) = Lack of evidence

Sn(A)Sn(A) Sp(A)Sp(A) 1100

Evidence that AEvidence that Ais successfulis successful UncertaintyUncertainty

Evidence that AEvidence that Ais not successfulis not successful

Interval Probability Theory Interval Probability Theory

Coin Toss:Coin Toss: 0 1

headsclassically:

‘open world’:

Life elsewhere ?:Life elsewhere ?:

Safety

Environment

Pitlochry system

M&E assets

Water control

Tail race

Turbines

***** Drum gates

*****

*******

Pitlochry system

M&E assets

Water control

Tail race

Turbines

***** Drum gates

*****

*******

Cost

Pitlochry system

M&E assets

Water control

Tail race

Turbines

***** Drum gates

*****

*******

Views of performance in selected areas of interest

Model manager

The asset - a

precious resource

Overview

Reservoir F

Dam Water control

Tail race

Towers ***** Drum gates

*****

*******

Hierarchical model of the system

Database of performance

indicators

•Inspections•Reports•Instrumentation•Analysis

0

0.2

0.4

0.6

0.8

1

1.2

0 200 400 600 800

Performance indicator

Val

ue

Values and objectivesRegulatory standards

Library of value functions

Key principles: 1/2Key principles: 1/2 Simple – only one kind of blob and link!Simple – only one kind of blob and link!

The blobs are processes with objectivesThe blobs are processes with objectives

The degree to which a process meets its The degree to which a process meets its objectives is expressed explicitlyobjectives is expressed explicitly

Its performance or dependabilityIts performance or dependability

The asset or system of interest is The asset or system of interest is described described hierarchicallyhierarchically

Each hierarchy layer represents a fairly Each hierarchy layer represents a fairly complete description of the system of complete description of the system of interestinterest

Key principles: 2/2Key principles: 2/2

Layers in the hierarchy can be related Layers in the hierarchy can be related to different levels of decision-making & to different levels of decision-making & to the contingency planning processto the contingency planning process

Evidence of performance is assembled Evidence of performance is assembled from all available sourcesfrom all available sources from expert judgement, visual from expert judgement, visual

inspection reports, instrumentation inspection reports, instrumentation model analysis etcmodel analysis etc

Best built as a challenged group Best built as a challenged group activityactivity

Generating specific viewsGenerating specific views

Multi-attribute weights can be changed Multi-attribute weights can be changed to emphasise specific points of view to emphasise specific points of view on system performance, e.g. safety on system performance, e.g. safety or economics. or economics.

If any performance indicator is If any performance indicator is irrelevant to that aspect of irrelevant to that aspect of performance is can be set a weight of performance is can be set a weight of zero.zero.

Key featuresKey features

Rapid prototyping of models for Rapid prototyping of models for exploring decision scenariosexploring decision scenarios

Comprehensive model building Comprehensive model building process for operational modelsprocess for operational models It is not just a software packageIt is not just a software package

Rapid communication, exploration Rapid communication, exploration and modification with the built modeland modification with the built model

Presidential safety

The Human Decision MakerThe Human Decision Maker

- Creative- Creative - Responsible- Responsible - Operating in an Open World- Operating in an Open World - Achieving :- Achieving :

SatisfactionSatisfactionSelf esteemSelf esteem

Therefore:Therefore: enhance rather than replaceenhance rather than replace

Evidence for a DecisionEvidence for a Decision

Generating the Generating the OptionsOptions

Assembling the Assembling the EvidenceEvidence

Making Making the Choicethe Choice

Determining the Determining the ObjectivesObjectives

Modelling Option PerformanceModelling Option Performance

Comparing Options with Comparing Options with Objectives and states of natureObjectives and states of nature

Taking Action to maximise value and mitigate riskTaking Action to maximise value and mitigate risk

Assessing Assessing Risks & ValuesRisks & Values

Mind mappingMind mapping

Monte-Monte-CarloCarlo

Fuzzy Fuzzy etcetc

Decision TreesDecision Trees AHPAHP

Evidence for a DecisionEvidence for a Decision

Generating the Generating the OptionsOptions

Assembling the Assembling the EvidenceEvidence

Making Making the Choicethe Choice

Determining the Determining the ObjectivesObjectives

Comparing Options with Comparing Options with Objectives and states of natureObjectives and states of nature

Risk Risk ManagementManagement

QRA QRA etcetc

PeriMetPeriMetaa

Assessing Assessing Risks & ValuesRisks & Values

Taking Action to maximise value and mitigate riskTaking Action to maximise value and mitigate risk

Modelling Option PerformanceModelling Option Performance

Philosophical problemsPhilosophical problems

““Seventeenth century natural scientists Seventeenth century natural scientists dreamed of uniting the ideas of dreamed of uniting the ideas of rationality, necessity and certainty into a rationality, necessity and certainty into a single mathematical package, and the single mathematical package, and the effect of that dream was to inflict on effect of that dream was to inflict on Human Reason a wound that remained Human Reason a wound that remained unhealed for three hundred years – a unhealed for three hundred years – a wound from which we are only recently wound from which we are only recently beginning to recover”beginning to recover”Stephen Toulmin 2001 “Return to Reason” Stephen Toulmin 2001 “Return to Reason”

Harvard University Press, p13Harvard University Press, p13

Culture informs process which Culture informs process which defines toolsdefines tools

Why before How

Culturee.g.

Processese.g.

Tools e.g.

WHY

PartneringBest value

Managing Value - RiskGetting AgreementsIncentivising successBuilding the teamEmpowering the supply chainContracts ProfilesProcedures Process mapsWorkshopsRich pictures

WHYHOW

HOW

Building the modelBuilding the modelModel the Model the ProcessProcess

Why before howWhy before how Assemble the evidenceAssemble the evidence

All sources – mapped to common expressionAll sources – mapped to common expression Record the Record the AttributesAttributes

Why Who What When Where How?Why Who What When Where How? Use a Rich Use a Rich Uncertainty CalculusUncertainty Calculus

Gives a powerful handle on the Gives a powerful handle on the dependabilitydependability of of each of the processeseach of the processes

The process of constructing the model The process of constructing the model encourages creative collective reflection encourages creative collective reflection on how the asset system performson how the asset system performs

Evidence from sub-Evidence from sub-processesprocesses

PeriMeta maps the conditional probabilities PeriMeta maps the conditional probabilities to linguistic variables to linguistic variables

H

E1

E2

E3

Propagation of uncertainty requires input of a Propagation of uncertainty requires input of a set of conditional probabilities set of conditional probabilities

Summary of Judgements Summary of Judgements RequiredRequired

EvidenceEvidence - for and against separated - for and against separated SufficiencySufficiency - How much of the evidence is - How much of the evidence is

directly relevant to the parent process? directly relevant to the parent process? 1 = it is sufficient on its own to fully determine 1 = it is sufficient on its own to fully determine

the success of the parentthe success of the parent

DependencyDependency - How much overlap of - How much overlap of evidence is there between the sub-evidence is there between the sub-processes?processes? Issues of bias and redundancyIssues of bias and redundancy

NecessityNecessity - Will the parent fail if the sub- - Will the parent fail if the sub-process fails?process fails? Higher necessity puts more weight on the redHigher necessity puts more weight on the red

The Essence of Risky The Essence of Risky DecisionsDecisions

Examples of UseExamples of Use Oil industryOil industry

Expert interpretation of sparse data managing Expert interpretation of sparse data managing the oilfield assetthe oilfield asset

Political riskPolitical risk Water sectorWater sector

Assessing sustainability of supplyAssessing sustainability of supply Assessing safety of contractAssessing safety of contract

Civil EngineeringCivil Engineering Flood defence decisionsFlood defence decisions Measuring ‘Egan performance’Measuring ‘Egan performance’ Highways Agency PFI MACHighways Agency PFI MAC

Assessment of Terrorist RiskAssessment of Terrorist Risk

Reservoir estimate Reservoir estimate dependabilitydependability

‘Things’ coming out of the woodwork

Field Development Success?Field Development Success?

Spend?

Value of InformationValue of InformationTesting strategiesTesting strategies

Key Drivers for the Highways Key Drivers for the Highways AgencyAgency Procurement have asked for a generic Procurement have asked for a generic

performance specification to apply to new performance specification to apply to new MAC contractsMAC contracts

The modernising government initiative The modernising government initiative (OGC/Treasury/NAO) requires outcome (OGC/Treasury/NAO) requires outcome based processes to improve efficiency and based processes to improve efficiency and effectivenesseffectiveness

SSR are responding to HA’s business needsSSR are responding to HA’s business needs Mapping of rate of progress showed that Mapping of rate of progress showed that

standard approach would be too slowstandard approach would be too slow

PropositionProposition To develop a performance regime that To develop a performance regime that

connects outcomes to what people do (i.e. connects outcomes to what people do (i.e. process) through consultation and ‘learning process) through consultation and ‘learning to improve’:to improve’: Provide a framework for high level performance Provide a framework for high level performance

specification and decision supportspecification and decision support

Motivating people by helping them to understand Motivating people by helping them to understand

where they fit and their contribution to outcomes where they fit and their contribution to outcomes

Provide confidence and competence through Provide confidence and competence through

systematic rigour and recognition of uncertaintysystematic rigour and recognition of uncertainty

Alignment Alignment WOOs

Work on Outputs and OutcomesPurpose: To match route based outputs to network level outcomes/targets.

SUNSSetting up Network StrategyPurpose: To identify business improvements and best working practices. Output: Desk instructions John Bagley (Leeds)

Network Strategy + KPIs

Purpose: Publish KPIs against which to measure performance of key Agency asset management and service delivery activities.

PRIDe Performance Measurement Group

Purpose: To develop better metrics for Area Performance Indicators to facilitate benchmarking across the network. To develop CCC compliance indicators.

Maintenance Contractors

Individual Providers have various systems for demonstrating good performance.

Performance RegimePurpose: To develop performance regime for PFMAC Output: performance specification to allow greater freedom to innovate and improve

Keith Shaw

Keith Shaw

Glynn Harrison

John Fitch

Nick Harding

Halcrow

We are not re-inventing the wheel!

OD Quality Management

(OD Process Mapping)

Keith ShawSimon Smith

Corporate Planning Team

Lisa Scott / Dick TysonPPDG - Performance Planning Development GroupHA Performance Man. – Framework for HA

Supply Chain Management

Integrated teams and continuous

improvementDavid Parker

?

?

Keith Shaw

Cabinet Office and OGC are

promoting the recognition of

uncertainty

Stakeholder DiagramStakeholder Diagram

HAMAC - Internal

Other Stakeholder

s

Road Users

Government

Other Stakeholder

s

Road Users

CommutersFreight

Many Others

Consultants

Contractors

Others

Many Others

Local Authorities Emergency

Services

Suppliers

Measurement Boundaries Measurement Boundaries

Measurement needs to be at the Contract BoundaryMeasurement needs to be at the Contract Boundary

HAMAC

HA Service Boundary

Contract Boundary

Service Delivery

Delivery Slice

Demonstrate

Detail

MAC

Measured Process

Indicators used to measure delivery of

outcomes

Measured Process

Indicators used to measure delivery of

outcomes

DDeliver

DDemonstrateProcess

Demonstrating competence

Process

Demonstrating competence

Process

Demonstrating competence

Process

Demonstrating competence

DDetail Procedures / QMSProcedures / QMS Procedures / QMSProcedures / QMS

DDev

elop

Con

tinuo

us Im

prov

emen

t ove

r tim

e

DDev

elop

Con

tinuo

us Im

prov

emen

t ove

r tim

e

Contract Boundary

Outcomes

Where are we going?Where are we going?

No right ‘answer’, so how can you be sure No right ‘answer’, so how can you be sure you have the right answer!? you have the right answer!?

We are searching for a model that:We are searching for a model that: is robust, and with which the owners are is robust, and with which the owners are

comfortable. comfortable. enables exploration of the performance enables exploration of the performance

regime by all sorts of stakeholdersregime by all sorts of stakeholders Will drive improvement in outcomesWill drive improvement in outcomes Enables alignment to paymentEnables alignment to payment

Journey to Process Model Journey to Process Model

Who? Process owners a useful guideWho? Process owners a useful guide

When ?Reflecting process cycles

Why

How

ProcessWinter ServiceOwnerNetwork ManagerPurpose- Safe mobility in winterRelevant outcome measures- Journey times- Accident rates- Material used (proxy sustainability)

Procedure/Process

Owner

Purpose

Customer Perspective

Safety

Measurement

Procedure/Process

Owner

Purpose

Customer Perspective

Safety

Measurement

Procedure/Process

Owner

Purpose

Customer Perspective

Safety

Measurement

Performance Measurement

Staff (A Good Employer))

HIGHWAYS AGENCY KEY OBJECTIVES

Attributes Attributes of of ProcessProcess

Innovation & Learning

Delivering in Partnership

Best Value (Finance)

Sustainability

NOTE: HA Balanced Scorecard

Perspectives in Blue

Customer perspective

Safety

Types of EvidenceTypes of Evidence Types of evidence considered:Types of evidence considered:

MeasuredMeasured Performance Measurement Group (PMG) Performance Measurement Group (PMG)

measuresmeasures Existing contract Area Performance Indicators Existing contract Area Performance Indicators HA High Level KPI’s (e.g. Balanced Scorecard, HA High Level KPI’s (e.g. Balanced Scorecard,

Government Objectives, etc.)Government Objectives, etc.) Instrumentation, analytical and computer modelsInstrumentation, analytical and computer models

LinguisticLinguistic Expert judgement from form reports or Expert judgement from form reports or

interviewsinterviews

Value FunctionsValue Functions Performance targets are Performance targets are

expressed as value expressed as value

functionsfunctions

Translate PI’s onto a Translate PI’s onto a

non-dimensional 0-1 non-dimensional 0-1

scalescale

0 = failure0 = failure

1 = total success1 = total success

Allows different types of Allows different types of

evidence to be brought evidence to be brought

togethertogether

Safer Travel

Reduce the number of people killed and seriously injured on

trunk roads in 2002-03 by 694 to 4297 (compared with the 1994-98

average 4991)

Mapping Value FunctionsMapping Value Functions

Performance Indicator

Val

ue

Highways Agency Highways Agency measurementmeasurement

Library of Value Functions – HA KPI’s & Targets – 2003/04Library of Value Functions – HA KPI’s & Targets – 2003/04

Maintain at least 85% of the network in good condition.

Effective Maintenance Safer Travel

Reduce the number of people killed and seriously injured on

trunk roads in 2002-03 by 694 to 4297 (compared with the 1994-98

average 4991)

Winter serviceWinter service

Worked exampleWorked example

Winter service processWinter service process

WHY

HOW

WHEN

Propagated

Outcome measures of Outcome measures of successsuccessCurrent Total number 29Current Total number 29 8 Key performance measures +8 Key performance measures +

12 Covered in quality plan 12 Covered in quality plan 9 Other measures 9 Other measures

Reduced to 3 key measuresReduced to 3 key measures Accident rateAccident rate Average journey time through systemAverage journey time through system Quantity of grit used (environmental constraint)Quantity of grit used (environmental constraint)

Plus conformance with plan Plus conformance with plan to fulfil public dutyto fulfil public duty

Benefit of outcome measuresBenefit of outcome measures Simple measures common for many processesSimple measures common for many processes Moderated to be relevant to process context Moderated to be relevant to process context

eg MOORI (Met office open road indicator). eg MOORI (Met office open road indicator). Benchmarked to achieve continuous Benchmarked to achieve continuous

improvementimprovement Empowers those doing the job to improve Empowers those doing the job to improve

performance by performance by Better measurement of their processBetter measurement of their process Continuous improvementContinuous improvement InnovatingInnovating

ConclusionsConclusions Progress is on programme Progress is on programme Strong evidence of need to improve measurementStrong evidence of need to improve measurement Key generic measures to be testedKey generic measures to be tested

Safer travel –KSI reductionSafer travel –KSI reduction Mobility – hours of congestionMobility – hours of congestion Customer Satisfaction – local surveysCustomer Satisfaction – local surveys Minimise adverse environmental impactMinimise adverse environmental impact

Robust process model to enable context based Robust process model to enable context based measurement at MAC boundarymeasurement at MAC boundary

Next stage wider engagement - validationNext stage wider engagement - validation

Overall PeriMeta ConclusionsOverall PeriMeta Conclusions Shared understanding of the state of the Shared understanding of the state of the

asset asset across teams across teams up and down the organisationup and down the organisation with all stakeholders ..simplywith all stakeholders ..simply

Performance Regime Performance Regime health & vulnerability overview health & vulnerability overview

Sensitivity and value of informationSensitivity and value of information Decision recording Decision recording

corporate memory, transparency, corporate memory, transparency, auditabilityauditability

Prediction

Uncertainty

Models

Uncertainty

Risk

Decision

Monitoring

Uncertainty

Hazards

Environment

Vulnerability

Systems

Uncertainty

Information

Surprise

Outcome

Uncertainty is a property of information – fuzziness, incompleteness and randomness

Risk is the likelihood of an uncertain event or behaviour, and its consequences for our intended purpose or objectives, set in a context that needs to be understood

Hazard - set of incubating preconditions for failure

Vulnerability – susceptibility to disproportionate damage from an event or behaviour

Surprise - an unexpected event – an unrecognised risk

TerminologyTerminology

FuzzinessFuzziness - Imprecision of - Imprecision of definition definition

Incompleteness - That which we do Incompleteness - That which we do notnot know, choose not know, choose not toto include or cannot include or cannot afford to include afford to include

RandomnessRandomness - Lack of a - Lack of a specificspecific pattern pattern

The Nature of UncertaintyThe Nature of Uncertainty

The Nature of UncertaintyThe Nature of Uncertainty

Fuzziness Incompleteness Fuzziness Incompleteness RandomnessRandomness

Tools for Uncertainty Tools for Uncertainty ManagementManagement

Analogue StudiesAnalogue Studies Case Based ReasoningCase Based Reasoning Parametric StudiesParametric Studies Safety FactorsSafety Factors Monte Carlo Simulation Monte Carlo Simulation

Bayesian ReasoningBayesian Reasoning Fuzzy MethodsFuzzy Methods Neural NetsNeural Nets

Genetic AlgorithmsGenetic Algorithms Evidential Evidential Reasoning Reasoning Process modelsProcess models

Just because...Just because...

... past futures have resembled past pasts, ... past futures have resembled past pasts,

it does not follow that future futures will it does not follow that future futures will

resemble future pasts.resemble future pasts.

Bryan Magee Bryan Magee “Popper” “Popper” 19731973

Understanding risksUnderstanding risks

Predict from

history

Understanding what we do not know

Look out

for change

Priority

actionoccurence

consequence

frequent

infrequent

highlow

VulnerabilityVulnerability

Now done in many areas manuallyNow done in many areas manually Complexity of systems means that is Complexity of systems means that is

now unreliable to depend on unaided now unreliable to depend on unaided human identificationhuman identification

Need systems approach as opposed Need systems approach as opposed to a reductionist paradigmto a reductionist paradigm

JuniperJuniper

‘‘There is a need for “generating new There is a need for “generating new processes for imagining imaginative processes for imagining imaginative outcomes” ’.outcomes” ’.

Schneider “The future of climate potential for Schneider “The future of climate potential for interaction and surprises” In Downing: Climate interaction and surprises” In Downing: Climate Change and World Food Security. Springer Berlin Change and World Food Security. Springer Berlin (1996 p79)(1996 p79)

Risk Management PlansRisk Management Plans

( KnowRisk – Australian RM software )( KnowRisk – Australian RM software )

Uncertainty measurementsLayout that gives view on Uncertainty

QRA QRA

Limited scope – applicable in tightly Limited scope – applicable in tightly constrained physical environmentsconstrained physical environments

Needs frequency databaseNeeds frequency database Makes bold assumptionsMakes bold assumptions Difficult to bring in ‘soft’ systemsDifficult to bring in ‘soft’ systems

Yet we need to mix hard and soft……..Yet we need to mix hard and soft……..

HA HA BalanceBalanced d ScorecarScorecardd

Staff Perspective How effective are we at managing, developing and motivating our workforce?

Efficiency & Finance

How good are we at managing our financial resources?

Customer Perspective

How effective are we are delivering customer requirements?

Delivering in Partnership

Do we work in synergy with our partners?

Innovation & Learning

How good are we at preparing for the future and learning from the past?

HA

Vision

Safe RoadsReliable Journeys

Informed Travellers‘Travelling with Confidence’

7 Oct 027 Oct 02