AITPM Technical Forum – Risk and Governance – August 2015Stephen Luke

What am I covering?

• Internal processes for risk and governance

• Case Study : west coast intercity rail franchise

• Emerging UK considerations : Aqua (Quality analysis for government)

Context

Mott MacDonald Credentials

• We are at the forefront of developing guidance and methods.

• We have developed industry standard software, for example for demand modelling (DIADEM) and for business case appraisal (TUBA, INCA, WITA and PAR).

• We contribute to industry guidance, for example we have written the latest Highways England guidance on Appraisal of Technology Schemes.

Mott MacDonald Credentials

• We have a wide range of application experience, from large scale strategic and multi-modal applications to detailed microsimulation models of traffic management schemes or pedestrian movements

• Highway modelling - Norwich Ring Road, UK

• PT modelling – Abu Dhabi Surface Transport masterplan, UAE

• Demand modelling

• Microsimulation – Calgary LRT, Adelaide

• Pedestrian modelling – North West Rail link, Melbourne Metro

• Junction modelling – Sydney LRT

• Economic Appraisal – Transparent Economic Assessment

• Freight modelling – Transnet, South Africa

• Land use modelling

• Aviation modelling – London airports

• Rail Modelling – High Speed 2

• Research

• Transport modelling for private financed projects

Wider Thoughts

• It’s not about software!

• Decision making without traditional transport models – other ways of quantifying policy or project impacts

Who is making the decisions?

Do they really know what they want?

When do they need an answer?

How robust does the answer need to be?

Do they understand the data available?

Have they any confidence in model outputs?

Have they any technical knowledge?

Are they politically motivated?

Have they pre-conceived answers?

Application of Transport Modelling

• As a front-end service to help determine the policy or infrastructure solutions

• Provide an evidence base to support the preparation of a scheme business case

• Provide an evidence base for the scheme environmental appraisal

• To support transport infrastructure management business, for example for Managing Agent Contractor contracts

Continuous Improvement

Develop Transport

Model

Update Risks

Mitigation Opportu-

nities

Best practice

processes

Checking Review

Learning

ContinuousImprovement

Transport Modelling Practice

• Delivery– Peer Assist throughout project delivery

– Checking plans

– Share lessons

• Staff Development– Encourage writing papers, attending

conferences, technical committees

– Helping to meet career aspirations

Transport Modelling Community site

• Contacts and expertise

– Skills matrix

• Best Practice

• Links and resources

Best practice papers – Why do we need them?

• Simple errors in models have stopped schemes progressing

• Clients have lost confidence and the industry has recognised change is needed

– HA best practice networks

– TRAMPNET best practice network set up by PTRC

• There are gaps in published guidance

• Modelling is a risky business!

Best Practice Papers

1. Peer assist and checking requirements

2. Addressing risks and liabilities in receiving and delivering transport models and data

3. Model design and data requirements

4. Record keeping and version control

5. Spreadsheet management

6. Network building

7. Checking transport models

8. Matrix building

9. Matrix estimation

10. Calibration and validation

11. Forecasting

12. Microsimulation

Best Practice – Key general messages

• We need to identify and mitigate risks - our processes are designed to help us do this

• ALL transport modelling projects must be subject to peer assist

• Apply standard QES procedures

• Models should be checked, and the checks recorded so this can be demonstrated

• We must all use best practice procedures (or have a good reason not to)

Best Practice – Peer Assist

• The rationale behind the application of Peer Assist in transport modelling is to identify problems early, to have an explicit approach to the identification and mitigation of risk, and to make better use of the expertise available

• PA at scoping stage to identify and cost a checking plan

• Peer Reviews at key stages, at least at Tender and 1/3 and 2/3 through delivery

Best Practice – Spreadsheet management

• Name to describe calculation

• Keep old versions in superseded directory

• Use Spreadsheet with QA front sheet for all calculations

• Describe the calculations: the description should be of sufficient detail such that any other technical colleague can open up the spreadsheet and be able to understand how the calculation works, and how they might update it if any of the key inputs have changed.

Best Practice – Checking networks

• Input data checks

• • Source material

• • New coding against scheme drawings or other data

• Model run checks

• Version of software

• File names

• Assignment warning messages

• Assignment convergence

• Demand model warning messages

• Demand model convergence

Best Practice – Checking networks 2

• Output results checks – Network statistics

– Changes in trip matrices

– Journey times

– Simulation junction performance

– Routings

– Assignments

– Fixed matrix

– TUBA

Best Practice – Matrix Building

• Lots of existing data sources

• Cleaning and verifying survey data

• Building station matrices

• Combining station matrices to get complete observed matrices

• Synthetic matrix methods and infilling

• Validation of matrices

• Sense checking

Best Practice - Calibration and Validation

• Check guidelines at beginning – including requirements for reporting

• Lots of published guidance in DMRB and WebTAG and from TfL

• Calibration – adjustments to model, but a large part is checking

• Validation – comparison with independent data

Summary

• Transport modelling is a risky business!

• Protection by using best practice procedures

• Keep up to date

– Best practice papers

– Transport modelling community

• Plan checking as part of Peer Assist and scoping

• Record checks

West Coast Intercity Rail Franchise

https://www.gov.uk/government/publications/report-of-the-laidlaw-inquiry

http://www.nao.org.uk/report/lessons-from-cancelling-the-intercity-west-coast-franchise-competition/

• A very public failure

• Technical errors with modelling

• Risk and governance issues

• Comprehensive documentation of lessons learnt in public domain

Annual revenue >$1.5bn pa (2009-10)

Background

• Nearly all UK intercity rail services are franchised

• Initially 25 Train Operating Companies (TOC’s)

• Net cost operation

• TOC separate from operating Group

• Initial franchises awarded solely on financial bids

• GFC (and overbidding), franchises failed

Background

• Macroeconomic risk?

• Cap and collar – surpluses capped, losses collared

• Subordinated Loan Facility (SLF) – Insurance

• GDP Mechanism

• The DfT Model developed for another purpose was used to calculate the SLF

• Bids submitted in excel model form

Findings

• DfT model did not account for inflation (assumed it did)

• Technical confusion regarding elasticity factors to be used

• Bids not properly quantify risk and understated SLF

• DfT model not provided to bidders (lack of transparency)

• Adjustments made by Govt. tender assessment team. Inconsistent, not clearly documented

In laymans terms

• There was a lack of transparency. The Department did not give bidders enough information on which to base their bids.

• The Department did not follow its own published guidance.

• The amount of capital that the two final bidders were asked to put into their bids was understated and inconsistently determined.

• The Department’s planning and preparation was inadequate.

• Roles and responsibilities for the project were unclear and resources were stretched.

• The Department’s governance lacked efficacy.

• Quality assurance was inadequate – limited scope and late (audit afterwards!)

Lessons

• Clarity of objectives helps decision makers to form appropriate judgements by being a touchstone to refer back to throughout the decision-making process.

• Strong project and programme management brings together and coordinates the different streams of work, identifies interdependencies and the sequence of events – the critical path – a programme needs to follow.

• Senior oversight acts as a sense check.

• Effective engagement with stakeholders, such as suppliers, helps by contributing their knowledge, signalling problems and brings them into the process.

• Internal and external assurance provides a sense check and can identify any areas of concern to management

Recommendations

• Clarity of objectives

– Apply project and programme management disciplines to forming policy. Set timetables, identify key tasks and their dependencies, identify a critical path for making policy changes and allocate clear roles and responsibilities to deliver individual elements and the policy as a whole.

– Identify the technical tools and models it requires to implement policy before delivery commences. Develop, quality assure and test these processes before it moving on

– Provide training to staff on any new tools or policies. Before projects enter operational stages, staff need training so that they understand objectives and how to apply processes and tools.

• Project and programme management

– Timetables for major projects and programmes so they are realistic. It should consider the ‘usual’ timescales for typical projects and programmes, identify novel factors that might impact on these and be cautious in shortening existing timetables.

– Key decision points build in sufficient time to properly consider decisions, include contingency in case extra work is required, and consider other options if it cannot decide to proceed.

AQUA

• The Aqua Book:

• Guidance on producing quality analysis for government

• https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government

AQUA

effective quality assurance is achieved by creating an environment that is conducive to quality assurance and by embedding appropriate and proportionate processes

AQUA

• environment: creating the conditions in which quality assurance processes can operate effectively, facilitated by a culture that values quality assurance and welcomes effective challenge, a well understood chain of responsibility and sufficient time for quality assurance; and

• process: establishing a clear process for every stage of the analytical life-cycle. This includes ensuring there is a shared understanding about the purpose and any limitations of the analysis

Model Definition

AQUA

AQUA

Summary : Analysis with RIGOUR

• Repeatable

• Independent

• Grounded in reality

• Objective

• Have understood and managed uncertainty

• Results should address the initial question robustly

• Establish how much we can rely upon the analysis for a given problem.

… TO THIS

WHAT ABOUT THIS?

www.mottmac.com