5/19/2018 Sally Edwards and Simon Gault
1/32Model Development Unit
Strategy Directorate
1
Developing & MaintainingDynamic Micro-Simulation
Models at DWP
Sally Edwards
Simon Gault
ESRC 2ndApril 2009
5/19/2018 Sally Edwards and Simon Gault
2/322 Model Development UnitStrategy Directorate
Developing Models at DWP
1.Overview
2.Objectives
3.Development Standards4.Simplifying Complexity
5.Maintenance Protocol
6.What worked well and Lessons Learned
5/19/2018 Sally Edwards and Simon Gault
3/323 Model Development UnitStrategy Directorate
Introductions
Simon Gault
Statistician, with a strong background in Economics
Extensive experience in developing models
Manages the Forecasting Simulation models team
Sally Edwards
Systems / Business Analyst
Predominantly worked in the private sector on large scale IT systems
Manage the Genesis Simulation Model
Members of a the Model Development Unit(MDU) which hasresponsibility for Micro-Simulation Models at DWP
MDU consists of 20+ people
5/19/2018 Sally Edwards and Simon Gault
4/324 Model Development UnitStrategy Directorate
Micro-Simulation Models
Dynamic Micro-Simulation Models, based on our standard
architecture, Genesis:
Pensim2private & state pension income
InformIntegrated Forecasting model for working agebenefit claimants
Individual Benefit Forecastingfor RP, IB, DLA/AA
Employment Model, currently being validated
Static Micro-Simulation Models:
Policy Simulation Model(PSM)
Employment and Hours Model(EHM)
5/19/2018 Sally Edwards and Simon Gault
5/325 Model Development UnitStrategy Directorate
Key factors taken into Consideration during
Development
Key factors considered when we designed our first dynamic micro-simulationmodel, Pensim2 - also valid for our other models:
Simplify the complexity of the model as much as possible, using a modularapproach and allowing a simple prototype model to be developed quickly
Several developers work on each model at the same time
Large staff turnover
Models are intended for long term use (e.g. Pensim2 anticipated to have15-20 year life) so most of the people using/maintaining the models arenot involved in their development
Clear documentation, processes, training and user guides are essential
Excel and SAS are the preferred tools, as both are well understood andextensively used among the Analysts at DWP
5/19/2018 Sally Edwards and Simon Gault
6/326 Model Development UnitStrategy Directorate
Developing Models at DWP
1.Overview
2.Objectives for Pensim2 (first DMS model)
3.Development Standards4. Simplifying Complexity
5.Maintenance Protocol
6.What worked well and Lessons Learned
5/19/2018 Sally Edwards and Simon Gault
7/327 Model Development Unit
Strategy Directorate
Original Objectives for Pensim2
Pensim2 Feasibility Study document 2002) Objectives
1. Flexibility 8. Efficiency (in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10. Done with own expertise4. User-friendliness 11. Done with own resources
5. Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14. Independence of base data
(i.e. desktop model)
5/19/2018 Sally Edwards and Simon Gault
8/328 Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution sodecision was made to split the model into two separate parts,satisfying most of the objectives:
5/19/2018 Sally Edwards and Simon Gault
9/329 Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution sodecision was made to split the model into two separate parts,
satisfying most of the objectives:
Excel based front end Pensim2
User friendly, transparentcode in a standard format
Easy to understand and flexible, enabling analysts to changeparameters & structure of model without understanding theunderlying code.
Easy to maintain and handover
Allowed independence of base data
5/19/2018 Sally Edwards and Simon Gault
10/3210 Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution sodecision was made to split the model into two separate parts,
satisfying most of the objectives: :
Excel based front end Pensim2
User friendly, transparentcode in a standard format
Easy to understand and flexible, enabling analysts to changeparameters & structure of model without understanding theunderlying code.
Easy to maintain and handover
Allowed independence of base data
SAS code generator - Genesis
Developed with our own expertiseandown resources
Robustgenerator, producingreliable outputs
5/19/2018 Sally Edwards and Simon Gault
11/3211 Model Development Unit
Strategy Directorate
Basic structure of Genesis
Dynamic Micro-Simulation Models at DWP
GENESISModel Engine
generates SAS model
and runs the simulation
Simulation
Modelparameter
sheets (Excel)
SASBase
Data
SASpost-
simulation
Data
- Standard Excel sheetsused to define the model
- SAS chosen for the model engine and the output data,
as SAS is the standard data analysis tool used
SAS Static
code
5/19/2018 Sally Edwards and Simon Gault
12/3212 Model Development Unit
Strategy Directorate
Developing Models at DWP
1.Overview
2.Objectives
3.Development Standards4.Simplifying Complexity
5.Maintenance Protocol
6.What worked well and Lessons Learned
5/19/2018 Sally Edwards and Simon Gault
13/32
13 Model Development UnitStrategy Directorate
Development Standards
Two separate development teams, with different
development standards adopted
Genesis Model Engine
Developed by IT staff Used standard IT project development procedures
Pensim2 Analytical team
Economists and Statisticians
Less structured approach to development
5/19/2018 Sally Edwards and Simon Gault
14/32
14 Model Development UnitStrategy Directorate
Genesis Model Engine Development Standards
Highly structured approach to the development
Project Management Protocol based on Prince2 methodology, although notall aspects were needed
Documentation 2 types Development documentationproduced at each stage of project, that
was signed off and used as input to the next stage Key Projectdocumentation (*)- maintained and kept up-to-date with
each new release
Development Processessimple process diagrams to provide clearguidelines, particularly useful for new staff
Detailed Requirements explaining the full functionality of the modelengine. Document translates Economist language into IT language(*)
5/19/2018 Sally Edwards and Simon Gault
15/32
15 Model Development UnitStrategy Directorate
Genesis Model Engine Development Standards
Design Diagrams and documentation used in development stage only
Change Control processchanges estimated and impacted before beingaccepted/rejected/on hold (*)
Problem Log processproblems recorded, prioritised, fixed/removedfrom the scope/user error (*)
Validation Process - detailed Test Plan based on Requirements document,used to ensure that each requirement was fully satisfied
Test Packensured existing functionality continued to work when changeswere made to the model engine (*)
User Guidecontains details of functionality written in simple format (*)
Trainingmaterials, inc presentations and training text for self-study (*)
5/19/2018 Sally Edwards and Simon Gault
16/32
16 Model Development UnitStrategy Directorate
Pensim2 Model Development
Pensim2 Analytical development followed a less structured
approach
Generally this was appropriate, as detailed requirements were notpossible to establish up-front
Effort was predominantly spent on regression analysis
Analysis results reviewed to determine the key components toinclude and the most appropriate sources for the assumptions
Using the standard Genesis templates, the Pensim2 model wasmore easy to develop, rather than if a traditional DMS model hadbeen built
5/19/2018 Sally Edwards and Simon Gault
17/32
17 Model Development UnitStrategy Directorate
Developing Models at DWP
1.Overview
2.Objectives
3.Development Standards4.Simplifying Complexity
5.Maintenance Protocol
6.What worked well and Lessons Learned
5/19/2018 Sally Edwards and Simon Gault
18/32
18 Model Development UnitStrategy Directorate
Key Aim of Genesis Structure - Simplify Complexity
No Hard Coding Rule which allows complete flexibility for:
Structure of the simulation, i.e. order of events processed
Data structures, i.e. data tables, data relationships and variables
Parameters, e.g. indexes, rates, dates
Regression Equations
Selection filters
Static SAS code, if required (e.g. addition of new cases to a model)
Ability to add / remove modules easily
Each model is defined in simple Excel spreadsheets, using standard templates,that users can easily understand and modify
5/19/2018 Sally Edwards and Simon Gault
19/32
19 Model Development UnitStrategy Directorate
Key Aim of Genesis Structure - Simplify Complexity
Data Access routines
Enable easy access to data so user does not have to write complex code
Default data access will process the current object in the current simulationyear
Facility to process a variable belonging to a related object, e.g. partner PAR;Relationships are not hard-coded and new relationships can easily be
defined
Data manipulation routines provide commonly required data facilities,e.g. maximum value during past 10 years = MA10
E.g.MA10_PAR_pa_salaryreturns the maximum value in the past 10 yearsof
the salaryvariable for the individualspartner,taken from thepa table
Keep Equations simple
Equations, filters & process flows are defined in a simple, standard format,that is unambiguous. Hence the models are easy to maintain
5/19/2018 Sally Edwards and Simon Gault
20/32
20 Model Development UnitStrategy Directorate
Developing Models at DWP
1.Overview
2.Objectives
3.Development Standards4. Simplifying Complexity
5.Maintenance Protocol
6.What worked well and Lessons Learned
5/19/2018 Sally Edwards and Simon Gault
21/32
21 Model Development UnitStrategy Directorate
Change Management
Change Management Process is followed for all models
Every change goes through the formal Change Control procedure, i.e.requirements documented, impact assessed, agreed by users & signed off
Change Registerlisting all changes, each with a separate reference number
For each change to the model, a separate Change Controlform is produced
The model code contains the Change ref number, with a comment
Some changes are repeated regularly, e.g. assumptions, so clear
documentation is helpful for the repeat change
Questions arise after a change has been implemented - these can be easilyanswered by referencing the Change Control documentation
5/19/2018 Sally Edwards and Simon Gault
22/32
22 Model Development UnitStrategy Directorate
Problem Management
Problem Management Process is followed for all models
Everyproblem/error discovered in a model goes through the formalProblem Log procedure, i.e. requirements documented, impact is assessed,agreed by users & signed off
Problem Log Register, listing all problems identified, including those that
turn out to be User Errors or misunderstandings; PL ref number assigned
For each problem recorded, a separate Problem Logform is produced
Users are notified of significant problems that affect the model results
Many of the problems are minor and are picked up by the developmentteam these are recorded formally, even if they do not impact the results
5/19/2018 Sally Edwards and Simon Gault
23/32
23 Model Development UnitStrategy Directorate
User Model
Our models fall into 2 categories: Maintained & owned by the Model Development Unit Developed by Model Development Unit; handed over to User
team, who then own and maintain the model
Models owned by the Model Development Unit have a User Groupand/or a Steering Group consisting of representatives from eachteam using the model
The purpose of these groups is to agree the content, priority and
timing of changes made to the model
5/19/2018 Sally Edwards and Simon Gault
24/32
24 Model Development UnitStrategy Directorate
User Group Purpose
Specifically for Pensim2, our largest model:
Change Requests and Problem Logs are reviewed by the UserGroup, where appropriate
A key user sponsor is assigned to each change. The sponsor
ensures that the change is accurately specified & will sign-offtesting
The User Group are responsible for signing off new versions ofthe model before they are released
The users also have a separate User Forum where they presentanalysis carried out on the model outputs and explain how theresults have been used
5/19/2018 Sally Edwards and Simon Gault
25/32
25 Model Development UnitStrategy Directorate
Release Management
Again, specifically for Pensim2:
New Releases are implemented approximately every 6 to 9 months
Each Release includes one or two key components, e.g. Private PensionReform, and a bundle of smaller changes and problem fixes
The timing and content of each Release are agreed with the User Group
When a new Release is built: Each change/fix is added to the model one at a time Hence each change is reviewed and signed off separately Enables us to fully understand and agree the effect of each change.
Time consuming but gives confidence in the results
Clear audit trail
All previous versions of the model remain available
5/19/2018 Sally Edwards and Simon Gault
26/32
26 Model Development UnitStrategy Directorate
Developing Models at DWP
1.Overview
2.Objectives
3.Development standards4.Simplifying Complexity
5.Maintenance protocol
6.What worked well and Lessons Learned
5/19/2018 Sally Edwards and Simon Gault
27/32
27 Model Development UnitStrategy Directorate
Original Objectives Satisfied
Pensim2 Feasibility Study document 2002) Objectives
1. Flexibility 8. Efficiency (in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10.Done with own expertise4. User-friendliness 11. Done with own resources
5.Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14.Independence of base datai.e. desktop model)
5/19/2018 Sally Edwards and Simon Gault
28/32
28 Model Development UnitStrategy Directorate
Objectives Partially Satisfied
Pensim2 Feasibility Study document 2002) Objectives
1. Flexibility 8. Efficiency (in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10. Done with own expertise4. User-friendliness 11. Done with own resources
5.Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14.Independence of base data
i.e. desktop model)
5/19/2018 Sally Edwards and Simon Gault
29/32
29 Model Development UnitStrategy Directorate
Objectives that proved more tricky
Pensim2 Feasibility Study document 2002) Objectives
1. Flexibility 8. Efficiency in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10. Done with own expertise4. User-friendliness 11. Done with own resources
5.Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14.Independence of base datai.e. desktop model)
5/19/2018 Sally Edwards and Simon Gault
30/32
30 Model Development UnitStrategy Directorate
What went well
Generic framework has proved successful and easy to use
Genesis architecture has been remarkably robust
Modules have been reused/copied from one model to another
Quicker development of new dynamic micro-simulation models,compared with a traditional build
Automated documentation facility and error checking producedusing VBA, listing where variables are used and providing an
overview of each module, to ensure consistency
Lessons learnt from Pensim2 development were used whendeveloping subsequent models
5/19/2018 Sally Edwards and Simon Gault
31/32
31 Model Development UnitStrategy Directorate
Lessons Learned
Key lessons learnt from Pensim2 development:
Tried to make the model too complicated particularly the analytical side
We should have developed a set of simple modules for a Phase 1 delivery,then added complexity to the more important modules for Phase 2
The structure of some of our modules was more complicated than the datawould support, e.g. Labour Market process
Insufficient documentation produced by the Analytical team during thedevelopment (e.g. Base Data Creation)
Genesis Engine code is difficult to understand and hence modify, althoughit is robust and does not require much modification
Some Genesis Engine functionality was over-ambitious and subsequentlydropped (e.g. Alignment of polychotomous variables)
5/19/2018 Sally Edwards and Simon Gault
32/32
32
Any questions?
Top Related