A modeller’s dilemma: overfitting or underperforming

A Modeller’s Dilemma: Overfitting or Underperforming?

Tracey Pershouse & Yun Bu (AECOM)

About this presentation

Requirements for transport modelling

Three case studies

o Trip generation

o Trip distribution

o Trip distribution and mode choice

Summary

Requirements for Transport Modelling

Requirements for Transport Modelling

Replicating base year situations: In practice, it is used as key measurement of model quality and is often the focus of model development

Forecasting future year implications: In practice, it receives less attention although forecasting is the ultimate use of models

Case Study – Trip Generation

Case Study – Trip Generation

Alternative B: Introduce retail job density as an additional explanatory factor for the number of attractions of shopping trips

Presentation Title May 1, 2023

Modelling trip attraction to major shopping centres:

HBS trip attraction = a * number of retail jobs + b * Ln(retail job density, in jobs / hectare)

Alternative A: Flags a number of shopping centers as ‘special generators’ and apply adjustment factors derived from observed data

HBS trip attraction = [a * number of retail jobs] * adjustment factors

Validation Performance


The Alternative B produces considerably worse validation results for the largest centres (A and B) but better outcomes for relatively smaller shopping centres (C to H).

Forecasting Performance

Alternative A: growth in jobs and growth in trip attractions are similar at all the shopping centres

Alternative B, faster growth rate for trip attractions at the shopping centres with relatively lower density, and slower growth rate at D where the density reaches a very high level

Case Study – Trip Distribution


May 1, 2023

Calibrate gravity model for HBW trip distribution:

Alternative A:

- Use observed trip productions and attractions as inputs for model calibration, even though it is modelled trip ends are that input to the gravity model for model application

- Only include the OD movements with observed trips > 0, excluding the ODs with zero trips, even though some of the zeros may reflect travellers’ choice of not travelling



Calibrating gravity model for HBW trip distribution:

Alternative B:

- Use modelled trip productions and attractions as inputs for calibrating the gravity model parameters

- Include all the o-d pairs including those with zero trips. This requires aggregating the movements to sector (SA2) level to remove the zeros that are due to the small sample size in HTS

Base Year Performance – for model calibration

May 1, 2023

Observed number of trip productions and attractions are used to produce the trip distribution curves

Base Year Performance – for model validation


Modelled number of trip productions and attractions are used to produce the trip distribution curves

Forecasting Performance


Modelled elasticity of changes in trip length in response to cost increase

Case Study – Trip Distribution and Mode Choice



Alternative A: mode choice occurs after the trip distribution (or destination choice)

This ordering is computationally convenient: OD trip tables from distribution provide inputs to the mode choice

Arguably, this order also fits in a modelers’ experience of real life in so far as on many occasions travelers’ decide on their destinations first and then choose between transport modes

Develop destination and mode choice model for HBW trips:



Develop destination and mode choice model for HBW trips:

Alternative B: mode choice occurs before the trip distribution (or destination choice).

Less computationally straightforward: the generalized cost input for mode choice must be aggregated at trip production level, instead of at trip (origin-destination) level.

Due to the aggregation of cost inputs, the resultant base year validation outcomes tend to be less desirable than the destination choice first approach.

Validation Performance - Overall mode shares


Similar performance between the two alternatives

Validation Performance – Mode share for SA2 to SA2 movements


Alternative A:

Validation Performance – Mode share for SA2 to SA2 movements


Alternative B – similar performance to Alternative A

Forecasting performance

May 1, 2023

Car GC = 30 mins

PT GC = 70 min

Car GC = 30 mins

PT GC = 70 min

D1

D2

Base case assumptions:

1000 trips from origin to destination D1, D2, via Car or PT

Destination and Mode choice results in base case (for both alternative):

D1, Car = 478 trips

D1, PT = 22 trips

D2, Car = 478 trips

D2, PT = 22 trips


May 1, 2023

Car GC = 40 mins

PT GC = 70 min

Car GC = 30 mins

PT GC = 70 min

D1

D2

Scenario testing:

Car travel cost to D1 increases from 30 to 40 minutes

Destination and Mode choice results , Alternative A (double constrained)

D1, Car = 432 trips (decrease from 478 in Base)

D1, PT = 68 trips (increase from 22 trips in Base)

D2, Car = 478 trips (unchanged from Base)

D2, PT = 22 trips (unchanged from Base)


May 1, 2023

Car GC = 40 mins

PT GC = 70 min

Car GC = 30 mins

PT GC = 70 min

D1

D2

Scenario testing:

Car travel cost to D1 increases from 30 to 40 minutes

Destination and Mode choice results , Alternative B

D1, Car = 394 trips (more significant decrease)

D1, PT = 106 trips (more significant increase)

D2, Car = 480 trips (nearly unchanged)

D2, PT = 20 trips (nearly unchanged)

Summary

Implications

Overfitting: Use synthetic measures for fitting base year outcomes to observed data, which may affect forecasting performance adversely.

Underperforming: Accommodate underperformance (compared with overfitting option) in base year validation, in exchange of improved forecasting performance

Another option:

Common practice?

Thank You

A modeller’s dilemma: overfitting or underperforming

Education

Transcript of A modeller’s dilemma: overfitting or underperforming