Analysis of usability of scheduling algorithms: a case study in the

Netherlands Railways

R.J. Jorna, W. van Wezel & J. BosFaculty of Economics and Business, University of Groningen (Groningen, The Netherlands)Contact: r.j.j.m.jorna@rug.nl

Third Rail Human Factors Conference;Lille, France, March 3-5, 2009

2

Structure

Introduction– Netherlands Railways and their planning– Background & Problem statement: effects of planning support

Research:– What does “inadequate use” of algorithms mean?– Results of experiments with 8 planners– Comparison planners and OR-specialist

Algorithms and the function-allocation theory Conclusions, statements and further research

3

The Netherlands Railways and Planning

Netherlands Railways (NS): 2700 railroad carriages; per day 5000 train services; 384 stations.

NS overall: about 350 planners; 4 major kinds of planning:

– A: timetables;– B: planning rolling stock and rolling staff;– C: local and central planning (stock and staff);– D: year plan (long term) & day plan (short term)– Sequence: timetable, rolling material, rolling staff

and then shifts for individual persons.

4

Background and problem statement

In staff planning shift schedules need to be changed frequently (see Figure 1).

Here, study of 8 planners of Day Planning (Central) (re)schedule shifts of train drivers/ticket collectors.

Planners use DSS CREWS (see Figure 2), with a GUI to create plans by dragging and dropping.

Recently, an algorithm to automatically create plans is included. Planners can influence algorithm by setting parameters.

Problem: Use of algorithm did not result in expected increase in performance. What is going on?

5

Example of change within shifts

Figure 1

6

Figure 2Screen shot of CREWS

7

Theoretical assumptions: general

Task performance with DSS is a function of – Planning problem complexity,– Quality of automation (software support level),– Mental load of planners,– Many other aspects (trust, cost, cooperation, etc.)

Use of algorithmic support can be studied in quasi-experiments with planners

8

Cognitive workloadSituation awareness

ComplacencySkill degradation

Quality of automationRisk of automation failure

TrustCost of incorrect decisions

How can we apply thisto planning / scheduling?

Factors influencing the adequate use of planning support

9

Theoretical assumptions: model

Relation between independent variables and performance

Two conditions: manualand algorithm support

Two conditions: simple and complex problem

10

Experiments with planners (1)

Procedure– Planner reads assignments, solves problem with system;– For Mental Load: planner uses NASA TLX questionnaire;– During experiment, number of activities of planner are

counted: a) use of searching functionality in system; b) # backtrackings (withdrawn reasoning steps); # decisions resulting in constraint violations

Simple problem Complex problem

Number of shifts 31 94 Number of stations 4 6

11

Experiments with planners: results simple problem (2)

Task performance for simple problem (N = 8)

Indicators (1) Average Manual (2)

Algorithm vs Manual (3)

Average Algorithm (4)

Conflicts 0,1 0 0 # Passenger tasks 24,6 + 14,7 # Taxi rides 0 0 0 >30 minutes exceeding shift length 1 0 0,6 Extra shifts 0 0 0,3 Empty shifts 1,4 + 3,7 Solution time (minutes) 63,3 + 43,1 # Reasoning steps 137,1 + 53 Reasoning Steps withdrawn 5 + 2,3 Wrong steps 8,7 + 3,1 Search / Filter 47,4 + 11,6 > Related to stock 16,9 + 2,7

For simple problem: algorithm better than manual; for complex the reverseQuestion: WHY?

+: algorithm better

12

Experiments with planners: results mental load and complex problem (3)

No difference mental load for simple problem (N=8) nor for complex problem (N=2)

Experiment with complex problem took more than 8 hours (only two planners were analyzed)

Mental load for simple problem Planner Manually With algorithm Average 34,6 36,3 SD 9,8 15,5

13

Experiments with planners: conclusions (4)

Some conclusions: For algorithms, planners are concerned with usability:

– does planner understand outcome of algorithm?– can schedule be explained to the staff that will execute it?– can planner change schedule easily, etc.)

(all of the above are difficult to formalize) If algorithm is not aligned to planner, no increase of performance In using algorithms, significant individual planner differences. Individual planner differences concerning: mental load, time to

solve problem, quality of outcome. Planners spend much time on configuring parameters and

interpreting the outcome (from protocols).

14

Experiment comparing planners, algorithm and developer (5)

 (n=2) Planners without algorithm

Planners with algorithm Developer

Time (minutes) 290 185 60Extra shifts 14 12 7Empty shifts 12 8 6Number of steps 709 271 27Passenger tasks 56 48 36Taxi 0 0,5 7Conflicts 0 0 0Exceed max shift length 16 11 3

Planners can not plan as good as OR specialists?

15

What is wrong with planners?

Planners do not use their algorithms correctly and should be trained

or Algorithms are not developed to be usable by planners

(The algorithm was designed by winners of 2008 Informs “Franz Edelman Award for Achievement in OR and the Management Sciences”. Should they, the developers, become planners?)

Is planning/scheduling in railways a matter of control of dynamic situations and should support be adjusted to this kind of task?

16

12-02-2008 | 16

CALCULATEDURING THE

NIGHT

FIX ERRORS

17

Basic assumptions concerning planners in algorithmic support

Interaction with algorithms implies knowledge of the model being used, e.g.:

– Specify parameters; specify weights on goal functions– Steer the backtracking process of the algorithm– Fix the solution manually

How does interaction between parameters lead to constraint violations and goal realization? What goal?

– Planners do not have this knowledge !!– Planners will never have this knowledge !!

We propose that Interaction should be determined a priori, instead of a posteriori

18

The perspective of the control of dynamic situations

“Dynamic situations are situations which are only partly controlled by the human. He/she then has to keep the situation between acceptable limits while managing high risk and/or time pressure”

Examples: Air traffic control, nuclear power plant, aircraft operation, flexible manufacturing system

In the field of Cognitive Ergonomics, Function Allocation is successfully used to design automation

19

Conclusions / statements

The fundamental problems concerning planning support did not change in 30 years

Practice is ahead of academia We don’t fully understand ‘old’ planning, yet

businesses started ‘new’ planning Organization theory (various organizational forms)

related to planning hinders adequate support Algorithms do not pay attention to human problem

solving

20

Conclusions / further research

Balance requirements of tasks, algorithmic possibilities, and human limitations/creativity

Include the user in the development process from the beginning, but:

Not necessarily take the limitations of current humans for granted! Training or even partly replacing human planners is an option as well.

However, many goals of planning/scheduling are often confusing / difficult to formalize!

21

22

Interactive Scheduling Systems;gap between theory and practice

Combinatorial problems are hard to tackle Problems in practice change Interactive systems are rare Operations managers lack understanding of

scheduling systems Cost/benefit ratio hard to prove Human planners don’t have a good grasp of the

implications of their decisions Schedulers are buffered from outside pressure

23

The planner and his tasks

• Macro task division• Organizational setting• Who does what? Order acceptance, stock planning,

scheduling, rescheduling, etc.• Micro task division

• Collecting information• Counting• Checking errors• Evaluating• Problem solving