Moe wynn caise13 presentation

PROFILING EVENT LOGS TO CONFIGURE

RISK INDICATORS FOR PROCESS DELAYS

25th International Conference on Advanced Information Systems Engineering,

Valencia, Spain

21 June 2013

Anastasiia Pika, Wil M. P. van der Aalst, Colin J. Fidge,

Arthur H. M. ter Hofstede, and Moe T. Wynn

Queensland University of Technology & Eindhoven University of Technology

Presenter: Moe T. Wynn

Risk-aware Business Process Management

2

“It will investigate, evaluate and enhance current approaches for the identification, analysis, evaluation, treatment, and overall management of risk as it relates to business processes”

BPM lifecycle

ARC Discovery Grant Project 2011-2013. Risk-aware business process management

ISO 31000:9000

Risk Management Process

3

Process-related risk is one that threatens the achievement of one or

more process goals

A negative effect in terms of timeliness, cost, or quality of the

outcome.

Is caused by any combination of process design, resource

behaviour, or case-data.

E.g., Breaching SLA agreements in terms of completion times,

over-running agreed budgets, producing low-quality outputs

.

International Organization for Standardization. Risk management: vocabulary = Management

du risque: vocabulaire (ISO guide 73). Geneva, 2009.

Process-related risk

Risk - “effect of uncertainty on objectives” where “an effect is a

deviation from the expected - positive and/or negative.”

Research Scope

4

Objective: Develop techniques for identification of process-

related risks

Time-based process related risk: Case Delays

Question: Can we identify the risk of case delays by

analyzing the behaviour of a process?

<a, b, c, d, e>

<a, b, c, d, e>

<a, b, c, c, c, c, d, e>

<a, b, c, d, e>

<a, b, c, d, e>

When an activity is repeated multiple

times, the likelihood of a case delay is high.

Starting point: exploiting event log data

5

Modern organisations automate their business processes and processes’ execution data is usually recorded in event logs.

Process mining provides techniques and tools that help to extract knowledge about processes from event logs.

Research Approach

6

Goal: develop a method that can identify the risk of delay for cases with a high degree of precision

This idea was presented at BPI 2012 workshop [14]

Define Process Risk Indicators

(PRIs) Configure PRIs

Identify the presence of PRI instances in a current case

Step 1: Define Process Risk Indicators

7

Process Risk Indicator (PRI) - a pattern observable in an event log whose presence

indicates a higher likelihood of some process-related risk.

Activity-based PRIs

PRI3: Multiple activity repetitions PRI4: Presence of a “risky” activity

PRI1: Atypical activity execution time PRI2: Atypical waiting time

PRI 6: Atypical sub-process duration

Activity Resource

A R1

B R23

C R12

D R11

E R5

F R4

- -

Process Risk Indicators (Resource-based)

8

Resource-based PRIs

PRI5: Multiple resource involvement PRI7: High resource workload

PRI8: Use of a “risky” resource

Activity Resource

A R1

B R1

C R1

D R1

E R5

F R4

- -

Activity Resource

A R1

B R1

C R111

D R1

E R5

F R4

- -

PRI instantiation from logs: approach in [14]

9

“Sample standard deviations” approach for outlier detection:

Cut-off thresholds for a PRI:

Limitations:

Assumption of a normal distribution

Assumption that any atypical behavior is “risky”

E.g., a large variation in execution time of a short activity has the same impact on case delay predictions as those of a long activity

Results:

Indicators can predict case delays but obtained a high level of false positive predictions

cases that are predicted to be late but in the end are not

Step 2: Configure Process Risk Indicators

10

Motivation: calibrate PRIs so that process semantics are considered

How: using information about the known outcomes from cases in the

past (whether they are delayed or completed on time)

Learn the threshold values for PRIs for a desired precision level

Input parameter: precision level – 80%, 85%, 90%

Input parameter: a log (training set)

Example of configuring PRI 1:

“Atypical activity duration” 11

If the duration of activity A is more than t days there is a high risk of the case delay.

10? 20?

??

To define t:

2. Calculate for each value in C precision of delay predictions in the training set: • If activity A was executed for more than 12 days 60% of cases were delayed

• If activity A was executed for more than 16 days 90% of cases were delayed …

3. Assign t the smallest value from C that allows predicting delays with a

desired degree of precision. • t = 16 if we would like 90% precision level

10 12 14

18 16

20

…

C

1. Define a pool of candidates C

including values:

Example of configuring PRI 5:

“Multiple resource involvement” 12

If more than t resources are involved in a case there is a high risk of the case delay.

5? 8?

??

To define t:

2. Calculate for each value in C precision of delay predictions in the training

set: • If more than 7 resources were involved 80% of cases were delayed

• If more than 8 resources were involved 95% of cases were delayed …

3. Assign t the smallest value from C that allows predicting delays with a

desired degree of precision. • t = 8

3 4 7

8 5

9

…

C

1. Define a pool of candidates C

including values:

Configuring other PRIs 13

Activity-based PRIs: PRI 1, 2, 3, 6

Resource-based PRIs: PRI 5, 7

PRI 4: Presence of a risky activity

we check if there exists an activity that is executed mainly in

delayed cases

PRI 8: Use of a risky resource

we check for each pair “activity-resource" if some resource's

involvement in the execution of an activity mainly occurs in

delayed cases

Step 3: Identify the presence of PRI

instances in a current case

14

Input parameter: log (test set)

Compare the values for a current case against the thresholds

of PRIs

Record a likelihood of a case delay (0 or 1)

if the number is higher than the value of the learned threshold t

if a ‘risky’ activity is present in the current case

If an activity is assigned to a ‘risky’ resource

Implementation within the ProM framework

15

Event log

ProM plug-in

1. Inputs: expected case duration

2. Learn cut-off thresholds for PRIs and Identify the presence of PRIs in cases

3. If any of the PRIs is identified in a case, we consider that there is a risk of delay

4. Compare predicted values with the real case durations

Case ID PRI 1 PRI 2 PRI 3 PRI 4 PRI 5 PRI 6 PRI 7 PRI 8 Risk

1000 1 0 1 0 1 1 1 1 1 102 0 0 0 0 0 0 0 0 0

103 0 0 1 0 0 0 0 0 1 106 0 0 0 0 0 0 0 0 0

305 0 0 0 0 0 0 0 0 0

554 0 0 0 0 0 0 0 0 0

Delayed

activity

Multiple

resources

Activity

repetitions

Validation with real event logs Experimental setup

16

Hold-out cross-validation training and test sets (75:25)

“Random” split and “Time” split (4:2 months)

Evaluation of precision and recall

Precision: the fraction of cases predicted to be delayed that are actually

delayed

Recall: the fraction of delayed cases that can be successfully predicted

against the actually delayed cases

Data pre-processing:

Completed cases

Recent data

Separating cases that are executed in different contexts (e.g., different

departments)

Validation with real event logs Data sets

17

Six Data sets from Suncorp, a large Australian insurance company

Represent insurance claim processes from different organisational units

Properties of data set A

Properties of data sets B1-B5

Validation with real event logs Results. Data set A. “Random” split experiment.

19

Legend:

• 95%, 90%, 80% - desired precision levels

• TP – True Positives (cases predicted

correctly as delayed)

• FP – False Positives (cases predicted to be

delayed but are not delayed)

• FN – False Negatives (delayed cases that

are not predicted to be delayed)

• TN – True Negatives (in time cases that

are also not predicted to be delayed)

• PRI 1: Atypical activity execution time

• PRI 2: Atypical waiting time

• PRI 3: Multiple activity repetitions

• PRI 4: Presence of a “risky” activity

• PRI 5: Multiple resource involvement

• PRI 6: Atypical sub-process duration

• PRI 7: High resource workload

• PRI 8: Use of a “risky” resource

Validation with real event logs Results. Data set A, “Time” split experiment.

20

Legend:

• 95%, 90%, 80% - desired precision levels

• TP – True Positives (cases predicted correctly as

delayed)

• FP – False Positives (cases predicted to be

delayed but are not delayed)

• FN – False Negatives (delayed cases that are not

predicted to be delayed)

• TN – True Negatives (in time cases that are also

not predicted to be delayed)




• PRI 4: Presence of a “risky” activity

• PRI 5: Multiple resource involvement



• PRI 8: Use of a “risky” resource

Data set A. “Random” split experiment (without

configuration) 22

Moment of delay prediction: motivation

23

Predicting delays early during a case’s execution is a

highly desirable capability

Early risk detection enables risk mitigation:

Risk elimination (e.g. reallocation of an activity to other

resource)

Reduction of impact (e.g. adding additional resources in a

case to decrease extent of delay)

Moment of delay prediction

Data set A, Random split, 90% precision level 24

x: The number of days since the beginning of a case when the risk of the case delay

was discovered.

y: The cumulative number of delay predictions at a certain point in time

Observations from the experiments

25

• Good predictors in all data sets:




• Good predictors in some data sets:


• PRI 4: Presence of a ‘risky’ activity


• PRI 8: Use of a ‘risky’ resource

• Early predictions:

• PRI 4: Presence of a ‘risky’ activity


• PRI 8: Use of a ‘risky’ resource

• Limitations of the data:

• High process variability in data

sets B1-B5

• No complete information about

resource workload

• Limitations of the approach:

• Assumption that a process is in a

steady state

• External context is not

considered

Conclusions and Future work

26

A method for predicting case delays with a high degree of precision

Utilise eight process risk indicators

Calibrate the threshold values for risk indicators using log data

Predict the likelihood of case delays using current case and log data

Experiments showed that this approach

decreases the level of false positive alerts,

significantly improves the precision of case delay predictions,

can predict case delays before a certain deadline

Future work:

Investigating the relation between PRIs and the extent of the expected delay

Alternative approaches: neural networks, decision trees

Applying the technique to other types of risks (e.g., budget overrun or low-quality output)

PROFILING EVENT LOGS TO CONFIGURE

RISK INDICATORS FOR PROCESS DELAYS

Thank You! Questions?

Email: [email protected]

Anastasiia Pika, Wil M. P. van der Aalst, Colin J. Fidge,

Arthur H. M. ter Hofstede, and Moe T. Wynn

Moe wynn caise13 presentation

Business

Transcript of Moe wynn caise13 presentation