Workflow Engine Performance Benchmarking with BenchFlow

VincenzoFermeFacultyofInformaticsUSILugano,Switzerland

WORKFLOW ENGINE PERFORMANCE BENCHMARKING

WITH BENCHFLOW

Vincenzo Ferme

2

The BenchFlow Project

Design and implement the first benchmark to assess and compare the performance of WfMSs that are compliant with Business Process Model and Notation 2.0 standard.

”

“

Vincenzo Ferme

3

What is a Workflow Management System?

WfMS Users

Applications

Application Server

InstanceDatabase

DBMS

Web Service

D

A

B

C

Vincenzo Ferme

4

Many Vendors of BPMN 2.0 WfMSs

https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines

Jan 2011

BPMN 2.0

Jan 2014

BPMN 2.0.2ISO/IEC 19510

Aug 2009

BETABPMN 2.0

Year Number Sum2009 1 12010 5 62011 4 102012 1 112013 8 192014 2 212015 2 232016 0 23

Grand Total 23

Num

ber o

f BPM

N 2

.0 W

fMSs

0

5

10

15

20

25

Year of the First Version Supporting BPMN 2.0 2009 2010 2011 2012 2013 2014 2015 2016

�1

Vincenzo Ferme

5

end-users, vendors, developersWhy do We Need a Benchmark?

Vincenzo Ferme

5


1. How to choose the best WfMS according to the company’s technical requirements?

2. How to choose the best WfMS according to the company’s business process models (workflows)? B

CD

Vincenzo Ferme

5


1. How to choose the best WfMS according to the company’s technical requirements?

2. How to choose the best WfMS according to the company’s business process models (workflows)? B

CD

4. How to identify WfMS’s bottlenecks?

3. How to evaluate performance improvements during WfMS’s development?

Vincenzo Ferme

6

The BenchFlow Benchmarking Process

Workload Model WfMS

ConfigurationsWorkload Metrics

KPIsMeasure

Input/Process/Output Model

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsWorkload Metrics

KPIsMeasure


Workload Mix

80%C

A

B

20%D

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsTest

TypesWorkload Metrics

KPIsMeasure


Workload Mix

80%C

A

B

20%D

Load FunctionsTest Data

Unsaved diagramUnsaved diagram

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsTest


KPIsMeasure


Workload Mix

80%C

A

B

20%D


Containers

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsTest


KPIsMeasure


Workload Mix

80%C

A

B

20%D


Performance Data

Process Instance Duration

Throughput

Containers

Vincenzo Ferme

Vendor

BenchFlow

Provide Benchmarking Methodology

Vendor

BenchFlow

Agree on adding Vendor's WfMS to the

Benchmark

Benchmarking Methodology Agreement Proposal

Signed Agreement

Vendor

BenchFlow

Provide Containerized Distribution of WfMS

Containerized WfMS Request

Containerized WfMS

Vendor

BenchFlow

Verify the Benchmark Results

Results Verification Outcome

Results Verification Request

Vendor

BenchFlow

Provide Draft Benchmark Results

Draft Benchmark ResultsVerified Benchmark Results

Not Valid

Are the Results Valid?

Community

BenchFlow

Publish Benchmark Results Valid

Figure 4: Benchmarking Methodology Choreography

pending on the WfMS’s architecture. The DBMSContainer can refer to an existing publicly availableContainer distributions. The containerized WfMSshould be publicly available (e.g., at the Docker Hubregistry8), or the Benchflow team should be grantedaccess to a private registry used by the vendor. Thesame applies to the Containers’ definition file, i.e., theDockerfile (Turnbull, 2014, ch. 4). While private reg-istries are a solution that can work with vendors ofclosed-source systems, they impact the reproducibil-ity of the results. For each WfMS version to be in-cluded in the benchmark, there must be a default con-figuration, i.e., the configuration in which the WfMSContainers start without modifying any configurationparameters, except the ones required in order to cor-rectly start the WfMS, e.g., the database connection.However, if vendors want to benchmark the perfor-mance of their WfMS with different configurations,for example, the configuration provided to users asa “getting started” configuration, or production-gradeconfigurations for real-life usage, they can also pro-vide different configurations. To do that, the Con-tainers must allow to issue the WfMS configurationsthrough additional environment variables9, and/or byspecifying the volumes to be exposed10 in order to

8https://hub.docker.com/

9https://docs.docker.com/reference/run/

#env-environment-variables

10https://docs.docker.com/

userguide/dockervolumes/

access the configuration files. Exposing the volumesallows to access the files defined inside the Contain-ers, on the host operating system. Precisely, the WESContainer has to, at least, enable the configuration of:1) the used DB, i.e., DB driver, url, username andpassword for connection; 2) the WES itself; and 3) thelogging level of the WES, and the application stacklayers on which it may depend on. Alternatively, in-stead of providing configuration details, the vendorscan provide different Containers for different config-urations. However, even in that case enabling the DBconfiguration is required.

In order to access relevant data, all WfMS Con-tainers have to specify the volumes to access theWfMS log files, and to access all the data useful tosetup the system. For instance, if the WfMS definesexamples as part of the deployment, we may wantto remove those examples by overriding the volumecontaining them. Moreover, the WES Container (orthe DBMS Container) has to create the WfMS’s DBand populate it with data required to run the WfMS.The vendors need to provide authentication configu-ration details of the WfMS components (e.g., the userwith admin privileges for the DBMS). Alternativelythe vendor may simply provide the files needed to cre-ate and populate the DB.

#mount-a-host-directory-as-a-data-volume

7

Methodology Framework

+

Container Based Methodology and Framework

BenchFlow Frameworkarchitecture

InstanceDatabase

DBMSFaban Drivers

ContainersServers

DATA TRANSFORMERSANALYSERS

Performance Metrics

Performance KPIs

WfMS

Test

Exe

cuti

onA

naly

ses

Faban+

Web Service

Minio

harness

MONITORAdapters

CO

LLECT

OR

S

Vincenzo Ferme

Vendor

BenchFlow


Vendor

BenchFlow


Benchmark


Signed Agreement

Vendor

BenchFlow



Containerized WfMS

Vendor

BenchFlow




Vendor

BenchFlow



Not Valid


Community

BenchFlow












7

Methodology Framework

+

Container Based Methodology and Framework


InstanceDatabase

DBMSFaban Drivers

ContainersServers


Performance Metrics

Performance KPIs

WfMS

Test

Exe

cuti

onA

naly

ses

Faban+

Web Service

Minio

harness

MONITORAdapters

CO

LLECT

OR

S

Provides Great Advantages for Automation

and Reproducibility of Results, while

Ensuring Negligible Performance Impact

Vincenzo Ferme

8

[CLOSER ’16]Container-centric Methodology for Benchmarking Workflow Management Systems.

Vendor

BenchFlow


Vendor

BenchFlow


Benchmark


Signed Agreement

Vendor

BenchFlow



Containerized WfMS

Vendor

BenchFlow




Vendor

BenchFlow



Not Valid


Community

BenchFlow












Vincenzo Ferme

9

BenchFlow Benchmarking Methodologyrequirements from the WfMS

Availability of APIs

• Deploy Process

• Start Process Instance

• Users API

• Web Service APIs

• Events APIs

Vincenzo Ferme

9



• Deploy Process


• Users API


• Events APIs

• Workflow & Construct:

• Start Time

• End Time

• [Duration]

Availability of Timing Data

Vincenzo Ferme

9



• Deploy Process


• Users API


• Events APIs

• Workflow & Construct:

• Start Time

• End Time

• [Duration]

Availability of Timing Data

Availability of Execution StateState of the workflow execution. E.g., running, completed, error

Vincenzo Ferme

10


InstanceDatabase

DBMSFaban Drivers

ContainersServers


Performance Metrics

Performance KPIs

WfMS

Test

Exe

cuti

onA

naly

ses

Faban+

Web Service

Minio

harness

benchflow [BPM ’15] [ICPE ’16]

MONITORAdapters

CO

LLECT

OR

S

Vincenzo Ferme

11

BenchFlow FrameworkWfMSs’ specific characteristics

•Manages the Automatic WfMS deployment;

•Provides a “plugin” mechanism to add new WfMSs;

•Performs automatic Process Models deployment;

•Collects Client and Server-side data and metrics;

•Automatically computes performance metrics and statistics.

Vincenzo Ferme

12

Applications of the Methodology and Framework

[CAiSE ’16]Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns.

WfMS

MySQL

3 WfMSsWorkload

sequencePattern

EmptyScript 1

EmptyScript 2

…

0

750

1500

0:00 4:00 10:00

Metrics

• Engine Level

• Process Level

• Environment

Vincenzo Ferme

Micro-Benchmarking with Workflow Patterns

13

research questions

Selected three WfMSs: popular open-source WfMSs actually used in industry

1. What is the impact of individual or a mix of workflow patterns on the performance of each one of the benchmarked BPMN 2.0 WfMSs?

2. Are there performance bottlenecks for the selected WfMSs?

Vincenzo Ferme

14

sequencePattern

EmptyScript 1

EmptyScript 2

Workload Mix

Micro-Benchmarking with Workflow Patternsworkload mix

Sequence Pattern [SEQ]

• N. Russell et al, Workflow Control-Flow Patterns: A Revised View, 2006Pattern Reference:

Vincenzo Ferme

14

sequencePattern

EmptyScript 1

EmptyScript 2

Workload Mix


Sequence Pattern [SEQ]exclusiveChoicePattern

generatenumber

EmptyScript 1

EmptyScript 2case_2

case_1

Exclusive Choice and Simple Merge [EXC]


Vincenzo Ferme

15

arbitaryCyclesPattern

generate1 or 2

EmptyScript 1

EmptyScript 2

i++

i < 10

i >= 10case_1

case_2

Workload Mix


Arbitrary Cycle [CYC]


Vincenzo Ferme

16

Workload Mix


parallelSplitPattern

EmptyScript 1

EmptyScript 2

Parallel Split and Synchronisation [PAR]


Vincenzo Ferme

16

Workload Mix


implicitTerminationPattern

EmptyScript 1

Wait 5Sec

Explicit Termination Pattern [EXT]


EmptyScript 1

EmptyScript 2

Parallel Split and Synchronisation [PAR]


Vincenzo Ferme

17

Micro-Benchmarking with Workflow Patternsworkload mix design decisions

1. Maximise the simplicity of the model expressing the workflow pattern

2. Omit the interactions with external systems by implementing all tasks as script tasks

Vincenzo Ferme

18

Load Functions

Micro-Benchmarking with Workflow Patternsload function

Test Type

Load test

Inst

ance

Pr

oduc

ers

0300600900

12001500

Time (min:sec)

0:000:3

01:0

02:0

03:0

04:0

05:0

06:0

07:0

08:0

010

:00

Example for u=1500

Vincenzo Ferme

19

WfMSConfigurations Micro-Benchmarking with Workflow Patterns

configurations: WfMS environments

WfMS A WfMS B WfMS C

Vincenzo Ferme

19




MySQL MySQL MySQL

MySQL: Community Server 5.6.26

Vincenzo Ferme

19




MySQL MySQL MySQL


O.S.: Ubuntu 14.04.01 Ubuntu 14.04.01 Ubuntu 14.04.01J.V.M.: Oracle Serv. 7u79 Oracle Serv. 7u79 Oracle Serv. 7u79

Vincenzo Ferme

19




MySQL MySQL MySQL



App. Server: Ap. Tomcat 7.0.62 Ap. Tomcat 7.0.62 Wildfly 8.1.0. Final

Vincenzo Ferme

19




MySQL MySQL MySQL




Max Java Heap: 32GB

Vincenzo Ferme

19




MySQL MySQL MySQL




Max Java Heap: 32GB Max DB Connections Number: 100

Vincenzo Ferme

19




MySQL MySQL MySQL


WfMS’s Containers: official ones, if available on DockerHub




Vincenzo Ferme

19




MySQL MySQL MySQL


WfMS’s Containers: official ones, if available on DockerHubWfMS’s Configuration: as suggested by vendor’s documentation




Vincenzo Ferme

20


configurations: WfMS deployment

TEST ENVIRONMENT

CPU64 Cores

@ 1400 MHz

RAM 128 GB

Load Drivers

CPU12 Cores

@ 800 MHz

RAM 64 GB

WfMS

CPU64 Cores

@ 2300 MHz

RAM 128 GB

DBMS

Vincenzo Ferme

20



TEST ENVIRONMENT

CPU64 Cores

@ 1400 MHz

RAM 128 GB

Load Drivers

CPU12 Cores

@ 800 MHz

RAM 64 GB

WfMS

CPU64 Cores

@ 2300 MHz

RAM 128 GB

DBMS

10 Gbit/s 10 Gbit/s

Vincenzo Ferme

20



TEST ENVIRONMENT

CPU64 Cores

@ 1400 MHz

RAM 128 GB

Load Drivers

CPU12 Cores

@ 800 MHz

RAM 64 GB

WfMS

CPU64 Cores

@ 2300 MHz

RAM 128 GB

DBMS

10 Gbit/s 10 Gbit/s

O.S.: Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS

Vincenzo Ferme

20



TEST ENVIRONMENT

CPU64 Cores

@ 1400 MHz

RAM 128 GB

Load Drivers

CPU12 Cores

@ 800 MHz

RAM 64 GB

WfMS

CPU64 Cores

@ 2300 MHz

RAM 128 GB

DBMS

10 Gbit/s 10 Gbit/s

O.S.: Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS Ubuntu 14.04.3 LTS

1.8.2 1.8.2 1.8.2

Vincenzo Ferme

21

Micro-Benchmarking with Workflow Patternsperformed experiments

Individual patterns

1. 1500 (max) concurrent Instance Producers starting each pattern [SEQ, EXC, CYC, EXT, PAR]

Mix of patterns

2. 1500 concurrent Instance Producers starting an equal number of pattern instances (20% mix of [SEQ, EXC, CYC, EXT, PAR])

Vincenzo Ferme

21


Three runs for each execution of the experiments

Individual patterns


Mix of patterns


Vincenzo Ferme

21


Three runs for each execution of the experimentsThe data collected in the first minute of the execution has not been included in the analysis

Individual patterns


Mix of patterns


Vincenzo Ferme

22

Number of Process Instances

Computed Metrics and Statisticsengine level metrics

Throughput

Execution Time

Database Size

WfMS UsersLoad Driver

InstanceDatabase

Application Server

DBMS

Web Service

D

A

BC

…

Vincenzo Ferme

23

individual patterns: throughput (bp/sec)Micro-Benchmarking with Workflow Patterns

SEQ EXC EXT PAR CYC

WfMS A

WfMS B

WfMS C

1456.79 1417.17 1455.68 1433.12 327.99

1447.66 1436.03 1426.35 1429.65 644.02

63.31 49.04 0.38 48.89 13.46

u=600

u=800

u=1500

Vincenzo Ferme

24

Micro-Benchmarking with Workflow Patternsindividual patterns: WfMS C Scalability for SEQ pattern

WfMS C uses a synchronous instantiation API, load drivers must wait for workflows to end

before starting new ones

Vincenzo Ferme

24




Res

pons

e T

ime

0

14

28

500 1000 1500 2000

u Instance Producers

Thr

ough

put

0

40

80

500 1000 1500 2000


Vincenzo Ferme

24




Res

pons

e T

ime

0

14

28

500 1000 1500 2000


Thr

ough

put

0

40

80

500 1000 1500 2000


We have found a bottleneck! (~60 bp/sec)

Vincenzo Ferme

25

Computed Metrics and Statisticsprocess level metrics

Throughput


InstanceDatabase

Application Server

Web Service

D

A

BC

DBMS…

Instance Duration

Per

Proc

ess

D

efini

tion

Vincenzo Ferme

26

Micro-Benchmarking with Workflow Patternsindividual patterns: mean (instance duration time)

Vincenzo Ferme

26


sequencePattern

EmptyScript 1

EmptyScript 2

[SEQ]

Vincenzo Ferme

26

implicitTerminationPattern

EmptyScript 1

Wait 5Sec

[EXT]


Vincenzo Ferme

26



EmptyScript 1

EmptyScript 2

[PAR]

Vincenzo Ferme

26


CYC Instance Producers:

u=600

[CYC]

arbitaryCyclesPattern

generate1 or 2

EmptyScript 1

EmptyScript 2

i++

i < 10

i >= 10case_1

case_2

Vincenzo Ferme

27

Computed Metrics and Statisticsenvironment metrics


InstanceDatabase

Application Server

Web Service

D

A

BC

DBMS

CPU/RAM Usage per Second

IO Operations/Network Usage per Second

…

Vincenzo Ferme

28

Micro-Benchmarking with Workflow Patternsindividual patterns: mean (RAM) and mean (CPU) usage

RAM CPU

Vincenzo Ferme

29


Individual patterns


Mix of patterns


Three runs for each execution of the experimentsThe data collected in the first minute of the execution has not been included in the analysis

Vincenzo Ferme

30

Micro-Benchmarking with Workflow Patternsmix of patterns: duration, RAM, CPU

Vincenzo Ferme

31

Micro-Benchmarking with Workflow Patternssummary

Even though the Executed Workflows are Simple

Vincenzo Ferme

31


There are relevant differences in workflows’ duration and throughput among the WfMSs


Vincenzo Ferme

31



There are relevant differences in resource usage among WfMSs

CPURAM


Vincenzo Ferme

31



One WfMS has bottlenecks

There are relevant differences in resource usage among WfMSs

CPURAM


Highlights

Highlights

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsTest


KPIsMeasure


Workload Mix

80%C

A

B

20%D


Performance Data


Throughput

Containers

Benchmarking Process

Highlights

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsTest


KPIsMeasure


Workload Mix

80%C

A

B

20%D


Performance Data


Throughput

Containers

Benchmarking ProcessVincenzo Ferme

8


Vendor

BenchFlow


Vendor

BenchFlow


Benchmark


Signed Agreement

Vendor

BenchFlow



Containerized WfMS

Vendor

BenchFlow




Vendor

BenchFlow



Not Valid


Community

BenchFlow


Benchmarking Methodology

Highlights

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsTest


KPIsMeasure


Workload Mix

80%C

A

B

20%D


Performance Data


Throughput

Containers


8


Vendor

BenchFlow


Vendor

BenchFlow


Benchmark


Signed Agreement

Vendor

BenchFlow



Containerized WfMS

Vendor

BenchFlow




Vendor

BenchFlow



Not Valid


Community

BenchFlow



Vincenzo Ferme

10


InstanceDatabase

DBMSFaban Drivers

ContainersServers


Performance Metrics

Performance KPIs

WfMS

Test

Exe

cuti

onA

naly

ses

Faban+

Web Service

Minio

harness


MONITORAdapters

CO

LLECT

OR

S

BenchFlow Framework

Highlights

Vincenzo Ferme

6


Workload Model WfMS

ConfigurationsTest


KPIsMeasure


Workload Mix

80%C

A

B

20%D


Performance Data


Throughput

Containers


8


Vendor

BenchFlow


Vendor

BenchFlow


Benchmark


Signed Agreement

Vendor

BenchFlow



Containerized WfMS

Vendor

BenchFlow




Vendor

BenchFlow



Not Valid


Community

BenchFlow



Vincenzo Ferme

10


InstanceDatabase

DBMSFaban Drivers

ContainersServers


Performance Metrics

Performance KPIs

WfMS

Test

Exe

cuti

onA

naly

ses

Faban+

Web Service

Minio

harness


MONITORAdapters

CO

LLECT

OR

S

BenchFlow FrameworkVincenzo Ferme

26

EmptyScript 1

Wait 5Sec

[EXT]


EmptyScript 1

EmptyScript 2

[SEQ] EmptyScript 1

EmptyScript 2

[PAR]

CYC Instance Producers:

u=600

[CYC]

generate1 or 2

EmptyScript 1

EmptyScript 2

i++

i < 10

i >= 10case_1

case_2

Micro-Benchmark with WP

Vincenzo Ferme

33

•We want to characterise the Workload Mix using Real-World process models

• Share your executable BPMN 2.0 process models, even anonymised

Process Models

• We want to characterise the Load Functions using Real-World behaviours

• Share your execution logs, even anonymised

Execution Logs

•We want to add more and more WfMSs to the benchmark

• Contact us for collaboration, and BenchFlow framework support

WfMSs

Call for CollaborationWfMSs, process models, process logs

benchflowbenchflow

[email protected]

http://benchflow.inf.usi.ch

WORKFLOW ENGINE PERFORMANCE BENCHMARKING

WITH BENCHFLOW


http://benchflow.inf.usi.ch

BACKUP SLIDES


Vincenzo Ferme

36

Published Work

[BTW ’15]C. Pautasso, V. Ferme, D. Roller, F. Leymann, and M. Skouradaki. Towards workflow benchmarking: Open research challenges. In Proc. of the 16th conference on Database Systems for Business, Technology, and Web, BTW 2015, pages 331–350, 2015.

[SSP ’14]M. Skouradaki, D. H. Roller, F. Leymann, V. Ferme, and C. Pautasso. Technical open challenges on benchmarking workflow management systems. In Proc. of the 2014 Symposium on Software Performance, SSP 2014, pages 105–112, 2014.

[ICPE ’15]M. Skouradaki, D. H. Roller, L. Frank, V. Ferme, and C. Pautasso. On the Road to Benchmarking BPMN 2.0 Workflow Engines. In Proc. of the 6th ACM/SPEC International Conference on Performance Engineering, ICPE ’15, pages 301–304, 2015.

Vincenzo Ferme

37

Published Work

[CLOSER ’15]M. Skouradaki, V. Ferme, F. Leymann, C. Pautasso, and D. H. Roller. “BPELanon”: Protect business processes on the cloud. In Proc. of the 5th International Conference on Cloud Computing and Service Science, CLOSER 2015. SciTePress, 2015.

[SOSE ’15]M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models. In Proc. of the 9th International IEEE Symposium on Service-Oriented System Engineering, SOSE 2015, 2015.

[BPM ’15]V. Ferme, A. Ivanchikj, C. Pautasso. A Framework for Benchmarking BPMN 2.0 Workflow Management Systems. In Proc. of the 13th International Conference on Business Process Management, BPM ’15, pages 251-259, 2015.

Vincenzo Ferme

38

Published Work

[BPMD ’15]A. Ivanchikj, V. Ferme, C. Pautasso. BPMeter: Web Service and Application for Static Analysis of BPMN 2.0 Collections. In Proc. of the 13th International Conference on Business Process Management [Demo], BPM ’15, pages 30-34, 2015.

[ICPE ’16]V. Ferme, and C. Pautasso. Integrating Faban with Docker for Performance Benchmarking. In Proc. of the 7th ACM/SPEC International Conference on Performance Engineering, ICPE ’16, 2016.

[CLOSER ’16]V. Ferme, A. Ivanchikj, C. Pautasso., M. Skouradaki, F. Leymann. A Container-centric Methodology for Benchmarking Workflow Management Systems. In Proc. of the 6th International Conference on Cloud Computing and Service Science, CLOSER 2016. SciTePress, 2016.

Vincenzo Ferme

[CAiSE ’16]M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns . In Proc. of the 28th International Conference on Advanced Information Systems Engineering, CAiSE ’16, 2016.

39

Published Work

[ICWE ’16]C. Jürgen, V. Ferme, H.C. Gall. Using Docker Containers to Improve Reproducibility in Software and Web Engineering Research. In Proc. of the 16th International Conference on Web Engineering, 2016.

[ICWS ’16]M. Skouradaki, V. Andrikopoulos, F. Leymann. Representative BPMN 2.0 Process Model Generation from Recurring Structures. In Proc. of the 23rd IEEE International Conference on Web Services, ICWS '16, 2016.

Vincenzo Ferme

40

Published Work

[SummerSOC ’16]M. Skouradaki, T. Azad, U. Breitenbücher, O. Kopp, F. Leymann. A Decision Support System for the Performance Benchmarking of Workflow Management Systems. In Proc. of the 10th Symposium and Summer School On Service-Oriented Computing, SummerSOC '16, 2016.

[OTM ’16]M. Skouradaki, V. Andrikopoulos, O. Kopp, F. Leymann. RoSE: Reoccurring Structures Detection in BPMN 2.0 Process Models Collections. In Proc. of On the Move to Meaningful Internet Systems Conference, OTM '16, 2016. (to appear)

[BPM Forum ’16]V. Ferme, A. Ivanchikj, C. Pautasso. Estimating the Cost for Executing Business Processes in the Cloud. In Proc. of the 14th International Conference on Business Process Management, BPM Forum ’16, 2016. (to appear)

Vincenzo Ferme

41

Docker Performance

[IBM ’14]W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An updated performance comparison of virtual machines and Linux containers. IBM Research Report, 2014.

Although containers themselves have almost no overhead, Docker is not without performance gotchas. Docker volumes have noticeably better performance than files stored in AUFS. Docker’s NAT also introduces overhead for workloads with high packet rates. These features represent a tradeoff between ease of management and performance and should be considered on a case-by-case basis.

”

“Our results show that containers result in equal or better performance than VMs in almost all cases.

“”

BenchFlow Configures Docker for Performance by Default

Vincenzo Ferme

42

Benchmarking Requirements

• Relevant

• Representative

• Portable

• Scalable

• Simple

• Repeatable

• Vendor-neutral

• Accessible

• Efficient

• Affordable

• K. Huppler, The art of building a good benchmark, 2009 • J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993 • S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A

challenge to software engineering, 2003

Vincenzo Ferme

43

BenchFlow Benchmarking MethodologyWfMS

Configurations

requirements from the WfMS

NON-CORECORE

Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

Vincenzo Ferme

43


Configurations


NON-CORECORE

WfMS

Initialisation APIs

Deploy Process

Start Process Instance


Vincenzo Ferme

43


Configurations


NON-CORECORE

WfMS

Initialisation APIs

Deploy Process


+ WfMS

Claim Task Complete Task

User APIs

Create User Create Group

Pending User Tasks


Vincenzo Ferme

43


Configurations


NON-CORECORE

WfMS

Initialisation APIs

Deploy Process


+ WfMS


User APIs


Pending User Tasks


Invoke WS

Web Service APIs

Vincenzo Ferme

43


Configurations


NON-CORECORE

WfMS

Initialisation APIs

Deploy Process


Event APIs

Issue Event

WfMS

Pending Event Tasks

+ WfMS


User APIs


Pending User Tasks


Invoke WS

Web Service APIs

Vincenzo Ferme

44


Configurations

requirements from the WfMSTable 1: Summary of Core and Non-core APIs to be imple-mented by the WfMS

Functionality Min Response Data

Cor

eAP

Is

InitialisationAPIs

Deploy a process Deployed process IDStart a process instance Process instance ID

Non

-cor

eAP

Is

User APIs Create a user User IDCreate a group of users User group IDAccess pending tasks Pending tasks IDsClaim a task*Complete a task

Event APIs Access pending events Pending events IDsIssue events

Web serviceAPIs

Map tasks to Webservice endpoints

*Optional depending on the WfMS implementation

2.1.2 System Under Test

The SUT refers to the system that is being tested forperformance. In our case a WfMS which, accord-ing to the WfMC (Hollingsworth, 1995), includes theWorkflow Enactment Service (WES) and any envi-ronments and systems required for its proper func-tioning, e.g., application servers, virtual machines(e.g., Java Virtual Machine) and DBMSs. The WESis a complex middleware component which, in addi-tion to handling the execution of the process models,interacts with users, external applications, and Webservices. This kind of SUT offers a large set of con-figuration options and deployment alternatives. TheDBMS configuration as well as the WES configura-tions (e.g., the granularity level of history logging,the DB connection options, the order of asynchronousjobs acquisition), may affect its performance. More-over, WfMSs often offer a wide range of deploymentalternatives (e.g., standalone, in a clustered deploy-ment, behind a load balancer in an elastic cloud in-frastructure) and target different versions of applica-tion server stacks (e.g., Apache Tomcat, JBoss).

To add a WfMS in the benchmark, we require theavailability of certain Core and Non-core APIs. Asummary of the same is presented in Table 1.

The Core APIs are Initialisation APIs necessaryfor automatic issuing of the simplest workload to theSUT in order to test SUT’s performance. They in-clude: 1) deploy a process and return as a response anidentifier of the deployed process (PdID); and 2) starta process instance by using the PdID and return as aresponse the new instance identifier (PiID).

Depending on the execution language of theWfMS and the constructs that it supports, other Non-

core APIs might be necessary for testing more com-plex workloads. For instance, if we are targetingBPMN 2.0 WfMSs we might also require the follow-

ing APIs:For applying workloads involving human tasks,

the following User APIs are necessary: 1) createa user and return the identifier of the created user(UsID); 2) create a group of users, return the cre-ated group identifier (UgID), and enable adding usersby using their UsIDs; 3) pending user/manual tasks:access all the pending user/manual task instances ofa given user/manual task identified by its id (Jordanand Evdemon, 2011, sec. 8.2.1) as specified in themodel serialization. We want to obtain all the pend-ing tasks with the given id of all the process instances(PiIDs) of a given deployed process (PdID). The APIhas to respond with data, enabling the creation of acollection that maps the process instances to the listof their pending tasks <PiID, UtIDs> and <PiID,MtIDs>; 4) claim a user/manual task identified byUtID/MtID, if tasks are not automatically assigned bythe WfMS; and 5) complete a user/manual task iden-tified by UtID/MtID by submitting the data requiredto complete the task.

To issue a workload containing process mod-els with catching external events, the followingEvent APIs are necessary: 1) pending catchingevents/receive tasks: access the pending catchingevent/receive task instances of a given event/taskidentified by its id (Jordan and Evdemon, 2011, sec.8.2.1) specified in the model serialization. We wantto obtain all the pending catching events/receive taskswith the given id of all the process instances (Pi-IDs) of a given deployed process (PdID). The APIhas to respond with data enabling the creation of acollection that maps the process instances to the listof their pending catching events/receive tasks <PiID,CeIDs> and <PiID, RtIDs>; and 2) issue an event toa pending catching event/receive task identified by us-ing CeID/RtID. We require the APIs to accept the datanecessary to correlate the issued event to the correctprocess instance, e.g., a correlation key.

Finally, to be able to issue a workload defining in-teraction with Web services and/or containing throw-ing events, the WfMS has to support a binding mech-anism to map each Web service task/throwing eventto the corresponding Web service/throwing event end-point. The WfMS should preferably allow to specifythe mapping in the serialized version of the model,so that the binding can be added before deploying theprocess.

Since many WfMSs are offered as a service, it issafe to assume that many WfMSs expose, what wecall, the Core APIs. In our experience with systemswe have evaluated so far (e.g., Activiti, Bonita BPM,Camunda, Imixs Workflow, jBPM), they support notonly the core APIs, but also the non-core APIs. The


Vincenzo Ferme

45

BenchFlow Frameworksystem under test

Docker Engine

Vincenzo Ferme

45


Docker EngineContainers

Vincenzo Ferme

45


Docker Engine

Docker Machine

prov

ides

Containers

Vincenzo Ferme

45


Docker Swarm

Docker Engine

Docker Machine

prov

ides

Containers

Vincenzo Ferme

45


Docker Swarm

Docker Engine

Docker Machine

prov

ides

manages

Containers

Servers

Vincenzo Ferme

45


Docker Swarm

Docker Engine

Docker Compose

Docker Machine

prov

ides

manages

Containers

Servers

Vincenzo Ferme

45


Docker Swarm

Docker Engine

Docker Compose

Docker Machine

prov

ides

manages

Containers

Servers

SUT’s Deployment Conf.

deploys

Vincenzo Ferme

46

Server-side Data and Metrics Collection

WfMS

Start Workflow

asynchronous execution of workflows

Vincenzo Ferme

Users

D

A

BC

Start

Application Server

Web Service

Load Driver

DBMS

WfMS

InstanceDatabase

46


WfMS

Start Workflow


Vincenzo Ferme

Users

D

A

BC

Start

Application Server

Web Service

Load Driver

DBMS

WfMS

InstanceDatabase

46


WfMS

Start Workflow

End


Vincenzo Ferme

47

Server-side Data and Metrics Collectionmonitors

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

Vincenzo Ferme

47


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

MONITOR

Vincenzo Ferme

48


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

• CPU usage• Database state

Examples of Monitors:Monitors’ Characteristics:• RESTful services• Lightweight (written in Go)• As less invasive on the SUT as possible

Vincenzo Ferme

48


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

CPU Monitor

DB Monitor

• CPU usage• Database state

Examples of Monitors:Monitors’ Characteristics:• RESTful services• Lightweight (written in Go)• As less invasive on the SUT as possible

Vincenzo Ferme

49

Server-side Data and Metrics Collectioncollect data

DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

Vincenzo Ferme

49


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

MinioInstanceDatabase

Ana

lyse

sC

OLLEC

TO

RS

Vincenzo Ferme

50


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

• Container’s Stats (e.g., CPU usage)• Database dump• Applications Logs

Examples of Collectors:Collectors’ Characteristics:

• RESTful services• Lightweight (written in Go)• Two types: online and offline• Buffer data locally

Vincenzo Ferme

50


DBMSFaban Drivers

ContainersServers

harnessWfMS

Test

Exe

cuti

on Web Service

Stats Collector

DB Collector

• Container’s Stats (e.g., CPU usage)• Database dump• Applications Logs

Examples of Collectors:Collectors’ Characteristics:

• RESTful services• Lightweight (written in Go)• Two types: online and offline• Buffer data locally

Vincenzo Ferme

51

Test

Exe

cuti

on

A high-throughput distributed messaging system

MinioInstance

DatabaseAna

lyse

s

DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Stats Collector

DB Collector

Synchronisation with the Analysesschedule analyses

Vincenzo Ferme

51

Test

Exe

cuti

on


MinioInstance

DatabaseAna

lyse

s

DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Stats Collector

DB Collector

CO

LLECT


Vincenzo Ferme

51

Test

Exe

cuti

on


MinioInstance

DatabaseAna

lyse

s

DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Stats Collector

DB Collector

CO

LLECT

Publish


Vincenzo Ferme

51

Test

Exe

cuti

on


MinioInstance

DatabaseAna

lyse

s

DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Stats Collector

DB Collector

CO

LLECT

PublishSu

bscr

ibe


Vincenzo Ferme

51

Test

Exe

cuti

on


MinioInstance

DatabaseAna

lyse

s

DBMSFaban Drivers

ContainersServers

harnessWfMS

Web Service

Stats Collector

DB Collector

CO

LLECT

PublishSu

bscr

ibe

Read


Vincenzo Ferme

52

Performance Metrics and KPIsamount of data

REALISTIC

DATA

Number of

Repetitions

1

1053

Vincenzo Ferme

52

Performance Metrics and KPIsamount of data

REALISTIC

DATA

Number of

Repetitions

1

1053

Number of WfMSs

1

Vincenzo Ferme

53

Micro-Benchmarking with Workflow Patternsmix of patterns: throughput (bp/sec)

MIX

WfMS A

WfMS B

WfMS C

1061.27

1402.33

1.78

Vincenzo Ferme

54

Computed Metrics and Statisticsfeature level metrics


InstanceDatabase

Application Server

Web Service

D

A

BC

DBMS

Throughput

…

Instance Duration

Per

Con

stru

ct

(typ

e, n

ame)

Vincenzo Ferme

55

Computed Metrics and Statisticsinteractions metrics

Response Time

Latency

…


InstanceDatabase

Application Server

DBMS

Web Service

D

A

BC

Vincenzo Ferme

56

Computed Metrics and Statisticsdescriptive and homogeneity statistics

• Descriptive Statistics (e.g., Mean, Confidence Interval, Standard Deviation, Percentiles, Quartiles, …)

• Coefficient of Variation across trials

• Levene Test on trials

Workflow Engine Performance Benchmarking with BenchFlow

Technology

Transcript of Workflow Engine Performance Benchmarking with BenchFlow