OW2 Clif Use Case OW2con11, Nov 24-25, Paris

OW2Con 2011, November 23-24, Orange Labs, Paris.

www.ow2.org. 1

Wide-spreading performance testing at Orange with OW2 CLIF:an SOA use case

Bruno Dillenseger, Orange Labs

Among the tens of real CLIF use cases at Orange, WSOI is the web-service oriented infrastructure that runs over 500 web services consumers and providers, and handles about half a billion calls per month for the Orange Group.


www.ow2.org. 2

Testing a service performance and resilience to high traffic

System Under Test

Load injectors :● send requests, wait for replies, measure response times● according to a given scenario defining the workload● for example, emulating the load of a number of real users→ virtual users

resource probes

Execution, control and monitoring of load injectors and probes.

Probes measure usage of arbitrary computing resources

test supervision

load injector 1

load injector 2

load injector n

resource probes


www.ow2.org. 3

CLIF, an outstanding load testing frameworkAdvanced features

● dynamically adjustable number of virtual users● support for tests of any scale

– from one to millions of virtual users– from one to more than 1000 load injectors

● integrated resources monitoring● embedded reporting tool

Versatility● OS-independent (Java 1.5+)● integration to Eclipse● continuous integration (Hudson/Jenkins)● command line (through ant and maven)● custom probes and load injectors


www.ow2.org. 4

Orange's motivation for performance testing is great

As an integrated telecommunication operator, Orange manages a huge variety of technologies

● networks, protocols● equipments● service platforms...

More than 221 millions clients in 42 countries!● quality of service, user experience and user confidence are

key priorities for Orange● performance issues are critical (testing, sizing, capacity

planning)


www.ow2.org. 5

CLIF, an OW2 project lead by OrangeThe CLIF open source project was jointly launched in 2003 by INRIA and Orange in ObjectWeb/OW2

Maintaining CLIF is strategic for Orange● versatility/adaptability to almost all technologies

– HTTP, SOAP, REST, FTP, DHCP, LDAP, DNS, Diameter, Radius, EAP, GBA, GTPP, TR69, SIP, RTP, proprietary protocols...

● much cheaper than specific commercial tools● growing confidence (feedback from the community)● community contributions● research transfer applied to performance testing

– software components, autonomic computing, cloud computing...


www.ow2.org. 6

The WSOI use case: Orange's web-service oriented infrastructure

Enablers & Services

Infrastructure

Consumersbefore WSOI

SMS/MMS, address book,

storage...

identity, log, accounting, billing...

Enablers & Services

Infrastructure

Consumers

Enablers & Services

Infrastructure

Consumers

Enablers & Services

Infrastructure

Consumers

Enablers & Services

Infrastructure

Consumers

Enablers & Services

Infrastructure

Consumers

Enablers & Services

Infrastructure

Consumers

with WSOIEnablers & Services

WSOI

Consumers & Partners

Unified and shared infrastructureBenefits:● cost reduction (common security, access control, scaling, supervision, maintenance)● ready-to-use and no-cost infrastructure for new servicesChallenge:● Performance and availability!

end-user services: personal storage, TV program, music

Enablers & Services

Infrastructure

Consumers

Proliferation of specific infrastructures


www.ow2.org. 7

WSOI performance challenge● Over 500 web services consumers and providers● Half a billion calls per month

juil.-0

9

août-

09

sept.

-09

oct.-0

9

nov.-

09

déc.-

09

janv.-

10

févr.-

10

mars-1

0

avr.-

10

mai-10

juin-

10

juil.-1

0

août-

10

sept.

-10

oct.-1

0

nov.-

10

déc.-

10

janv.-

11

févr.-

11

mars-1

1

avr.-

11

mai-11

juin-

11

juil.-1

1

0

50 000 000

100 000 000

150 000 000

200 000 000

250 000 000

300 000 000

350 000 000

400 000 000

450 000 000

500 000 000

WSOI - Calls by month

IAS RequestsSI RequestsB2B RequestsTotal Requests

Month

Cal

ls (

Mill

ion

)


www.ow2.org. 8

Inside WSOIEnablers & Services

WSOI

Consumers & Partners

Load balancer

XML appliances

Configuration server LDAP directory

Main features:● XML validation● routing● access control● service level management● security● encryption, compression● scalableMostly supported by hardware (XML appliance)


www.ow2.org. 9

WSOI testbed with CLIF

Fake services and enablers

WSOIHTTP injectors

SNMP probe

SOAP requests

system probes(cpu, memory)

Performance qualification of WSOI:● request throughput and response times● WSOI load:

– CPU and memory usage on configuration server and LDAP server– XML appliance load (via SNMP monitoring)

● according to a variety of requests and responses sizes


www.ow2.org. 10

Create a CLIF test project

The WSOI "project" will contain all test definitions, input data and results:

● test plans files (.ctp)– definition of injectors and probes to deploy

● scenarios files (.xis)– virtual users behaviors– load profile (number of active virtual users)

● input data– a SOAP request to replay (captured with the

XML appliance)● raw measures (report)● moving statistics on measures gathered

during test executions (stats)


www.ow2.org. 11

Scenario: import necessary plug-ins

The FileReader plug-in will load the captured SOAP request and let it available to the HttpInjector.


www.ow2.org. 12

Scenario: define virtual users behaviors


www.ow2.org. 13

Scenario: define load profiles

Evolution of the number of active virtual users with behavior B0 occording to time (in seconds).

Note: the number of active virtual users may be set and changed manually also at test execution time.


www.ow2.org. 14

Define your load injectors and probes


www.ow2.org. 15

Run a test

Control of probesand load injectors

Monitoring of probesand load injectors


www.ow2.org. 16

Browse the measuresraw measures fromone test execution

CSV-formatted text filewith raw CPU usage

test planback-up

moving statisticsfrom monitoring

CSV-formatted text filewith moving statistics onrequests throughput,response times and errors


www.ow2.org. 17

Performance analysis and reporting


www.ow2.org. 18

Advanced usage: self-regulated load injection on WSOI

What is the maximum number of virtual users such that the XML appliance load keeps less than 80%?

load injection

system

probes

controller

system under test

probes

invocation

alarms, resource consumption, profiling

alarms, response times, errors

saturation criteria

control (feedback)

load injection policy

CLIF is enhanced with a load controller component monitoring the SNMP probe and

controlling the load injectors


www.ow2.org. 19

Conclusion

CLIF is not just "yet another load injection" software● high power and scalable● versatility

– user interfaces, supported protocols, monitored resources● more advanced features to come

The WSOI use case for Orange● qualification of an XML appliance-based SOA infrastructure● captured SOAP requests replayed with a plain HTTP injector● full system load monitoring, including the XML appliance

3561 CLIF downloads in November 2011 for new 2.0.7 production release

● go to clif.ow2.org

http://clif.ow2.org/


www.ow2.org. 20

Questions time

[email protected]

OW2 Clif Use Case OW2con11, Nov 24-25, Paris

Technology

Transcript of OW2 Clif Use Case OW2con11, Nov 24-25, Paris