Openbet - How we ensure the stability of our platform

HOW TO ENSURE THESTABILITY OFAN ECOMMERCEPLATFORMAT TIMES OF PEAK DEMAND

Players • Performance • Partnership

Written by Jon Bramley

HOW TO ENSURE THESTABILITY OFAN ECOMMERCEPLATFORMAT TIMES OF PEAK DEMAND

TABLE OF CONTENTS03 INTRODUCTION04 WHAT IS A PEAK?05 SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM07 HOW DO WE MANAGE PEAK LOADS?08 EXPECTED PEAKS10 POSTMORTEM11 AN OMNI-CHANNEL PLATFORM12 SUMMARY13 WHO ARE OPENBET?

Within the world of internet betting,another bookmaker is just a click awayso performance and stability areparamount to a successful day.

”“

3

INTRODUCTIONThe world of sportsbetting goes incycles driven by the major sporting events of the year. Bookmakers want to use these events to maximise both their revenues and customer acquisitionopportunities.

This means releasing new features,running new promotions, launching dedicated microsites, etc.

So not only is the platform underincreased customer demand due to the natural draw of the event but also due to the effects of the increased marketing.

This means being prepared for both new levels of load on the platform and ensuring consistent high performance and stability throughout new peaks andalongside new, potentially disruptive, functionality.

In this paper we will explore different mechanisms and tools that can be used to ensure success and stability acrossthese high profile, peak days.

4

WHAT IS A PEAK?

Grand National 2015for OpenBet

Black Friday 2015for Amazon

Any system can have periods or days of unusual traffic. In the sportsbetting world we are used to ‘peaky traffic’ due to the pattern of betting naturallybeing weighted towards specific events and increasing just before the start as everyone gets their last-minute bets in.

Saturday afternoon just before thePremiership matches all kick off istraditionally a heavy betting period, for example.

However, to bring some real perspective to the challenge of peak loads, the most demanding day in the UK bookmakers’calendar is the UK Grand National, held each April.

This national institution sees millions of otherwise non-betting peoplehaving their annual punt on the horse race and puts a load on the system of up to 400% of a regular Saturday.

49,000ONLINE BETS PER MINUTE

15.3 MillionONLINE BETS

5,160ITEMS SOLD PER MINUTE

7.4 MillionITEMS SOLD

vs









5

SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORMAlthough a betting website could be thought of a just a specialisede-commerce platform, it actually has a number of characteristics thatdifferentiate it and increases complexity.

Shifting product catalogue

A large scale bookmaker will have around 4 million active betting opportunities live at any one time - but these opportunities do not live long and the entire catalogue will be replaced with another set incrementally over a period of just a few weeks. Within that time the price of each item will change many times. Over the course of a 90 minute football match, the 4,000 betting opportunities available will have over 35,000 price changes applied. And that’s just for one match - we will have over 200 events in-play simultaneously and many thousands moreavailable for pre-match betting.

Complicated purchase process

The purchase process in sports betting involves:

• Validating the player’s account and their balance

• Validating that the player has not exceeded their betting limits

• Validating that the betting opportunities selected have not exceeded their risk limits

• Validating that the betting opportunities can be combined in the chosen way

• Validating that the opportunities and requested prices are still valid

• Recalculating liability for the bookmaker once the bet is placed

6

SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM

0

20000

40000

60000

80000

100000

120000

140000

160000

Total updates

FOOTBALL CRICKETTENNIS

43688

142577

30313

Settlement and payouts

The biggest difference between betting and a regular ecommerce platform is, of course, what the player is buying: a bet. At some later point, the bet needs to be settled against the bet result andpotentially pay the player their winnings.

The main consideration here is that the settlement process itself is computationally intensive and also needs to happen as soon as possible after the result is available, both give the player a good experience, but also to maximise the recycling of winnings into another bet.

Per Event Football Tennis Cricket

Number of marketse.g. First goal scorer 1008 830 3868

Number of market updatese.g. activation/suspension 8163 6693 39551

Number of selectionse.g. Arsenal to win 2-0 4167 2353 11980

Number of selection updatese.g. odds change suspension 35520 23602 102810

Total updates 43688 30313 142577

7

HOW DO WE MANAGE PEAK LOADS?Due to the cyclical and peaky nature of sportsbetting our platform is already built to manage these types of load.

Constant development of the system means that performance needs to be constantly monitored.

Each system is looked after by a dedicated lead who understands thatsystem’s ‘normal’ and works closely with operators to manage network and hardware growth alongside the system. • We make heavy use of caching even for durations of only a couple of

seconds to smooth out the large number of data requests and changing prices.

• We use push technology to publish price and status updates directly to connected users.

• We cache HTML snippets to help build dynamic pages.

• Our in-house Application Server is built to scale both horizontally and vertically and can dynamically use resources as needed.

• Functionality can be split across servers to balance load and a server

can be brought online in seconds and use our caching layer to manage unexpected peaks.

• No player state is held at the Application Server level so traffic can be managed by any server.

• Active monitoring of software, hardware and database usage catches

issues before they affect players.

Total bets per hour

242322212019181716151413121110987654321

8

It all starts months before, where a working group of technicalexperts review the previous year’s performance and lessons learned. Milestones are agreed alongside a readiness checklist that has evolved over the years

01

02 Closer to the event, a feature freeze is implemented that allows the team to focus on performance and stability. Enhanced load testing is started to spot weak areas. This load testing is not trivial to prepare as it must predict the levels of activity across the whole system: fromregistrations, to payments, casino activity, systemtransfers, bet placement and bet settlement. All of which place unique stresses on the system and, in particular, the centralised player account.

03 About a month before Grand National another racing festival - Cheltenham - is a useful yardstick for the preparedness of the system and allows further performance and stability changes to be made if the post-event analysis shows it is necessary.

BeforehandEXPECTED PEAKSFor those periods when we do expect very high peaks - like the Grand National - then we are able to plan and manage these days differently. 04During the final few weeks, load testing continues

and bookmaker marketing efforts are put into action.

THEGRAND

NATIONAL

63%OF REMOTE

BETS PLACED ON SATURDAY

ACCOUNTING FOR8MBETS ON MOBILE

1/2 MILLIONNEW ACCOUNTS

24 HOURSCREATED IN

11 BillionACCOUNT TRANSACTIONS

1 BillionSPORTS BETS

4 MillionCONCURRENT BETTING OPPORTUNTIES

AT ANY TIME ON THE PLATFORM

32,000BETS PER MINUTE AT OUR PEAK

2014 IN NUMBERS:

63%OF REMOTE




24 HOURSCREATED IN






2014 IN NUMBERS:

63%OF REMOTE




24 HOURSCREATED IN






2014 IN NUMBERS:

63%OF REMOTE




24 HOURSCREATED IN






2014 IN NUMBERS:

63%OF REMOTE




24 HOURSCREATED IN






2014 IN NUMBERS:

63%OF REMOTE




24 HOURSCREATED IN






2014 IN NUMBERS:

63%OF REMOTE




24 HOURSCREATED IN






2014 IN NUMBERS:

63%OF REMOTE




24 HOURSCREATED IN






2014 IN NUMBERS:

9

On The DayEXPECTED PEAKSOn the day itself, communication is vital. Each operator team is present and locate themselves in a single area of the office to make coordination easier.

Slack is used for instant messaging with two dedicated channels: one for general communication across teams and another for the escalation of incidents. Account managers and the executive team are subscribed to this second channel.

In the case of issues, communication to all other teams is key so they can be prepared for additional traffic (players moving to a site that is available) and to look out for similar issues. Experts jump in to analyse the problem and get it fixed as quickly as possible and identify the cause in case of knock-on effects to other operators.

Throughout the day activity and load is constantly recorded for later analysis and to help plan for the following year.

• Registration, payments and bet placement are the key focus areas for stability early in the day but this shifts to bet settlement after the events leading up to the big races to ensure players have their winnings returned to them for further bets they’d like to place.

Settlement can be throttled by the amount of system resources dedicated to it but also can have prioritisation applied based on specific events, markets or even betting selection and by prioritising winning bets over non-winnings ones.

The winning customer experience is key and postponing the settlement of non-winning bets to a quieter period later in the day saves vital system resources for continued bet placement.

• Potential Bottlenecks caused by the wide number of third party systems our platform integrates with (payments, verifications, system transfers) are monitored carefully with timeouts kept as low as possible to prevent execution threads getting locked up.

• As pressure builds, there is a list of non-essential activity and functionality that can be disabled to ensure the key functions of registration, login, and bet stay functional.

• Active monitoring of the load on each part of the system is used to balance functionality across servers more evenly and increase capacity where the models didn’t predict load accurately.

15.3 MillionONLINE SPORTS BETS


60 MillionONLINE ACCOUNT TRANSACTIONS

74% BetsPLACED VIA MOBILE CHANNEL

10

POSTMORTEM After the event

• Analyse any major issues, their route cause, immediate resolution and if any longer term solution is required

• Analyse performance versus load for bottlenecks and hot points, and determine solutions. Add these to the backlog with priority based on the severity

• Analyse the load patterns to feed into updated load test profiles

• Share findings across teams and add shared learnings to the backlog

• Carry out a retrospective for the day to take forward suggestions into next year 25

45

35

55

65

2013 20152014

Year

Acc

ou

nt

Tran

sact

ion

s (m

illio

ns)

2016

32.6

52.9

43.3

60.5

11

AN OMNI-CHANNEL PLATFORMSportsbetting has increasingly andrapidly become a mobile-first activity. The advent of the smartphone has materially changed the platform usage profile of online betting such that it is now the majority of online bets that are taken via mobile. For some operators over 80% of bets are taken via mobile channels, but even the annual average across all operators is over 70%.

OpenBet’s omni-channel platformseparates the functionality of the sportsbook from the player front-end and so it does not matter where the activity is taking place - the samerobust back-end processes all bets and the same functionality is available to all channels. It is the responsibility of the front-end application to deliver the right player experience for the channel being used.

Grand National Mobile Usage

<50%

2014

54%

2015

73%

2016

45K

19.3 MILLION

63%OF REMOTE



PEAK BETS P E R M I N U T E

BETS IN 24 HOURS

2013 2014 2015


24 HOURSCREATED IN

OVER 54 MILLIONACCOUNT TRANSACTIONS

IN 24 HOURS

YEAR ON YEAR GROWTH24%*

*TOTAL ACCOUNT TRANSACTIONS IN 24 HOURS

PLACE BET

12

SUMMARYIn summary, you should always beprepared for localised peaks in demand and ensure that new feature releases are assessed for changes in systembehaviour. Regular, automated load testing is ideal but any regular analysis will help understand normal behaviour and local peak effects. For expected, high load times - whether due to new functional releases, seasonal behaviour (Christmas rush) or a promotional event - preparation can make all the difference.

Learn the lessons of previous situations and stay actively engaged throughout the peak period with fall-back plans in the case of unexpected behaviour. Year

Total Sports Bets On Grand National day

201520142013201220112010200920082007200620052004

Remember that peak loads today will be your normal behaviour in the not too distant future. ”

“

13

WHO ARE OPENBET?

TrustedInstrumental to the success of major operators around the globe, OpenBet powers the largest share of the world’s online bets: taking over a billion sports bets in 2014 alone. OpenBet is

trusted by some of the world’s most successful operators including William Hill,

Ladbrokes, Paddy Power, Betfair, SkyBet and PMU. It is our performance and exacting levels of probity that have seen us become known as

the supplier of choice for regulated gaming markets around the globe.

DynamicOpenBet prides itself on product innovation and technical excellence. This makes it the

technology provider and betting platform of choice for regulated gaming markets around

the globe. OpenBet’s inherent culture of innovation has enabled the development of

the industry’s first omni-channel gaming solution and ensures that OpenBet stands out as the market leader within the complex world

of gaming software.

EstablishedFounded in 1996, OpenBet powers the largest

share of the world’s online bets with global software deployments in multiple, regulated

gaming jurisdictions. With six offices worldwide including North America, Australia, Singapore and London, over 20 customers in

the online gaming industry and over 200 integrated partners worldwide, OpenBet

provides a truly open gaming platform and a product suite that ensure the very best and

most flexible solutions available.

EST1996

EST1996

EST1996

For more information about OpenBet, please visit www.openbet.com or email [email protected].

Players • Performance • Partnership

Openbet - How we ensure the stability of our platform

Documents

Transcript of Openbet - How we ensure the stability of our platform