Rovio © 2016 Confidential

Rovio © 2016 Confidential

Rovio’s Infrastructure The New Path ForwardPaul Winston VillacortaSenior Cloud Engineer, Cloud OperationsRovio Entertainment

Rovio © 2016 Confidential

HybridCloud

Private

Cloud

PublicCloud

3rd Party ManagedPrivate Cloud

Rovio © 2016 Confidential

Private Cloud Public Cloud

Rovio © 2016 Confidential

ORCHESTRATION

TRAFFIC MANAGEMENTTRAFFIC MANAGEMENT

TRAFFIC MANAGEMENTSERVICES

TRAFFIC MANAGEMENTDATABASE

Amazon CloudServices

External ServiceProvider

CDN

ExternalMonitoring

Incident Management

System

CONTINUOUSINTEGRATION

CONTINUOUSDELIVERY

Rovio © 2016 Confidential

EXTERNAL TRAFFIC MANAGEMENT

SERVICES

INTERNAL TRAFFIC MANAGEMENT

MICROSERVICES SERVICES MICROSERVICES

DATABASE TIER

ORCHESTRATION / AUTOMATION LOGIC

CACHING COMPONENTS STORAGE

MON

ITOR

ING

LOG

COLL

ECTI

ON

INTE

RNAL

DNS

OPERATIONAL JOB SCHEDULER

VIRTUALIZED

Rovio © 2016 Confidential

IT IS HUGE!!

Rovio © 2016 Confidential

200+DatabaseInstances

60+Services

700 - 2000Running

Instances

{API}

45000+API Requestsper second

3BAnalyticsProcessedevery day

Rovio © 2016 Confidential

AUTOMATION

WE DON’T ALWAYS AUTOMATE STUFFBUT WHEN WE DO IT’S END-TO-END

Rovio © 2016 Confidential

AUTOMATION• Scalability – Automatic node creation

• Availability – Automatic node replacement

• Efficiency – Automatic intelligence

Load Testing, Disaster Recovery, Rolling Upgrades, Compliance, Portability, etc…

Rovio © 2016 Confidential

AUTOMATION TOOLS• Scripting / Programmable Utilities• Puppet, Fabric, Ansible• Docker, Packer• Vagrant

Rovio © 2016 Confidential

AUTOMATION

• Builder Script is written in Python• Utilizes REST APIs• 1,100 lines of code• Week of development + testing

• 5 seconds to create new project• We have saved 70 steps and 140 clicks

Continuous Integration /Continuous

Delivery

Rovio © 2016 Confidential

AUTOMATION

• It’s GUI based• It’s API driven• It supports standalone CLI• It’s cloud aware• It’s cluster aware• It’s inside the cloud• It works like a charm

Rovio’s Orchestration

Server

Rovio © 2016 Confidential

AUTOMATION

OrchestrationServer

Cloud Provide

r

New Node

Cloud Properties

Rovio © 2016 Confidential

AUTOMATION

OrchestrationServer

Internal DNS

Monitoring

Ready Node

PACKAGE

PUPPETMANIFEST

S

Rovio © 2016 Confidential

AUTOMATION

OrchestrationServer

Internal DNS

Monitoring

Ready NodeTraffic

Management

Rovio © 2016 Confidential

LESSONS LEARNED

METRICS AND VISIBILITY - SMALL FAULT MARGINS - LEARNING FROM MISTAKES

Rovio © 2016 Confidential

Metrics and Visibility• Sensible Metrics• Actionable Items• Metrics that can be visualized and aggregated

Rovio © 2016 Confidential

Metrics and Visibility

Rovio © 2016 Confidential

Metrics and Visibility

Rovio © 2016 Confidential

Small fault margins• Small mistakes build up quickly• Small mistakes are unpredictable• Small mistakes costs money!

Rovio © 2016 Confidential

Learning from mistakes• NEVER leave critical components UNATTENDED!• Do it RIGHT from the BEGINNING!• Do a proper root-cause analysis and publish the findings

in your organization.

Rovio © 2016 Confidential

CHALLENGES

Rovio © 2016 Confidential

CHALLENGES• Modernizing legacy applications• Cloud Providers configuration changes• Security• Legacy nodes• AZ Distribution• Architectural Challenges

Rovio © 2016 Confidential

Thank You!