UbiServicesDevOps Journey

June 2019 – Adaptavist

1

2

• Since University (2002), always worked on Online problems

• Manage technical people since 2005• Always been a technical manager• Joined Ubisoft in 2008 as Team Lead• Progressed as Project Lead in 2013• Now Product Director

Alain Cormier

2

3

• Founded in 1986• >15 000 employees• World Wide presence• > 40 studios around the world

• 3 studios in Quebec• Publisher of popular brands

3

4

Game Productions

• Very large teams (1000+)• Various expertise (Historian, Artists, Comedians,

Sounds specialists, Game designers, UX specialists, Programmers, etc)

• 3+ years of development for AAA games• Big bang release!

4

5

Harbour

• Provide Online Solutions to Game Productions• Multiplayers• Global Online Services• Development pipelines• Club (loyalty platform)

• ~300 employees• Global presence

• Montreal (200+)• Dusseldorf (20+)• Bucharest & Craiova (30+)• Shanghai (20+)

5

UbiServices storyPART | 2

Today and futurePART | 3

Take awaysPART | 4

QuestionsPART | 5

Devops – Key conceptsPART | 1

6

Devops – Key conceptsPART | 1

7

Common problem

8

1. Low-stress

2. Feedback Loop

3. Decouple Deploy

From Release

4. Small Batch Size

5. Autonomous Teams

6. Reduce Lead

Time

9

UbiServices StoryPART | 1

10

11

(2010) - Once upon a time…• Use custom techno (Rendez-Vous) to build online

services

• Each projects had its “branch”

• Services were forked and customized for

each productions

• Lot of time spent to retests, do load tests and to put in

place infrastructure for development and launch…

Repeated for all projects.

• Still in use today for Productions Custom Services

• Priority was isolation between projects

11

12

(2012) - Small group want to work differently

• Use standard technos

• Do it once for everyone (global services)• Path base loadbalancing

• Microservices architecture• Let’s start with Critical Service: authentication!

12

13

(2013) – Some challenges before first launch

• API – Referential & Versioning

• Autonomy• Collaboration with IT team• Monitoring

13

(2013) - API management: turn on the light!

14

15

(2014) – Grow

• Double project size• More Teams

• autonomous… but interdependent• From Load Balancer to API Gateway• Capacity planning

15

16

(2015) - Lets innovate

• Until now, C#/MSSQL shop• Dev teams want to explore new technos

• Start with one Critical Service: Players Notifications

• Stretch IT current knowledge• Do Kanban

• Start with one Critical Service: API Gateway

16

17

(2016) – The cloud!• IT prefer to keep us on premise.

• We insist. Need scalability and easier provisioning.

• New dedicated “squad team” from IT to support us

• Start with one Critical Service: Authentication!

• But…• Not as automated as we would like. Pretty much

done the same way as on premise.

• Heavy on operation.

• People get called regularly for dumb reasons (ex: log rotation).

• Which resulted

• Dev team take full ownership of ops. 17

18

(2017) – Autonomy

• All in to the cloud and automate!

• Improve deployment process (spawn new stack)

• Tools run scripts. Now everyone can spawn a

new stack of a version of the service with a few clicks.

• Closer to immutability. Less in-place changes.

• Development team take ownership of spawning

their servers stack. They write less features,

but feel more autonomous.

• Sys admin team now focus on transversal infrastructure (log stack, jump servers, etc).

18

19

(2018) – Do more with less

• Hiring is a challenge.• Time to market is top priority.

• Prefer managed solutions.

• Let dev team focus on business value. • But keep them owner and responsible.

• Improve dev team agility: containers• Start with one Critical Service: account!

19

Today and FuturePART | 2

20

21

(Today) – The Ubisoft platform

• Working on a “Platform” for all Ubisoft

• New CiCd pipeline for microservices

• Enforce referential• Enforce immutability (no access to machines)• Helps with automated tests• Give API management, logs, metrics and alerts

for free to all projects.

• Containers• Help isolation• Contract is clear between devteam and infra

• Infrastructure as code (for real!)

21

22

(Future) – Challenges

• Onboard Productions into devops and

microservices.

• Review custom service development pipeline.

• Automatic deployment up to Live

environment.

• Move to Inner Sourcing mindset and accept

contribution from any internal teams.

22

Take awaysPART | 4

23

24

Recipe for changes

• Start with small team (strike team) of

highly motivated guys.• Make sure you get support from management.

• Choose critical project/feature.

• Demonstrate the improvement.• Propagate to all teams.• Rince and repeat.

24

25

Sys admin / DBA

• Still need them.• Although, no more authoritative on tech choice.

• More in an expert role and consulted when

needed by devteam.

• Have their own “services” (ex: log stack)

25

26

Teams

• Autonomous but interdependent

• Small: 5-10 people.

• Ramp-up new managers• Split teams (organic growth)

26

27

Devops & Microservices

• Tight together• Microservices mindset:

• Loosely couple components

• Highly cohesive components

• Independently Develop and Deploy• Each component has a contract and must

respect if (need tests!)

27

Thank you!

28