SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management...
Transcript of SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management...
![Page 1: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/1.jpg)
©2013 DISNEY
SysAdmins Unleashed!Jon Geibel | Director of Systems
Ron Johnson | Technology Manager
Walt Disney Animation Studios
1
![Page 2: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/2.jpg)
©2013 DISNEY
Founded, October 1923
First animated feature, Snow White and the Seven Dwarfs, 1937
Frozen - 53rd Animated Feature
800 artists & engineers in one vertically integrated studio
2
![Page 3: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/3.jpg)
©2013 DISNEY
30,000 Core Renderfarm60M Render Hours on Frozen1.5MW Data Center6PB Raw Storage1000 Linux Workstations800 Macs
Our scale...
3
![Page 4: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/4.jpg)
©2013 DISNEY
5,300
10,00030,000
2013
20122011
Scale is increasing...
4
![Page 5: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/5.jpg)
©2013 DISNEY
50-person Systems Team5
![Page 6: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/6.jpg)
©2013 DISNEY
21 Years Experience
50% Engineering, 50% Management
Engineers building an Engineering Organization
Use Scientific Method
Survived many top-down re-orgs
Who we are...
6
![Page 7: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/7.jpg)
©2013 DISNEY
DIRECTOR OF SYSTEMS
LINUX
MANAGER
CORE SERVICES
NETWORK
DATA CENTER
RENDERFARM
MANAGER
STORAGE
DATABASE
TECH SUPPORT
MANAGER
PROCUREMENT
MAC OS
WINDOWS
BACKUPS
MANAGER
ARCHIVING
SYSTEMS OPERATIONS
AUDIO VISUAL
MANAGER
POST PRODUCTION
Our old traditional model...
7
![Page 8: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/8.jpg)
©2013 DISNEY
DIRECTOR OF SYSTEMS
LINUX
MANAGER
CORE SERVICES
NETWORK
DATA CENTER
RENDERFARM
MANAGER
STORAGE
DATABASE
TECH SUPPORT
MANAGER
PROCUREMENT
MAC OS
WINDOWS
BACKUPS
MANAGER
ARCHIVING
SYSTEMS OPERATIONS
AUDIO VISUAL
MANAGER
POST PRODUCTION
Resource ContentionPigeonholing
Difficult to evolveIndirect communicationCareer path limitations
No empowerment
Problems...
8
![Page 9: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/9.jpg)
©2013 DISNEY
LINUX
CORE SERVICES
NETWORK
DATA CENTER
RENDERFARM
STORAGE
DATABASE
TECH SUPPORT
PROCUREMENT
MAC OS
WINDOWS
BACKUPS
ARCHIVING
SYSTEMS OPERATIONS
AUDIO VISUAL
POST PRODUCTION
Let’s move the managers out of the way, and remove silos...
9
![Page 10: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/10.jpg)
©2013 DISNEY
Small Autonomous Teams2-6 PeopleResponsible for R&D, Engineering & SupportAct as small Start-upsResults Oriented Focus
10
![Page 11: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/11.jpg)
©2013 DISNEY
Co-locatedEmbedded warroomAvoid formal meetings
11
![Page 12: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/12.jpg)
©2013 DISNEY
Team Roles
Team Lead
Primary Members Secondary Members
12
![Page 13: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/13.jpg)
©2013 DISNEY
Technical expert in area
Can have any titleProvides Tactical & Strategic leadership
Works directly with stakeholders
Lead Role
13
![Page 14: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/14.jpg)
©2013 DISNEY
Lead RoleLead works with managers to determine equipment, budget, & headcount
14
![Page 15: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/15.jpg)
©2013 DISNEY
Primary Role
Data Center Team
Dedicated to a team
Everyone has a Primary Role on one of the teams
15
![Page 16: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/16.jpg)
Secondary RoleSecondary Role Primary Role
©2013 DISNEY
Linux Team Storage Team
Data Center Team
Secondary Roles add technical and soft skills to a teamNo talent left behind
Playing Multiple Roles...
16
![Page 17: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/17.jpg)
©2013 DISNEY
So... Where did all of the Managers go??
17
![Page 18: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/18.jpg)
©2013 DISNEY
Same structure as other teamsManagers work as a team
18
![Page 19: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/19.jpg)
©2013 DISNEY
LINUX
MANAGER
CORE SERVICES
NETWORK
DATA CENTERXManagers no longer “own” functional areas; that’s the job of the team leads
19
![Page 20: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/20.jpg)
©2013 DISNEY
Career and Team Coaching Team Investment Strategies
Managers work as a team and provide two major functions
20
![Page 21: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/21.jpg)
©2013 DISNEY
Career & Team Coaching
For those that like sports analogies: Managers act more like coaches on sidelinePeople can pick their own Manager based on skill set and personalityTeams can have members that report to different Managers
21
![Page 22: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/22.jpg)
©2013 DISNEY
Fixes A Couple of Traditional Problems
When you change technical roles you no longer have to change Managers
You can change managers without changing what you do
22
![Page 23: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/23.jpg)
©2013 DISNEY
Investment Strategies
‣ Decides on where to invest‣ Build new teams as technology evolves‣ Select new Team Leads based on who has
best ideas and vision‣ Provide resources (people + capital) for
Team Lead‣ Helps define team goals that align with
Studio direction
23
![Page 24: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/24.jpg)
©2013 DISNEY
This is how we look today
LINUXCORE SERVICESNETWORKDATA CENTERSTORAGEDATABASESSYSTEMS
MANAGEMENTPROCUREMENTBACKUPSARCHIVINGTECH SUPPORTRENDERFARM
SYSTEMS OPERATIONS
MAC OS MEDIA MOBILE WINDOWS
No Hierarchy
Tech Leads are top of Food Chain
24
![Page 25: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/25.jpg)
©2013 DISNEY
NEXT EXPERIMENT:
Breaking down the Physical Walls
25
![Page 26: SysAdmins Unleashed! - USENIX...©2013 DISNEY 21 Years Experience 50% Engineering, 50% Management Engineers building an Engineering Organization Use Scientific Method Survived many](https://reader036.fdocuments.in/reader036/viewer/2022081405/5f0c75c97e708231d435830a/html5/thumbnails/26.jpg)
©2013 DISNEY
Thank you.
26