Deseret matrix marketing services powerpoint presentation dollar
SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps
-
Upload
saltstack -
Category
Technology
-
view
427 -
download
0
description
Transcript of SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps
![Page 1: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/1.jpg)
Mentoring Devs Into DevOpsJustin Carmony Director of Development Deseret Digital Media
![Page 2: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/2.jpg)
![Page 3: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/3.jpg)
Lets Measure The Audience
• Who here is a…
• System Administrator?
• Developer?
• Manager / Management?
• “DevOp?”
![Page 4: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/4.jpg)
Confession:I’m a Developer
![Page 5: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/5.jpg)
![Page 6: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/6.jpg)
Self-Taught OpsBecause There Was No One Else To Do It
![Page 7: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/7.jpg)
About Me• Director of Development
for Deseret Digital Media
• Utah PHP UsergroupPresident
• I Make (and Break) Web Stuff (10 years)
• Salt User in Production since 0.8
(I <3 Salt)
![Page 8: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/8.jpg)
This Presentation• Lessons learned at DDM & previous jobs
• Insight into our process of increasing “DevOps”
• We’re still learning, but this what we’ve found.
• Slides will be posted online, so don’t worry about copying slide content.
• Feel free to ask on-topic questions during, and we’ll have questions at the end.
![Page 9: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/9.jpg)
About DDM
• Deseret Digital Media runs local website like KSL.com, DeseretNews.com
• Running National and International Websites like OK.com, familia.com.br, etc.
• ~10 million pageviews a day across sites.
• ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.
![Page 10: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/10.jpg)
Lets Start With a Story!
![Page 11: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/11.jpg)
You Work for an Awesome Tech Company
![Page 12: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/12.jpg)
Team Is Working Hard to Build New Things!
![Page 13: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/13.jpg)
You launch your awesome product!
![Page 14: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/14.jpg)
A Few More Features…
![Page 15: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/15.jpg)
… and next thing you know…
![Page 16: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/16.jpg)
![Page 17: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/17.jpg)
Awesome Job Team, We Rock!
![Page 18: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/18.jpg)
![Page 19: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/19.jpg)
We Need !Real-Time XYZ Feature!
ASAP!
![Page 20: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/20.jpg)
We Need !Real-Time XYZ Feature!
ASAP!
![Page 21: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/21.jpg)
We Need !Real-Time XYZ Feature!
ASAP!
&#$%!
![Page 22: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/22.jpg)
![Page 23: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/23.jpg)
![Page 24: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/24.jpg)
“Huh, it works if you ! just turn off caching…”!
- Dev @ 80th Hour This Week
![Page 25: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/25.jpg)
![Page 26: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/26.jpg)
“I’m sure this ! will work…”
![Page 27: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/27.jpg)
![Page 28: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/28.jpg)
![Page 29: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/29.jpg)
![Page 30: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/30.jpg)
“Our servers are melting!”
![Page 31: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/31.jpg)
![Page 32: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/32.jpg)
![Page 33: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/33.jpg)
![Page 34: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/34.jpg)
![Page 35: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/35.jpg)
![Page 36: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/36.jpg)
![Page 37: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/37.jpg)
“We Need a Better Solution!”
![Page 38: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/38.jpg)
So…
![Page 39: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/39.jpg)
Where Do We Start?
![Page 40: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/40.jpg)
![Page 41: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/41.jpg)
![Page 42: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/42.jpg)
![Page 43: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/43.jpg)
![Page 44: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/44.jpg)
We Have This Problem
![Page 45: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/45.jpg)
Challenges We Faced
• Giant mesh-up of technologies
• Tightly-coupled & fragile infrastructure
• Debugging production only bugs was difficult
• Bugs that were part code, part environment were a nightmare to track down.
![Page 46: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/46.jpg)
![Page 47: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/47.jpg)
![Page 48: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/48.jpg)
So One Day… We Had A Genius Idea!
![Page 49: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/49.jpg)
Lets Hire a DevOp!
![Page 50: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/50.jpg)
I’m Not JokingWe Actually Said This
![Page 51: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/51.jpg)
Two Problems with this “Idea”
![Page 52: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/52.jpg)
Problem #1 - We Didn’t Understand What We Really Wanted
![Page 53: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/53.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit!
![Page 54: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/54.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
![Page 55: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/55.jpg)
Problem #2 - People Who Are Great At Dev & Ops Are Hard To Find
![Page 56: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/56.jpg)
Expectation:
![Page 57: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/57.jpg)
Reality:
![Page 58: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/58.jpg)
Honest Team Discussion: What is it we’re really looking for?
![Page 59: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/59.jpg)
We Discovered a Few Things
![Page 60: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/60.jpg)
What does DevOps Mean To Us?
• DevOps: Dev & Ops, a Culture of Collaboration
• Our Goal: “10 deploys a day without issues”
• Everyone shares the goal of quick development of features AND a stable system that stays up.
![Page 61: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/61.jpg)
Team StructureDevs: 30 Ops: 2
![Page 62: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/62.jpg)
Team StructureDevs: 30 Ops: 2
DevOps: 1
![Page 63: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/63.jpg)
Team StructureDevs: 30 Ops: 2
DevOps: 1
![Page 64: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/64.jpg)
Team StructureDevs: 30 Ops: 2
DevOps: 1
Hiring one person won’t just solve all our problems!
![Page 65: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/65.jpg)
Team Realizations
• Hardest problem already solved: awesome team
• No foreseeable rapid expansion, must operate at our current scale
• Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.
![Page 66: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/66.jpg)
Teams Already Had Some Ad-Hoc DevOps Tools
- Real-time Logging - Capistrano Deploys
- Nagios Alerts - Server Metrics - Puppet for File Mgmt
- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management
- Homebrewed Metrics Sys. - Homebrewed Alert System
![Page 67: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/67.jpg)
Teams Already Had Some Ad-Hoc DevOps Tools
- Real-time Logging - Capistrano Deploys
- Nagios Alerts - Server Metrics - Puppet for File Mgmt
- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management
- Homebrewed Metrics Sys. - Homebrewed Alert System
![Page 68: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/68.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
![Page 69: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/69.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
![Page 70: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/70.jpg)
We Formed A Strategy
![Page 71: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/71.jpg)
Step #1: Promote Dev to DevOp Role
![Page 72: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/72.jpg)
WAIT!Isn’t that the advice you just
said was a bad idea?!
![Page 73: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/73.jpg)
DevOp Engineer
• Well Defined Role:
• Ownership over the TOOLS to improve DevOps efforts.
• Resource for other teams to help use DevOps Tools.
• Easy to work with, aptitude for systems & ops, likes to try new things.
![Page 74: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/74.jpg)
Promoting From Within
• A seasoned dev for your team already knows:
• Your Pain Points
• Your System’s Quirks
• How the “Chaos Works”
• Knows the people & personalities on your team
![Page 75: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/75.jpg)
Step #2: Change Team Structure
![Page 76: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/76.jpg)
Team StructureDevs: 30 Ops: 2
![Page 77: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/77.jpg)
Team StructureDevs: 30 Ops: 2
![Page 78: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/78.jpg)
Team StructureGoal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops
![Page 79: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/79.jpg)
Team StructureGoal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops
![Page 80: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/80.jpg)
Team Structure
Dev Ops
![Page 81: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/81.jpg)
Team Structure
Dev Ops
![Page 82: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/82.jpg)
Increasing Ops Among Devs
• Identify Devs who liked “Ops” & wanted to Learn
• Pair Dev with Op / Director
• Learning Dev works on things, not Op /Director.
• Pair program if needed.
![Page 83: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/83.jpg)
Step #3: Increase Everyone’s Insight
![Page 84: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/84.jpg)
Step #3: Increase Everyone’s Insight
![Page 85: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/85.jpg)
![Page 86: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/86.jpg)
Metrics• Everyone has access to Network, Server, and
Application Metrics.
• Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite
• Each developer trained to add & track metrics in production.
• We’re okay with 98% uptime of stats to avoid complexity.
![Page 87: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/87.jpg)
![Page 88: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/88.jpg)
Real-Time Logging
![Page 89: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/89.jpg)
Real-Time Logging• Harder & more complicated at scale
• Still trying to solve well, we have lots of logs.
• Start with small window of data (i.e. 48 hours) and start to expand window.
• We’re trying Logstash, ElasticSearch, and Kibana right now.
• Generate Statistics off our Logs
![Page 90: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/90.jpg)
Tracking Changes
• Everything, everything, everything in git (we use GitHub)
• Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed *
* - Your can merge w/o a review, but must be willing to defend your choice
![Page 91: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/91.jpg)
Deploys
![Page 92: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/92.jpg)
Everyone Can Deploy
• Automated our deployment process to a single step.
• Everyone can deploy, deployments are logged
• Easy rollback is a requirement!
• Implementing feature flags to turn off single parts of our application.
![Page 93: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/93.jpg)
TestsUnit
Functional Integration
Acceptance etc
![Page 94: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/94.jpg)
Automated Tests
• If you want to trust your Devs, you need tests
• Legacy apps we wrote Integration Tests
• New Apps & Refactored Legacy Parts have Unit Tests
• Continuous Integration to make sure tests run
![Page 95: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/95.jpg)
![Page 96: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/96.jpg)
So, where’s the salt?
![Page 97: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/97.jpg)
Step #4: Devs Use The Ops Tools
![Page 98: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/98.jpg)
Devs can grok salt
![Page 99: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/99.jpg)
Safe Environment For Devs to Learn
![Page 100: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/100.jpg)
Safe Environment For Devs to Learn
salt \* cmd.run "rm -rf /tmp /*"
![Page 101: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/101.jpg)
Safe Environment For Devs to Learn
salt \* cmd.run "rm -rf /tmp /*"
Salt is awesome, but it can’t !recover from that
![Page 102: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/102.jpg)
Dev Salt MasterDevs Can Look Into Every Server
![Page 103: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/103.jpg)
Dev Salt Master• Every server has two minions:
• Admin Salt (aka root)
• Dev Salt (aka bob)
• Each connect to different master server:
• All Devs have access to Dev Salt Master
• Trusted Devs get access to Admin Salt Master
![Page 104: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/104.jpg)
Everything Salty in GitReminder:
![Page 105: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/105.jpg)
Dev Environment
• Developers own the Dev Environment
• Dev Teams manage the Salt States for their Env
• Vagrant + Salt for their Env
• Who makes changes? Developers
• DevOp helps advise & offer support
![Page 106: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/106.jpg)
Team Structure
Dev Ops
![Page 107: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/107.jpg)
Stage Environment• Stage & Production use same salt repos, different
branches
• Developers make all the changes for Application Servers
• All Changes through Pull Requests
• We’ll worry about env changes before code
• Small changes we quickly release, large or long running branches are scary & dangerous
![Page 108: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/108.jpg)
Production Environment
• Merge change to Production Branch
• salt \* state.highstate
• Reminder: Small quick changes over time, never a large change at once.
![Page 109: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/109.jpg)
Environment Caveats• Ops & DevOps Manage VM Hosts, Physical Load
Balancers, FireWalls, etc
• Ops & DevOps manage servers that deal with data:
• MySQL
• MongoDB
• etc
![Page 110: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/110.jpg)
Mentoring Devs
![Page 111: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/111.jpg)
Mentoring Devs
• Not every Dev will become an amazing DevOp
• Thats okay!
![Page 112: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/112.jpg)
Level of “DevOps” Skills
• Thinks about their impact on Ops: Everyone
• Able to debug issues with production: Most
• Able to make changes to environments: Many
• “Awesome DevOp”: Some
![Page 113: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/113.jpg)
Team’s “DevOps” Skills
0
25
50
75
100
Think Debug Change DevOp
Current Goal
![Page 114: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/114.jpg)
So Everything Is Awesome for us, right?
![Page 115: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/115.jpg)
Honesty Slide: We Have Skeletons In Our Closets
![Page 116: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/116.jpg)
Where We Are At
• All Dev Environments using Vagrant + Salt
• All New Stage & Prod Environments are Salty
• Some Legacy Stage & Production Envs are Salty
• Continuously working on getting out stuff salty.
![Page 117: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/117.jpg)
Making This Work For Your Team
![Page 118: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/118.jpg)
Honest Introspection• Determine for your
team what are your… • Strengths • Weaknesses • Problems • Goals
![Page 119: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/119.jpg)
Increase Team’s Insight • Make sure devs can see
& understand how their code performs
• Increase responsibility of team for those metrics.
• If they break it, they fix it. Do not always bail them out.
• Everyone can see everything.
![Page 120: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/120.jpg)
Increase Team’s Insight • Make sure devs can see
& understand how their code performs
• Increase responsibility of team for those metrics.
• If they break it, they fix it. Do not always bail them out.
• Everyone can see everything.
![Page 121: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/121.jpg)
Mentor Those With Desire / Aptitude
• Give Developers Safe Environment to Learn
• Let them submit code-reviewed changes for Stage & Production
• When teaching / mentoring, let the learner drive, kindly offer advice and help.
• It takes time, but worth the investment.
![Page 122: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/122.jpg)
A Few Final Thoughts
![Page 123: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/123.jpg)
Team Culture Matters
![Page 124: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/124.jpg)
Positive Influence
![Page 125: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/125.jpg)
Questions?
![Page 126: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/126.jpg)
Thank YouJustin Carmony
Email: [email protected]: @JustinCarmony
IRC: carmony #salt #uphpuWebsite: [email protected]
![Page 127: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.fdocuments.in/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/127.jpg)
p.s. we’re hiring, email / pm / tweet me