Don't Mind the Gap by Galen Emery
-
Upload
devopsdays-baltimore -
Category
Technology
-
view
54 -
download
0
Transcript of Don't Mind the Gap by Galen Emery
![Page 1: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/1.jpg)
Don’t Mind the GapDoing DevOps in an Air-gapped World
Galen EmeryFederal Solutions Architect3/8/2017
![Page 2: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/2.jpg)
Who am I?I work with the Federal Government and IntegratorsDoD, Civilian, or IC? Yes.I help users understand how to build and structure their infrastructure (using Chef) to solve their problemsFormer Windows Server AdminBeen with Chef since March 2014Born and raised in Seattle (Nathen, you’re wrong about Seattle crab)
![Page 3: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/3.jpg)
What we coverWhat is “air-gapped”?What issues do we encounter?AssumptionsHow to solve it (technical)How to solve it (process)What about … ?Food for Thought
![Page 4: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/4.jpg)
What is air-gapped?air-gappedadjective(of a computer) having no direct connection to the Internet or to any other computer that is connected to the Internet, for security reasons."a USB drive or other hardware approach would be required to infect the air-gapped machine"
![Page 5: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/5.jpg)
What about mitm?• Corporate man-in-the-middle• Terminates SSL sessions and re-initiates• Often includes packet inspection• Requires you to trust its generated certificates• May still include a firewall blocking access to sites
![Page 6: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/6.jpg)
Maybe its just a firewall• Restricts access to:• Github or gist.github.com• slack• Pastebin• Rubygems.org / some other artifact repository• etc
![Page 7: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/7.jpg)
No restrictions?• Congratulations!• You might still have a
firewall/proxy, its just permissive
• Do you trust those systems will be up when you need them?
![Page 8: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/8.jpg)
Issues
![Page 9: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/9.jpg)
Getting stuff into production• How do we “cross the gap”?• How do we distribute it?• How do we ensure its integrity?
![Page 10: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/10.jpg)
Process?!You want to talk about PROCESS? You kidding me?!
I just hope we can make one change!
![Page 11: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/11.jpg)
VelocityHow do we achieve velocity if we burn a DVD for every change?Or if a proxy is inspecting all of our packets, every time we download a package?
![Page 12: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/12.jpg)
Okay, I’m not air-gapped but…• I have a firewall between my systems and the internet• I have a proxy that inspects all traffic and slows me down• I don’t have {level of access necessary} to make changes• Tool {X} does not understand authenticating proxies
![Page 13: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/13.jpg)
Ground Rules
![Page 14: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/14.jpg)
You know that I know that you know…• How you get code into your high-side environment
![Page 15: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/15.jpg)
Everything is code• If its source code, it goes into SCM• If it is an artifact, it goes into artifact store, and has a checksum• We can (and do) write tests• We build a pipeline
![Page 16: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/16.jpg)
Solution (High-Level)
![Page 17: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/17.jpg)
Go from Low to High
![Page 18: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/18.jpg)
Assume you don’t have internet• Even if you do• Build this into your pipeline• Test this on the low side• Your systems should NOT assume internet access• Most tooling by default assumes it does
![Page 19: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/19.jpg)
Do this work on the low-sideBuild your pipeline in a way that removes internet access for your systems, they must grab their code and artifacts locallyIf you can build your infrastructure without internet access in Dev, you can absolutely do it in Prod
![Page 20: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/20.jpg)
Protect the Data, not the InfraThe data is sensitive, not the infrastructure itself. Keep the data in the high-side, but ensure that you build your infrastructure with the same code in Dev and Prod
![Page 21: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/21.jpg)
WorkstationThis is your “loading dock” for the rest of the infrastructureEverything comes through hereIt needs:• A way to serve files (directly or indirectly)• A way to create artifacts (zip, tar, etc)• built programmatically
![Page 22: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/22.jpg)
WorkstationUse the workstation to:• Stand up your artifact repo• Stand up your configuration management infra• Publish your artifacts• Run tests
![Page 23: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/23.jpg)
Workstation (Chef Example)Bring in to the workstation a zip/tar with the following• ChefDK• A FTP/SSH/SCP Server binary (if it doesn’t already exist on your box)• Your cookbooks• Any extra gems necessary• Chef packages (client, server)
Use Chef Zero to stand up your FTP Server, populate it with the artifacts and prepare it for use in the rest of the process
![Page 24: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/24.jpg)
What about dependencies?I could download, transfer, attempt to install, download, transfer, attempt to install..
I could skip using the gem and instead write it myself
I can create a full gem mirror of all 80,000+ gems on rubygems.org
I can install what I need into a directory, and then move that directory over as an artifact
![Page 25: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/25.jpg)
Simplest is to create an artifactI used “gem install –i $PATH $GEM” && tar cf $PATHIts not pretty but it worksBenefit is: I have a moment-in-time artifact of the gems I’m usingIdeally you’d create this at the end of your development pipeline
![Page 26: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/26.jpg)
You should be using an artifact repo• Use something that can store and manage your artifacts• You can version your artifacts• Often supports the correct dependency structure for your artifacts
![Page 27: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/27.jpg)
Once you have Workstation + Artifact• You can setup your configuration management• Use a tool, running from your workstation to setup the infrastructure
you need
• Example• Chef Provisioning SSH• Doesn’t require internet access• Can bootstrap Chef from a FTP or SCP server
![Page 28: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/28.jpg)
Everything is set, right?• Have workstation, Config Mgmt + Artifact store
• But I haven’t told any of my infrastructure that it shouldn’t reach out to the internet for X, Y or Z
![Page 29: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/29.jpg)
Tell your systems to stay inside• Remove unreachable Satellite repositories• If using ruby, remove rubygems from your sources. Add your artifact
store• If using Chef, update berksfile to an internal supermarket• If using X, update Y to Z
• Don’t do this manually, do it with code and test it on the low side!
![Page 30: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/30.jpg)
What if I have a proxy instead?Most of the tooling supports HTTP_PROXY and HTTPS_PROXYSome of it does notIt is often easier to design the system to assume no network access than to keep fighting proxiesThat said, if you can poke holes out to slack, rubygems, github, chef, etc you’ll be much happier (unless they go down)
![Page 31: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/31.jpg)
Changing the Process
![Page 32: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/32.jpg)
Do this in Dev• Do not wait for Prod to test your systems without internet access• Along these lines, harden your Dev systems to the same standards
as Prod
![Page 33: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/33.jpg)
Create and Use• CI/CD Pipelines• Unit and Integration Tests
![Page 34: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/34.jpg)
Identify where you assume risk• If you let users ssh/rdp into production, that is where the risk lies• If you only let users make changes to production through a pipeline,
that’s where the risk lies• If the risk lies in a pipeline and someone else has to approve code
changes, the risk of any single change/actor is much lower
![Page 35: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/35.jpg)
Imagine This Scenario• Nobody can ever log into Production• All changes flow through a pipeline that tracks who committed,
reviewed and shipped the change• All changes are tested through Dev, QA, etc before deploying to Prod• We build a new Prod every time we make a change
![Page 36: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/36.jpg)
What About … ?
![Page 37: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/37.jpg)
My laptop can’t reach X• If you can’t reach GitHub, or Slack, etc ever; not just in Production
that’s an issue
![Page 38: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/38.jpg)
Security won’t let me!
• Work with them. They are a vital component of your business (or should be)
• Identify where the risk is. The risk is with the data, not with the user.
• Consume Risk in Dev, not in Prod (Fail Fast)• Progress is coming
![Page 39: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/39.jpg)
How often do we update our packages?
![Page 40: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/40.jpg)
Where are the gates?
![Page 41: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/41.jpg)
What are the metrics?Velocity: How fast we shipEfficiency: How good we are at shippingRisk: How long does it take us to fix
![Page 42: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/42.jpg)
Food for Thought
![Page 43: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/43.jpg)
Alaskan King Crab• Biggest crab, legs alone can be a meal• Served steamed or chilled with drawn butter and lemon• Fishing Season:• October• January
![Page 44: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/44.jpg)
Dungeness Crab• Similar size to blue crab, hard shell• Served steamed or chilled with drawn butter and lemon• Fishing Season:• Starts in November• Ends in June/July
![Page 45: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/45.jpg)
Blue Crab• #CrabFeast• Old Bay• Fishing Season:• Starts In April
![Page 46: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/46.jpg)
Proof of Concept codewww.github.com/galenemery/chef-mreWant a USB Key with it?
![Page 47: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/47.jpg)
Google “stuxnet” or “vault 7”If you asked for a USB Key, please let me know who holds your clearance so we can have it revoked
![Page 48: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/48.jpg)
ACKNOWLEDGEMENTSRobb KiddChefConf 2014: Mind the Gap: Deployments with Chef in unforgiving Environments
![Page 49: Don't Mind the Gap by Galen Emery](https://reader035.fdocuments.in/reader035/viewer/2022070517/58cee2831a28ab333d8b49a1/html5/thumbnails/49.jpg)
ACKNOWLEDGEMENTsThank you JOKES ON YOU, THERES NOTHING HERE
Thank you DevOpsDays Baltimore