Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox...
-
Upload
puppet-labs -
Category
Technology
-
view
3.154 -
download
0
description
Transcript of Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox...
![Page 1: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/1.jpg)
The BlackBox projectSafely storing secrets and credentials in Git
for use by Puppet
Tom Limoncelli, SRE, StackExchange.comBlog: EverythingSysadmin.com
![Page 2: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/2.jpg)
125+ Q&A CommunitiesServerFault.comStackOverflow.com
(We <3 Puppet!)
StackExchange.com
![Page 3: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/3.jpg)
What are secrets?
Anything you don’t want exposed externally.
● SSL Certificates (the private bits)● Passwords● API keys
![Page 4: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/4.jpg)
Puppet manages secrets
![Page 5: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/5.jpg)
![Page 6: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/6.jpg)
If you store secrets in git, you’re gonna have a bad time.
![Page 7: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/7.jpg)
● Laptops get stolen.● Workstations have guest accounts● “Circle of Trust” now includes:
○ Everyone with admin access to workstations.■ Your desktop support people?
○ Everyone with admin access to your git server:■ Server team, storage team, backup team
○ Everyone you collaborate with that wants read-only access to Puppet manifests.
![Page 8: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/8.jpg)
You have 3 bad options:
1. Deny git access. (Hurts collaboration)2. Permit git access. (Hurts security)3. Email individual files. (Hurts… just hurts)
![Page 9: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/9.jpg)
Option 4: Encrypt secret parts
● If a file contains secrets, encrypt before checking into Git.
● Need to edit a secret?○ Decrypt - Edit - Encrypt
![Page 10: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/10.jpg)
What about Puppet master?
● After “git pull”, decrypt all files.○ Automate this as part of CI.
● Files are unencrypted “at rest”.● This does not decrease security:
○ No worse than what we were doing before.○ If you can break into root or puppet on the master,
you’ve already won.
![Page 11: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/11.jpg)
Easy, right?
Decrypt:gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt
![Page 12: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/12.jpg)
Easy, right?
Decrypt:gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt
● ...and don’t make any typos when entering the command● ...and don't accidentally check in the unencrypted version
![Page 13: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/13.jpg)
Security is 1% technology plus 99% following the procedures correctly.
Any process with more than 1 step probably won't be followed consistently most of the time.
Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0”, Alma Whitten", Usenix Security 1999
![Page 14: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/14.jpg)
Therefore…. we automate
Introducing: Blackbox
Scripts for keeping Puppet secrets in git/hg.
![Page 15: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/15.jpg)
User commands:
Decrypt for editing:blackbox_edit_start.sh file
Encrypt when done:blackbox_edit_end.sh file
![Page 16: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/16.jpg)
First time a file is encrypted:
Enroll a file into the system:blackbox_register_new_file.sh file
![Page 17: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/17.jpg)
Commands that act on all GPG files:
Decrypt all files: (for use on puppet master)blackbox_postdeploy.sh
Re-encrypt all files: (after new users added)blackbox_update_all_files.sh
![Page 18: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/18.jpg)
Everyone has their own key
This doesn’t use “symmetric encryption” where there is one passphrase to decrypt/encrypt all files.
We maintain a keyring of:● Each person that should have access.● A key for the Puppet master.
![Page 19: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/19.jpg)
Indoctrinate a new user:
1. New user does this:
● Create GPG key.● Add their username@host to blackbox-admins.
txt● git commit -a
(Currently a doc, not a script. Patches gladly accepted.)
![Page 20: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/20.jpg)
Indoctrinate a new user:
2. Existing admin does this:
$ gpg --import keyrings/live/pubring.gpg
$ blackbox_update_all_files.sh
$ git commit -a
![Page 21: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/21.jpg)
Demo: Edit a file
![Page 22: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/22.jpg)
Demo: Edit a file
![Page 23: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/23.jpg)
Demo: Edit a file
![Page 24: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/24.jpg)
Demo: Edit a file
![Page 25: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/25.jpg)
Demo: Edit a file
![Page 26: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/26.jpg)
Demo: Edit a file
![Page 27: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/27.jpg)
Demo: Edit a file
![Page 28: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/28.jpg)
![Page 29: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/29.jpg)
![Page 30: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/30.jpg)
Code is open source as of TODAY
● Entirely written in bash.● MIT License.● Download it now:
○ https://github.com/StackExchange/blackbox
![Page 31: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/31.jpg)
In the project’s first 9 months:
StackExchange/ServerFault has eliminated plaintext secrets in our Puppet git repo.● 7 SREs+Devs sharing the repo securely.● 50+ files now stored encrypted.
○ Mostly SSL certs and SSH private keys.● 40+ individual passwords/API keys:
○ Everything from SNMP communities, SaaS API keys, and many many passwords.
![Page 32: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/32.jpg)
Future plans
❏ Open source scripts.❏ More usability enhancements.❏ Better setup documentation.
![Page 33: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/33.jpg)
Join the open source projecthttp://github.com/StackExchange/blackbox
![Page 34: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/34.jpg)
Q&A
URLs from this talk:https://github.com/StackExchange/blackbox
EverythingSysadmin.com
![Page 35: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/35.jpg)
Shameless plugPre-order now! Save 35%
Ships in September.
informit.com/TPOSADiscount code TPOSA35
Read “rough cuts” today:safaribooksonline.com
![Page 36: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/36.jpg)
Q&A
URLs from this talk:https://github.com/StackExchange/blackbox
EverythingSysadmin.cominformit.com/TPOSA (code TPOSA35)
![Page 37: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/37.jpg)
● Easier transition. No Puppet code changes for big files like SSL certs.
● Faster. Zero run-time performance impact on master.
● eyaml didn’t exist when we started.
Why didn’t we use eyaml?
![Page 38: Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange](https://reader033.fdocuments.in/reader033/viewer/2022052822/554f3de7b4c90572088b5105/html5/thumbnails/38.jpg)