Building A Self-Documenting Application: A Study in Chef and Compliance
-
Upload
kevin-gilpin -
Category
Software
-
view
249 -
download
0
Transcript of Building A Self-Documenting Application: A Study in Chef and Compliance
2015 #ChefConf WorkshopBuilding a Self-Documenting Compliant Application
www.conjur.net | @conjurinc
© Confidential
Set-up Prerequisites
1.Sign up for the Conjur demo appliancehttp://info.conjur.net/secrets-management-demo
2. Login to the demo applianceIf you’re on Windows, use Putty
http://bit.ly/1NAHh3Y
Ready To Deep Dive?
© Confidential
Welcome!
I. AgendaII. Workshop Goals
III. Meet Your Workshop Hosts, Conjur IntroductionIV. Ready To Deep Dive? Set-up Conjur Prerequisites
● Part I: Introduction to DevOps and Compliance / Q&A ● Part II: Chef Secrets Management Overview / Q&A
Agenda
© Confidential
- Break -
● Part III: Practicum - Build a Sample App● Part IV: Compliance & Chef; Good Practice / Q&A ● Part V: Traceability, Auditability and Chef / Q&A
V. Wrap up
Agenda
© Confidential
Workshop GoalsAfter this workshop, you will:
● Have a deeper understanding of how compliance with Chef, access to the Chef Server, and Secrets Management are interrelated.
● Know how to implement a delivery pipeline that combines the two topics in a self-documenting and highly automated fashion.
© Confidential
Your Workshop Hosts
● Kevin Gilpin, CTO and Co-Founder, Conjur - Kevin has 19 years of technical leadership in
enterprise software, delivering solutions to the Fortune 500. He has held technical positions at Infinity
Pharmaceuticals, Praxeon Inc, i2 Technologies, and Trilogy, where he developed technology that
delivered pioneering interactivity on the web and has ultimately been adopted across the industry.
@kegilpin
● Elizabeth Lawler, CEO & Co-Founder, Conjur - Elizabeth Lawler is CEO and Co-founder of
Conjur, Inc. Elizabeth has over 20 years of experience working in highly regulated and sensitive data
environments. Prior to founding Conjur, Elizabeth was Chief Data Officer of Generation Health and held
a leadership position in research at the Department of Veterans Affairs. She has been a programmer
herself and is constantly working to make software development and IT systems easier to manage for
people working in regulated industries. @elizabethlawler
© Confidential
Your Workshop Hosts
● Dustin Collins, Developer Advocate, Conjur - Dustin is a polyglot engineer and Conjur's
developer advocate. He organizes the Boston DevOps meetup and is most interested in how we as a
community can iterate on our processes and tools to enable continuous delivery of quality software.
@dustinmm80
● Christopher Farnham, Novartis - Chris is a proven software development and operations team
leader within diverse domains, including banking, publishing, military and biotechnology. He has
demonstrated deep and broad knowledge of SDLC processes within highly regulated, demanding (but
ultimately person driven) environments. @chrisfarnham
© Confidential
Conjur IntroductionWe believe that security success for all organizations is
determined at the business and infrastructure levels
● Conjur is a privileged access security platform for large and fast-growing cloud and web-based enterprises using next generation technologies.
● It provides a smarter, transparent, automated way to scale and manage critical secrets and access control. It’s used on-premise or in your cloud (private, public, hybrid).
© Confidential
Conjur Introduction
Application Auth
Systems Access
Internal Network
Physical Infrastructure
Firewall
Control Plane
We provide the modern access management and audit solution for the missing pieces: Systems Admin and the Systems Control Plane.
Conjur Resolves The Access Control Gap
© Confidential
Conjur Introduction● Conjur solves three critical infrastructure security growth
issues:
> Secrets Management> Access Control (SSH) (e.g., machines, robots, humans)> Audit (Compliance)
● Integration-Friendly! We work with all technology stacks.
© Confidential
Conjur Clients
© Confidential
Part I.
Introduction to DevOps and Compliance
Compliance for People who Cringe at “Compliance”
© Confidential
How the Business Views DevOps
How does DevOps work?
Magic.
© Confidential
Compliance and DevOps
Source: DevOps: The Worst-Kept Secret to Winning in the Application Economy CA Technologies October 2014 cite: http://rewrite.ca.com/us/~/media/rewrite/pdfs/white-papers/devops-winning-in-application-economy.pdf
© Confidential
Goal: Reproducible, Reliable Processes
● Same packages● Same configuration
… and…
● Same security and compliance controls
© Confidential
● Automated and traceable authorizations for promotion of code to production
● Role-based access controls that acknowledge when DevOps personnel have access to production systems and document the specific use cases
● Encryption and logical access controls which essentially “lock-out” unauthorized access (to systems or data)
Adapted From Brightline https://www.brightline.com/2012/12/auditing-devops-developers-with-access-to-production/)
Auditors/Compliance want to see
© Confidential
● File integrity monitoring (and alerting) on changes to production code versus the traditional focus on critical operating system executable
● File access monitoring on the source code itself with appropriate alerting
● Extensive logging and daily, if not real-time, log review of the above data sources
Adapted From Brightline https://www.brightline.com/2012/12/auditing-devops-developers-with-access-to-production/)
Auditors/Compliance want to see
© Confidential
● Single source for all change requests
○ No change is deployed without a request
○ Intent is that most problems can be diagnosed via the change reporting systems
● Clear separation of roles between developer and deployer
○ Or at least developer cannot deploy without review
● Clear metrics to demonstrate improvement
○ Mean Time To Recovery
○ Percentage of failed deployments/changes
Goals for delivery automation
© Confidential
Dev, Test, & Prod Environments
Code Review
Infrastructure Source Code
Goal automate documentation of this
InfrastructureCode Developer
deploy
Continuous Build & Unit
Test
Config, Release, Deployment
commit on branch build
check
approval
tests pass
© Confidential
Part IIChef and Access Management
© Confidential
● Access control is determined by “gates” which can be opened and closed by human or automated processes
● SSH gates access to production machines by developers and code
● Secrets gate access to production services by developers and code
○ Access to a secret constitutes access to a production service
The Importance of SSH and Secrets
© Confidential
State of Chef + SSHCombining users + sudo cookbook is a popular solution.
Workflow
● Create a data bag for your users and place it in a my_users cookbook
● Create JSON entries for each user w/ group membership and public SSH keys
● Upload the data bag and cookbook to Chef serverInclude the sudo recipe in my_users cookbook to set passwordless sudo for your users
© Confidential
State of Chef + Secrets
chef-vault + IAM is a popular solution
workflow
● Create IAM JSON file defining access permissions to S3 buckets
● Script out creating IAM users/groups/policies for specific access to a set of secrets
● create a vault and encrypt the JSON output of the previous step, tied to a specific Chef query
© Confidential
State of Chef + Audit
Chef Analytics
● Tracks changes to Chef objects (clients, nodes, etc) and who made them in real-time
● You write controls into your recipes, compliance tests that report back to analytics after Chef run
Guardrail
● SAAS that tracks changes to infrastructure in real-time, can trigger alerts on specific changes
© Confidential
We will create a simple, but real infrastructure using Chef, Conjur, and Docker. Infrastructure activity and access will be fully managed and audited.
Workshop Flow - Overview
Chef Chef Server is the definitive source of server configuration.Chef deploy and validation keys provide access to Chef Server.Chef provides audit and verification of server installs.
Conjur Provides User, Group, and Host identity.Stores and distributes secrets.Manages SSH access to machines.Records audit of all permission changes, access to secrets, and access to hosts, plus custom audit records of cookbook pushes.
Docker Used to run infrastructure machines:deploy can push cookbooks to Chef Server. Accessible only to ops.sentry application server. Accessible to ops and developers.
© Confidential
◁ Move to the source directory and fetch all the code
Workshop Flow - Get the Code
cd chefconf15git submodule update --init --recursive
Submodule 'cookbooks/foundation' (https://github.com/conjurdemos/chefconf15-base.git) registered for path 'cookbooks/foundation'Submodule 'cookbooks/sentry' (https://github.com/conjurdemos/chef-sentry) registered for path 'cookbooks/sentry'…
© Confidential
◁ Create an ops group to own the key
◁ You’re a member of security_admin, so you’re also a member of ops.
Workshop Flow - Create the “ops” group
conjur group create --as-group security_admin ops
{ "id": "ops", "ownerid": "demo:group:security_admin", "gidnumber": null, "roleid": "demo:group:ops", "resource_identifier": "demo:group:ops"}
© Confidential
◁ Store the public key of the ‘demo’ user (that’s you) in Conjur.
Workshop Flow - Load your Public Key
conjur pubkeys add demo @~/.ssh/id_rsa.pub
Public key 'demo@ip-10-97-161-35' added
© Confidential
© Confidential
◁ Copy deploy.pem and validation.pem from your Chef Server
◁ Save to the local files : deploy.pem and validation.pem
◁ deploy.pem has permission to manage cookbooks
◁ validation.pem used to register new machines with Chef
Workshop Flow - Download the Chef Keys
© Confidential
Workshop Flow - Store the Chef deploy.pem
conjur variable create --as-group ops -m application/x-pem-file \chef-server/deploy/private_key
cat deploy.pem | conjur variable values add chef-server/deploy/private_key
{ "id": "chef-server/deploy/private_key", "mime_type": "application/x-pem-file", "kind": "secret", "version_count": 0}
Value added
◁ Create the variable and store the key
© Confidential
◁ Store the Chef server validation key in Conjur
Workshop Flow - Store the Chef validation.pem
conjur variable create --as-group ops -m application/x-pem-file \chef-server/validation/private_key
{ "id": "chef-server/validation/private_key", "mime_type": "application/x-pem-file", "kind": "secret", "version_count": 0}
cat validation.pem | conjur variable values add chef-server/validation/private_key
Value added
© Confidential
© Confidential
◁ Foundation image cookbook is pre-loaded into the Chef server. Along with Conjur auth and RBAC, this is the foundation of trust in the infrastructure.
◁ Base OS.
◁ Packages - Chef, Conjur
◁ SSH access - Conjur SSH - login with your own private key, always
◁ /etc/conjur.conf + /etc/conjur-demo.pem - Verified TLS for everything
◁ Only public info! No secrets.
https://github.com/conjurdemos/chefconf15-base
Workshop Flow - Foundation Image
© Confidential
“Deploy” server builds on the foundation image
◁ .conjurenv → Chef validation.pem
◁ client.rb → Chef client configuration
◁ deploy.sh → Docker entrypoint, uploads all cookbooks to Chef server. Cookbooks should be volume-mounted to Docker
Workshop Flow - Build Deploy Server
docker build -t deploy deploy/
© Confidential
Workshop Flow - Create Deploy Host
conjur host create --as-group ops deploy-01 | tee deploy-01.json
◁ Conjur - Create a Host identity for the Deploy server
◁ Assign ownership to the ops group
cat << ENV > deploy-01.envCONJUR_AUTHN_LOGIN=host/deploy-01CONJUR_AUTHN_API_KEY=$(cat deploy-01.json | jsonfield api_key)ENV
© Confidential
◁ Launch the deploy host
◁ Host does not have access to the secret … yet
Workshop Flow - Test the Deploy Host
docker run --env-file deploy-01.env --entrypoint conjur deploy env check
error: Some variables are not availableCLIENT_PEM: unavailable
© Confidential
◁ Give access to the Chef key to the deploy host
Workshop Flow - Grant Access to Chef
conjur resource permit variable:chef-server/deploy/private_key \host:deploy-01 \execute
Permission granted
© Confidential
◁ Run the deploy box as a daemon. We will use SSH to access it.
Workshop Flow - Launch the deploy box
docker run -d --env-file deploy-01.env \--name deploy \-v $PWD/cookbooks/:/cookbooks \-p 2200:22 \deploy
*** Running /etc/my_init.d/10_foundation-configure.sh...[2015-03-28T20:56:58+00:00] INFO: Forking chef instance to converge...[2015-03-28T20:56:58+00:00] INFO: *** Chef 12.0.3 ***…*** Runit started as PID 336Mar 28 20:57:09 613420620e92 syslog-ng[345]: syslog-ng starting up; version='3.5.3'
docker logs -f deploy
© Confidential
© Confidential
◁ Login to the deploy host
◁ On the deploy host, display user info. Group membership is controlled by Conjur.
Workshop Flow - Login to the deploy host
ssh -p 2200 localhost
demo@e9de200cc9da:~$
id
uid=1100(demo) gid=50000(conjurers) groups=50000(conjurers)
This icon means, run this command from inside a container
© Confidential
◁ Use the deploy host to list cookbooks
◁ Group ‘conjurers’ is whitelisted to passwordless-sudo this command.
Workshop Flow - List Cookbooks
sudo list-all
apt 2.7.0build-essential 2.2.1conjur-client 0.3.1foundation 0.1.1sshd-service 1.1.0terminal-login 0.2.3yum 3.5.3
© Confidential
◁ Exit the container with Control-D and navigate to the cookbooks directory
◁ Rename the sentry folder to end with your initials
Workshop Flow - Build the sentry Cookbook
cd cookbooks
mv sentry sentry-yourname ← changeme!cd sentry-yourname
© Confidential
Edit metadata.rb
◁ Change the cookbook name to end with your initials
◁ Increment the version to 0.2.0
Workshop Flow - Build the sentry Cookbook
© Confidential
◁ Edit recipes/default.rb
Workshop Flow - Build the sentry Cookbook
© Confidential
◁ Login to the deploy host
Workshop Flow - Login to the deploy host
ssh -p 2200 localhost
demo@e9de200cc9da:~$
© Confidential
◁ This sequence will vendor the Berksfile in /cookbooks/sentry, then upload the cookbook with dependencies.
Workshop Flow - Upload Cookbooks
sudo upload sentry-yourname
# ./upload-all /cookbooks/sentry/Resolving cookbook dependencies...Fetching 'sentry-keg' from source at .Using build-essential (2.2.2)Using python (1.4.6)Using yum (3.5.3)Using sentry-keg (0.1.6) from source at .Using yum-epel (0.6.0)Vendoring build-essential (2.2.2) to /cookbooks/sentry//.vendor/build-essentialVendoring python (1.4.6) to /cookbooks/sentry//.vendor/pythonVendoring sentry-keg (0.1.6) to /cookbooks/sentry//.vendor/sentry-kegVendoring yum (3.5.3) to /cookbooks/sentry//.vendor/yumVendoring yum-epel (0.6.0) to /cookbooks/sentry//.vendor/yum-epelUploading sentry-keg [0.1.6]Uploaded all cookbooks.
© Confidential
◁ Audit activity shows access to secrets, as well as the upload event
Workshop Flow - Inspect the audit
conjur audit all -s
[2015-03-30 18:28:25 UTC] demo:host:deploy-01 checked that they can execute demo:variable:chef-server/deploy/private_key (true)[2015-03-30 18:28:30 UTC] demo:host:deploy-01 created resource demo:chef-cookbook:foundation owned by demo:host:deploy-01[2015-03-30 18:28:31 UTC] demo:host:deploy-01 reported cookbook-deploy:upload; message: Uploaded cookbook 'foundation' with command options: --include-dependencies
© Confidential
◁ Run a MySQL server
◁ Display the connection info
Workshop Flow - Launch database server
cd ~/chefconf15/databasesource start.sh
Launching databaseDatabase launched! Check $DB_HOST and $DB_PASSWORD
echo $DB_PASSWORD
eee5377d52c6b033
© Confidential
◁ Store the DB password in Conjur
Workshop Flow - Store the database password
conjur variable create --as-group ops \sentry/mysql/password \$DB_PASSWORD
{ "id": "sentry/mysql/password", "mime_type": "text/plain", "kind": "secret", "version_count": 1}
© Confidential
◁ Return to $HOME/chefconf15
Workshop Flow - Chdir
cd ~/chefconf15
© Confidential
◁ Create the host
◁ Permit secrets access
Workshop Flow - Configure Sentry host
conjur host create --as-group ops sentry-01 | tee sentry-01.json
cat << ENV > sentry-01.envCONJUR_AUTHN_LOGIN=host/sentry-01CONJUR_AUTHN_API_KEY=$(cat sentry-01.json | jsonfield api_key)ENV
conjur resource permit variable:chef-server/validation/private_key \host:sentry-01 \execute
conjur resource permit variable:sentry/mysql/password \host:sentry-01 \execute
© Confidential
◁ Build the image
Workshop Flow - Build Sentry app
docker build -t sentry sentry/
© Confidential
◁ Run the Sentry box as a daemon.
Workshop Flow - Run Sentry
docker run -d --env-file sentry-01.env \--name sentry \-e DB_HOST=$DB_HOST \-e COOKBOOK_NAME=sentry-keg \-p 2201:22 \-p 8080:9000 \sentry
*** Running /etc/my_init.d/10_foundation-configure.sh...[2015-03-28T20:56:58+00:00] INFO: Forking chef instance to converge...[2015-03-28T20:56:58+00:00] INFO: *** Chef 12.0.3 ***…*** Runit started as PID 336Mar 28 20:57:09 613420620e92 syslog-ng[345]: syslog-ng starting up; version='3.5.3'
docker logs -f sentry
Change this to match your cookbook name
© Confidential
◁ Login to the sentry box
◁ On the sentry box, run chef-client
Workshop Flow - Configure sentry
ssh -p 2201 localhost
… chef-client run happens here
$ sudo conjur env run -c /.conjurenv -- sudo -E chef-client \-c /etc/chef/client.rb \-o $COOKBOOK_NAME
© Confidential
◁ You can now view the Sentry UI on port 8080 of your endpoint.
Workshop Flow - View Sentry UI
© Confidential
● Create a developers group and a user.
● Try to deploy, as this user.
Workshop Flow - Add New Team Members
© Confidential
Part IVRegulatory Environments and
Chef Server
Presented by Chris Farnham, Novartis @chrisfarnham
© Confidential
DevOps Mission
Development Operations strives to shorten and amplify feedback loops, thereby bringing software development and IT operations closer to end users.
© Confidential
Macro Trends
● Self-service● Virtualization● Automation● Manage sets of assets not individual assets● External collaborations● Data Explosion● Globalizaton● Mobile devices● Internet of Things
© Confidential
In order for Chef to manage system configurations in our regulated environment it needed to be regulated itself. This is daunting; the following high-level strategy really helped me deliver
First: Describe the system and list all activities
Second: Go through each activity and ask:
◁ Can I declare this activity out of scope?◁ Can I reuse (or map it to) a pre-existing policy or activity?◁ Can I leverage automation (in lieu of documentation)?◁ Document the activity at an appropriate level
The above is generally ordered best to worst (but ultimately not that bad)
Chef in a regulated environment
© Confidential
◁ Gather product documentation
◁ List all components and integrations to the system
◁ List all activities with the system with careful attention to status updates
◁ Describe and list the roles and responsibilities associated with the system
Example activities:
◁ Create/Modify/Update Cookbooks in Chef Server
◁ Bootstrap (add)/Remove Node
Protip: Declare a “maintenance” activity with a change tracking process of “none” ; because nothing changes when this activity occurs
System activities
© Confidential
Always look to manage scope and be aware of which integrated systems are particularly sensitive (read: important) and which integrated systems are managed by other groups
◁ Declare which products and systems are “in scope”◁ Explicitly declare what is “out of scope”
Examples:
◁ Repository development activities (e.g., Berkshelf, cookbook writing) were declared “out of scope”
◁ Repository changes (e.g., upload, remove) were declared “in-scope”◁ Supporting infrastructure (e.g., monitoring, Jenkins) was declared “out of
scope”
Managing scope
© Confidential
Documentation and processes already existed for many things, we just reused them.
Examples:
◁ Server provisioning and retirement ← On-premise◁ Incident management (we just needed to declare roles and list points of
contact)◁ Backup and recovery
Reusing existing activities
© Confidential
Leveraging automationAutomated processes, where everything happens in a traceable and auditable
manner (driven out of assets managed in source control) provide great flexibility and insulation within the change control process.
Example doc excerpt: “change tickets must be filed for all medium risk updates unless performed via source control driven automation.”
© Confidential
Figuring out what should be documented at a high-level and what should be detailed is an art but important
Control related documentation is usually tracked and signed (a rigorous process). Do yourself a favor and only put core, less frequently changed information within the core doc (the operations manual). Then make heavy use of referenced “working instructions” that can be updated without going through a heavy process
Documenting activities
© Confidential
◁ Really get to know the compliance group
◁ Ask to attend meetings, request 1 on 1s, have lunch with them
◁ Goal is to learn their language, concerns and establish a personal relationship
◁ Learn the groups, communication paths, workflows and activities within the organization
◁ How are change management decisions made?
◁ Who are the operational and compliance stakeholders?
Homework
© Confidential
We hope you learned A LOT today!
Next Steps...> Test drive our on-demand demo at conjur.net > Check out our dev docs at developer.conjur.net> Set up a live 1:1 demo to learn more> Subscribe to our blog & monthly newsletter> Follow us @conjurinc
Thank You!