Faster Secure Software Development with Continuous Deployment - PH Days 2013
-
Upload
nick-galbreath -
Category
Technology
-
view
463 -
download
3
Transcript of Faster Secure Software Development with Continuous Deployment - PH Days 2013
Faster Secure Software Development with Continuous Deployment
Nick Galbreath IPONWEB [email protected]
PHDays May 24, 2013 Moscow, Russia
Follow Along
Latest version posted online:
http://slidesha.re/199wo7Qhttp://client9.com/20130524
Nick Galbreath (@ngalbreath)
! Spoken at Black Hat, DEFCON, OWASP ! Book on cryptography ! but really... ! Engineering Management and Software
Development for high growth startups. ! Personal site http://www.client9.com/
libinjection
! Author of libinjection ! A very different way of doing SQLi detection ! Right now in another room, Vladimir
Vorontsov is showing how to bypass it (to be fixed shortly)
! Check it out and file bugs on github http://libinjection.client9.com/
! Found a bypass? What database and query is needed to exploit it?
RU ♥ SQLi
IPONWEB
! customized online advertising infrastructure and exchanges
! engineering offices in Moscow, with business offices in London, New York and Tokyo.
! YEAH IN MOSCOW ! YEAH WE ARE HIRING ! Send email to [email protected]
Well that's a bold statement...
Fixing Security byFixing DevelopmentUsing Continuous Deployment
and here's another
For web applications, our release-based software development lifecycle is still based on a pre-Internet model and is harmful to organizations andparticularly harmful for security.
What needs fixing?
! SQLi dropped from #8 to #14 in the latest White Hat "The State of Web Security" report. Good news, right?
! This means SQLi is only 7% of websites. That's 1 in 15.And this is the #14 vulnerability!
! And time to fix was on average 196 days. That's embarrassing.
Veracode claims 32% of incoming web applications have SQLihttps://info.veracode.com/state-of-software-security-report-volume5.html
https://reg.whitehatsec.com/WPstats0513
Even worse...
! Number 1 driver to fix security problems...compliance.
! Number 1 reason to not fix security is...compliance.
! Not.. ! keeping our employees and customers safe ! protecting corporate interests. ! improving quality ! being good at what we do.
Security Products #1 .. in security bugs VeraCode: State of Software Security, V4 December 2011
Security Product 74% Fail Rate
Let's Just Give Up
! “You could spend all your resources chasing such things as this,” William Ribich, the former president of Technology Solutions Group [ QinteliQ ], said in an interview in January. Ribich, who retired in November 2009, shortly after the discovery of a major data theft, said he needed to balance the uncertain risk that the hackers could use what they stole against a growing shopping list of security products and consulting fees.
! "You finally have to reach a point where you say ’let’s move on,’” he said.
http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html
^^^ The Russian and Chinese hackers did not move on ^^^^.
I would call that broken
! But preventing SQLi isn't a technically hard problem.
! And most security patches are very small. ! How did we get here?
Software Product Model
! Code flows between functional groups ! Product Managers spec code ! Engineers write code ! QA engineers tests code ! Release engineers package code ! Operations runs code ! ... and Security does something too.
High Distribution Cost
! The Software Product Model is designed for software where the cost of distribution is high. "High" might be financial, risk, time, resources, customer annoyance. ! Retail, physical product, CD/DVD ! Embedded of Exotic Hardware ! Safety, Medical or Defense Systems ! Operating Systems (desk or phone) ! Homework (1-time deploy)
Software Product Lifecycle
Change to Production
Web Applications Year 2000
! Mostly followed Software Product Model since that's all we knew.
! High barrier to entry ! Specialized Hardware, Software and
People needed to get started. ! Lots of engineering needed to keep
things running. ! (side note: CERT/CVE started in
1999)
True Story #1
! "Can't push out the spelling error fix – it's too risky"
! "That code as already been through QA, it's locked down."
! "Product has to prioritize that change, else we aren't touching it."
True Story #2
! We'll do an iteration, where we try to fix as many things as possible.
! This won't be a scheduled iteration,it will be done because so many things are piled up.
! So the spelling error will get fixed... uhh, who knows when.
Web Applications 2013
! Almost no barrier to entry ! Commodity hardware ! Programming not that hard ! Scaling problems can
be mostly outsourced (mostly)
Cost of Distribution 2013
! Frequently no compile step or it's very fast.
! Moving to production a few kilobytes or megabytes of code over 1Gbps, 10Gbps link.
! In other words... free
Failure is very different however
! Most web applications are data-driven. ! Frequently have social features, APIs,
user-generated content. ! Failures might be due to algorithmic
problems... but... ! Most likely to due to user input, bad data
in database or operational load. ! this means data in past can cause
problems in the future.
Releases and Problems
! When a web-release goes out, and has problems....
! Next week is spent tracking down who changed what, where.
! Re-QA ! Re-Push ! meanwhile new code is piling up.
When SPM meets Web Apps
! A long time between code being written and code being released.
! Might be weeks or months ! Feedback loop between code-in-dev
and code-in-production is broken ! When security or bug reports come in,
the author is likely on a different project.
Hypothesis
! It is impossible to simulate the production environment in development, either due to operational differences or data differences.
! No amount of QA or Security Testing can prove you don't have bugs, vulnerabilities, or cause severe operational problems.
! You have bugs and vulnerabilities,right now, in your application.
Impedance Mismatch
! Easy to write code, + ! Long release cycles + ! Security as an end-of-line or out
of band process == ! no one cares
! Something is going to break, and most people don't care.
Security Problems Will Happen
MTTF ≪ ∞
So the Answer is...
! Going slower? I'm sure your boss will love that suggestion
! More steps and process? In other words, slower.
! Asking for more people? Sure but good luck hiring them. Doesn't scale.
! Asking for more products? Since the others have worked so well.
Continuous Deployment
! Also known as Continuous Delivery. ! A System of Software Production
Characterized by Numerous Small Changes the Production Environment, initiated by the author of the change.
You change it, you push it to prod.
Deployment != Feature Release
time
chan
ge
New code goes out all the time. New features get turned on in a separate process.
"Writing Software"
! Software Developers think their job is writing software.
! And so, they love to make things perfect before anyone else sees it.
! Impolite: "data hiding" ! code is hiding on developer's computer ! or on some branch ! in other words invisible until it's ready.
Actually
! The software engineer's job is actually writing running software, that works well.
! This idea is so alien, that companies have to remind the engineers of this.
Rackspace Haiku
writing code is hardif you cannot deploy it does not matter
@paulvx from DevOpsDays Austin 2013
Facebook's Analog Labs Poster
"Move Fast and Break Things....
Except "Push" (deployment system)
via http://mitadmissions.org/blogs/entry/move_fast_and_break_things
Etsy Mantra Just Ship
Today's goal
! but for today the goal is getting the developer to care about their code in production.
! If you don't have that, I don't think you can really solve security problems.
How does this work?
! Really? Developers push their own code out?
! How is this not a disaster. ! How is this not a security disaster?
The Deploy Button
! What is you had a button that said "DEPLOY" ! That pushed to production, whatever
is current in your source control system. ! And took about a minute ! The change and who pressed the
button is logged, but that's it.
Part 1: Fear
! No one is going to push it ;-)
! Meanwhile code is piling up
Real example: A new hire I had at Etsy was afraid of deploying an HTML change that they made. "But I don't want to break the site!"
Part 2: First Push
! Someone brave will press the button ! And very likely the site will explode,
and a rollback will need to be done. ! They'll know since someone else will
have told them.
Part 3: With Graphs
! Let's get all those operational graphs out in the open. And put them right next to the button.
http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
Part 4: Push #2
! Repush ! Site might still explode ! But the developer is aware and
can rollback.
Take 5: Isolation
! Hmmm, the developer notices that in the change set, a million things are going out.
! Maybe just pushing out a smaller change will help isolate the issue.
Take 6: Success!
! Yes, the developer just pushed out some code and made the site better.
! The secret about continuous deployment is small changes that can be easily understood.
Take 7: Dark Pushes
! Now we got some bugs fixed, let's push a feature.
! First let's push out all the supporting files. Since they aren't being called, they do nothing and are safe to push out.
! Now everyone can see them
Take 8: Getting the feature live
! Instead of "all at once", we slowly ramp up a feature.
! if (user_id % 20 == 0): do new feature;
! we change change the percentage easily with another code push.
! or turn it down. Much nicer change log. ! While the site didn't explode, it's hard to see if
the feature is being used or not.
Take 9: Application Level Graphs
! Allow developers to instrument their code so they can see what is happening in production. ! Enter StatsD and other
UDP-based tools ! Enter centralized logging and in-
application method to make it easy to log problems.
Take 10: Communication
! So far good for one developer. ! To scale up, you'll need a system to allow
developers. ! IRC-like tools work well (e.g. "the push
channel") – skype, jabber, hangouts, etc
Along the way
! Expose production logs to developers ! Add in a staging-step where the code
goes to faction of the cluster, so developers can test with real traffic
! Try to make development closer to prod. ! Make "smoke tests" to catch basic errors ! Add syntax checkers to eliminate obvious
issues. ! Use static analysis to find bugs
Mistakes will happen
! Do postmortem analysis ! Everyone thought they were doing
the right thing at the time. ! "How can the environment be
changed to prevent this" and build tools to enforce it. ! (Rarely can you truly change people)
Sounds good, but what about...
That guy who pushes at 3am
! Courtesy and convention will converge very quickly when the site goes down at 3am and the developer starts getting calls ;-) ! Of hours pushes of course can
happen, when they notify operations.
What About Code Reviews?
! Yes, please do them. ! Nothing here prevents code reviews. ! In fact code reviews are easier since ! they are small ! they are in mainline not some branch
What about Security Reviews
! Please do them. ! Nothing here eliminates
architectural planning or review. ! This actually doesn't change the
SDLC very much.
What about Agile Methods
! (everyone seems to have a different idea of what Agile is but..)
! Agile methodologies typically work to improve the business spec / development cycle. (are you building what the customer wants)
! But doesn't address code deployment. ! They are complimentary practices.
What about Customer Service?
! "Don't they freak out with all the changes?"
! Remember: deployment != feature release ! Most deployments do very little from the
customer point of view ! Feature releases (frequently controlled by
ramp-ups or flags) always needs to be coordinated with product and customer service.
What about Compliance? PCI?
! Let me tell you about compliance... ! mechanism not policy ! compliance is a lot easier when it's done
every day instead of a once-a-year audit.
Back to Security
Obvious Benefit to Security
! Security patches can go out quickly ! You know this since they are now
just part of a normal development cycle and code goes out regularly. ! Why not clear out those low-priority
security problems?
More Importantly
! That Engineer who previously did not push code is now sensitized that their code has consequences and are responsive and empowered to fix it.
! It’s amazing how interested engineers become in security when you find problems with their code when they are able to fix it quickly themselves.
New Security Math
! Instead of focusing only on increasing MTTF, which will never be infinite ! more firewalls, more process, more magic
! You can focus on how fast can you detect faults, and how fast can you fix them.
! How low can you go? ! MTTD - Mean time to Detect ! MTTR - Mean time to Repair
Hack The Stack
! A side effect of this you now have tools to repurpose for security and monitoring of production ! Note that most changes are not
security problems.
Logging
! Due to allow developers to see application logging, it's now very easy to instrument the application to log security events. ! Or add logs to times when you are
under attack.
Graphing
! Make dashboards of ! SQLi and XSS attacks ! Every type of log-in failure ! Core Dumps ! Database Syntax Errors
Static Analysis
! You now have a place to insert them. ! Work with QA group to add more code
quality tests.
Post-Commit Checks
! Alert on when sensitive areas of the code are changed (auth, login)
! Alert on crypto usage (why is developer using MD5.. hmmm)
! Alert on your programming language's "dangerous functions"
This allows you to engage the developer at the start of the cycle.
Faster is Better
! You could do most of this in a normal release-cycle software lifecycle. ! The difference is you are finding
problems at the start instead of 10m before the launch and telling everyone to stop. ! The feedback loop works.
New Roles, Less Silos
! Developers: works with operations ! QA: works on building systems for
testing, to empower others to write better tests ! Release Engineering: tools to enable
code to flow faster ! Security: in-house consultancy,
secure-by-default architecture, monitoring
Getting Started
Goal: 50% reduction in deploy time
! Whatever your state of deployment is, no matter how many people are involved, no matter how long it currently takes, make a goal of cutting it in half. ! This is an easy sell to management
just on cost basis. ! Everything else flows from this.
Mechanism not Policy
! Strive for the fastest deployment mechanism for possible
! But you define the "continuous" in Continuous Deployment
! Yes, Etsy was 60+ deploys per day, with each having multiple authors.
! Current gig? we have rules of no more than 3 per week since our customer have asked for that, and only deployed at "low-tide"
In Other Contexts
In other contexts: Operations
! How fast can you deploy OS changes to you production environment? ! How fast can you deploy router
changes? ! How fast can you deploy patches to
the desktop
You probably don't do it that often since it's really painful and time consuming! That's exactly the problem.
In other contexts: software product
! here "production" might be getting code into the main branch and running automated build / test.
! It's the flow of code: little changes vs big.
In other contexts: silicon
! Continuous deployment already done for silicon! wut?
! Only small changes, with tests are allowed to be committed!
! Big changes are rejected.Learned the hard way that big changes are completely unmanageable
One more thing...
Your Attackers Use Continuous Deployment
Continuous Deployment Start with 50% Reduction Build the Deploy Button
Thanks [email protected]
http://client9.com/20130524