Join the conversation #devseccon

By Fraser Scott & Nick Drage

Epic Battle: DevOps vs Security

Fraser:Hello everyone and welcome to Epic Battle: DevOps vs Security.

Fraser:Hi I’m Fraser, aka Player 1

Fraser:I’m an ex-sysadmin, devops engineer. Now a cloud secops engineer at capital one. And I’m a DevOps fanboi.

Nick:Hi I’m Nick aka Player 2

Nick:I’ve been sysadmin, security operations, pentester, and security practice architect, I’ve been doing this since you were a teenager

Nick:While I’ve enjoyed all the talks here today, and have the greatest respect for the speakers, they’re all wrong.DevOps and DevSecOps are just another silly management fad that just results in poorly planned and hastily executed garbage like the Internet Of Things.

Nick:Brian Krebs DDoS

Fraser:DevOps and DevSecOps aren't management fads, they're a grassroots movement that comes from engineers finding out that working together let’s them get shit done faster than not working together. DevOps has just a hard time convincing management as much as it does traditionalists.Look, DevSecOps is just DevOps + Security. [CALMS slide] Some argue that Security is automatically included in DevOps, and it should be, but it’s nice to have a term that includes security so it doesn’t get forgotten about, hence DevSecOps.

Nick:Yes, but the benefits of DevOps could easily be achieved if Developers, Operations, and Security just talked to each other.

Nick:Use online meetings

Nick:Stop sitting in their own silos. And that way you’d still maintain separation of duties , access with lowest level of privileges, and all the other benefits of dividing roles.

Dev Ops

Sec

DevOpsSec

Fraser:Right. Exactly! That’s the cultural aspect. DevOps doesn’t mean one unicorn engineer doing all the things, it means breaking down the traditional silos. You might end up with a single functional team that has a mixture of software engineers, operations engineers, QA and security. Or maybe separate teams working together. The trick is getting the right people involved earlier on.

Nick:But if you still have multiple teams, then nothing has changed. People will still book pointless meetings, email word documents around, argue about whose responsibility something is, not read each other’s documentation etc. ( That guy, in the top left, with his black tshirt and earphones on… security )

Nick:Multiple teams, nothing has changed, just gimmicks.

Nick:Some more gimmicky than others

Fraser:Possibly, yes, but it doesn’t have to work that way. Think of teams as microservices. They can have well defined functions and “interfaces”. If you're constantly winging it or having to read 100 page documents you're going to have a bad time. Automation has a big part to play here because it removes the typical human barriers that introduce slowness and latency. Instead of emailing some team a document containing changes for review, a git commit could trigger automated tests that effectively carries out the decision making process the person would have made. You’re breaking down the traditional silos through automation. Instead of asking “how do i?”, go and look at the jobs and code.

Nick:But automation is just a way of breaking things at scale.

Nick:If you screw something up in an automated way, there’s nothing to stop it propagating out and becoming a massive disaster.

https://xkcd.com/327/

Fraser:Yes it’s true that things can go wrong, and with heavy automation things can go very wrong at a large scale very quickly. But even a single manual command can bring down an entire service, and that’s without any automation. Something like a DROP TABLE statement for example. But with everything-as-code etc., you can also automate the detection and response to problems. Automatically rollback changes whether they’re code or infrastructure. Sure, there may be some brief downtime, or a percentage of users get some errors for a while, but in order to maintain velocity and innovation you have to accept some risk of failure.

Nick:In security, failure is not an option! It’s one thing if a user can’t get their fix of cat pictures for 5 minutes, but it’s a whole other issue for their personal or financial details to be compromised by some random hackers.

I

<3 failure

Fraser:Agreed. Loss of availability is a best-case scenario for a security incident. But some failure is inevitable. The question is, do you accept it and learn to leverage it to your advantage, or do you pretend it doesn’t apply to you until one day everything is on fire. There have been an increasingly large number of breaches lately, even from within organisations that had relatively good security. Why?

Nick:Then they didn’t security hard enough. They probably thought they had excellent security because they had blinky boxes and were PCI compliant at some point in the past, but in reality if they were compromised there was a weakness. They just need to get more staff and put in more effort… security is a hungry process that needs to be fed.

Fraser:Just throwing more staff at ineffective security processes doesn’t work. That’s like trying to find a needle in a haystack by adding more hay. Yes, their security wasn’t perfect. But it also never could be.

Nick:No! Hackers only need to find one way in. We need to be perfect, at all times - once the hackers are in, we’re done for.

Nick:You don’t understand the threats we all face, you don’t have the cyber threat intelligence to understand the cyber threat actors in the cyber threat landscape in the cyber threat space.

Nick:I’ve got auditors to contend with, they’re a threat too. I’ve got to be secure, and compliant.

Nick:I’ve got to minimise the attack surface of every interface thats under threat, and as everyone and everything is a threat, that’s the attack surface of everything. I haven’t got time for fads.

Nick:I’ve got to protect the confidentiality, integrity, and availability of all the company’s assets.Company assets which undoubtedly come with default credentials and configurations, those assets are a threat.And I’ve got to find all those assets, which is only the first entry in the SANS Top 20.

Nick:You’re a developer, you only have to contend with passive failure. A hard drive fails or someone accidentally pushes some broken code. I'm up against the best sentient opponents the Internet has to offer. Opponents that react and adapt.

Nick:To do that I need to manage Firewalls

Nick:Internal firewalls

Nick:Ingress and egress filteringApparently allow users to talk to each other

Nick:Hardened proxies

Nick:Data Loss Prevention

Nick:End Point Threat Management

Nick:All this data needs to go into a Security Incident Event Management

Nick:Which needs to be analysed by SOC analysts and hunt teams

Nick:And I need a Data Lake to store it all. A Data Lake. Fraser, I don’t even know what a Data Lake is!

Fraser:Yes, security is hard. It's cat and mouse. But it's also unrealistic for security to control the entire environment. At that point security will always be a blocker. Security needs to be part of everyone’s job. I’m not saying that the security team is redundant, you’ll always need subject matter expects. But security being a constant blocker just won't scale and would end up stopping the organisation reaching it's goals and targets. Either that or you end up with shadow IT.

Nick:*I’M* Security, capital S, I’m the only one that knows best and gets to say “no” when needed. How do I say no to people if everyone is doing security?

Department of

NoCLOSED

Fraser: You don’t. The department of “No” has been disbanded.

Nick:If I don’t say no, the hacker’s will get in and then it’s game over, everything’s on fire.

Fraser:It’s inevitable that at some point somebody’s going to get in, in some way. You have to accept some level of risk. The only way you’re going to harden your environment to that extent, is by stifling innovation and progress. We all know that the only way to fully secure a device is to switch it off and bury it in concrete. There's more than just prevention, there's detection, containing and remediation. Defense in depth right? An may one need to find one way in, but once they're in they have to not make any mistakes for fear of being detected.

Nick:If you catch me at a good time I might agree with you on risk, but risk analysis requires deep contemplation over a long period of time by experienced and expensive risk consultants like myself.

Nick:It's impossible to build and deploy at the speed DevOps people want, and the same time as stay on top of managing risk within the organisation.

https://landing.google.com/sre/interview/ben-treynor.html

Fraser:Let’s step back a minute. Google has the concept of an “error budget”. The idea is that for a given service, the SLA is determined, and the error budget is defined as 1 minutes the SLA. Your spend is then the error budget minus the availability. So, let’s say for a given service, if at the end of the quarter is it shown that the service is well within the SLA, so has had very few availability issues and effectively hasn’t spent its error budget, then question is.. Could they have pushed faster or innovated more? That budget has basically gone to waste. But if they’d gone over budget, they’d probably have to slow down their deployments, writing more or better tests etc. This way development and ops are both working off the same incentive - spend that budget, and exactly that budget. No more, no less.

Nick:SLAs are easy, things are up or they’re not.

Nick:Risk is complex, you can’t analyse this complexity with a tool.

risk budget

Security meets Velocity

Fraser:Well, what if you could define something like a “risk budget”? If you’re have to accept some level of risk anyway, why not find a way to measure it and use that to align security and deployment velocity. Again, automation and measurement is key here.

Nick:OK, seeing as I’m winning by so much I’ll indulge you… how exactly would you do that?

Existing risk measurements

Documentation coverage Unit Test coverage Unit Test results Integration Test coverage Integration Test results

Fraser:It probably depends on the organisation, but you could use existing measurements such as documentation coverage, and unit and integration testing coverage and results. The assumption here is that well tested, well documented code is probably of a higher quality and therefore a higher security quality.

Other metrics

Continuous security testing Code-driven Threat Modelling Level of security engagement Pentest results % successful gate passes

Fraser:You’d probably also want to measure security testing coverage and results. Maybe you’re also generating threat modelling coverage and results, so that could be measured. Perhaps you could measure the level of engagement the project has had with the security team. A good annual pentest result could add a bonus to the risk budget for the service, and bad pentest result could reduce the risk budget.

Fraser:So let’s say a very important service has a risk budget of 1000 points per quarter. Currently their deployments, based on all the above measurements, are coming in at a “cost” of 400 points per production deployment. That would mean they could release to production 2 times per quarter. Not very agile. If they wrote more and better security tests, expanded their threat model coverage and involved the security team a bit sooner in the lifecycle, maybe their cost would go down to 155 points. That would mean they could deploy to production every two weeks.

Nick:So what you’re saying is that if a platform is provably insecure then it can be altered less, but if it’s provably secure, DevOps can deploy more often and add value?

Fraser:Not provably insecure or secure, but provably less secure or more secure. If you think about it, developers work with value. In fact, value is fundamental to devops. They’ll write a feature because there is a hopefully high likelihood that the feature will have a positive impact on sales or number of users etc, or whatever the key business metrics are. That sounds a lot like risk, just that the impact is expected to be positive not negative.

Risk= Impact x Likelihood

Value= Impact x Likelihood

Nick:OK, so if I’m going to buy into your way of thinking, which I might have to because we only have twenty minutes, risk is negative impact times likelihood, and value is positive impact times likelihood. So you’re saying that value-driven DevOps and risk-driven Security are basically two sides of the same coin.

Fraser:Yes, exactly! It’s all very YinYang. Two opposing sides that coexist. Both sides cannot exist without the other.

Sometimes,they are outto get you

Fraser:Devops has to take into account the negative side of human nature to succeed. It cannot ignore security - security is everyone’s responsibility. It cannot pretend that there aren't people out there who will actively try to attack and compromise their systems and data.

Trust, But Verify

Fraser:Security has to utilise the positive side of human nature to succeed. Yes there are people out there trying to get you, but not everyone is. Use a trust but verify model to scale out security controls.

Nick:So what you’re saying is that we can automate our security mindset into development and operations, so we’re all thinking about security, rather than the separate security team only enforcing security when they notice your project? So that would maybe free up security to focus on thinking about the bigger picture rather than be bogged down in millions of implementation details. They could focus on scaling out security not hoarding it.

So, according to the analogy I’ve been trying to force on this entire presentation, we both win

Nick:...we all win \o/

To DevOps: Be patient with Security. They have to go through a philosophical paradigm shift.

To Security: Be patient with DevOps. They have the best intentions and some pretty cool ideas.

To Everyone: Remember, empathy goes both ways.

Fraser - first lineNick - second lineFraser - third line

Join the conversation #devseccon

Thanks!

Fraser Scott @zeroXten

Nick Drage @SonOfSunTzu

Image credits: https://github.com/zeroXten/devseccon2016

Thanks!