© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Don “Beetle” Bailey, AWS Security

Josh Du Lac, AWS Professional Services

October 2015

SEC308

Wrangling Security Events

in The Cloud

What to expect from this session

• Tactical follow-on to previous talks

• Concrete examples of potential events and how you can

handle them

• Ideas for increasing security agility

• Specific AWS mechanisms to leverage

• More than 1 way to catch a cat burglar, so reinvent as

needed

• Relevant resources, including docs, code, and partners

“Intrusion Detection in the Cloud” redux

• AWS-specific areas to monitor for security-concerning

events

• Prerequisites

• Key concepts, such as security role, write-once storage

• Key services to leverage, events and behaviors to look

for

• Example detection of key configuration changes,

resource usage anomalies

• YouTube search “Intrusion Detection in the Cloud”

“Incident Response (IR) in the Cloud” redux

• Ensuring your existing IR process considers AWS

• More prerequisites

• Mechanisms for mitigation and investigation

• Tactics specific to AWS IR, such as constraining exposed

AWS credentials

• Tactics analogous to traditional IR, modified for AWS, such as

Amazon EC2 instance memory dumping, analysis

• YouTube search “Incident Response in the Cloud”

Security event wrangling = Response in depth

• Types of security events

• Detect -> Recover

• Investigate -> Protect

• Leveraging AWS mechanisms for increased security

agility

Example events of concern, signatures

• Configuration changes that impact ability to detect or

understand events

• Activities that are inconsistent with expectations

• Activities that violate policy

• Resources no longer available

• Resources more available than desired

• Event detection signatures != commercial product, and

may require careful thought vs. operations to develop

Protect, detect, react, recover, etc.

Protect

Detect

Recover

Investigate

AWS = Agility for security geeks

• Ability to programmatically inventory environment—knowing what you need to protect is key

• Awareness of what’s happening, what’s changing, from

AWS API activity to application behavior

• Detection and alerting mechanisms, freedom to create

and flexibility to configure and tune what’s appropriate

for YOU

• Analysis and response, via the same platform, natively

or with AWS partner solutions

AWS CloudTrail

• Records AWS API calls for your account and delivers log

files to you.

• Turn it ON!

http://docs.aws.amazon.com/awscloudtrail/latest/usergui

de/cloudtrail-user-guide.html

CloudTrail events

• A record in JSON format that contains information about

requests for resources in your account.

• Describes which service was accessed, what action was

performed, and any parameters for the action.

• Helps you determine who made the request.

• The event data is enclosed in a Records array.

http://docs.aws.amazon.com/awscloudtrail/latest/usergui

de/send-cloudtrail-events-to-cloudwatch-logs.html

Example CloudTrail event"Records": [{

"eventVersion": "1.0",

"userIdentity": {

"type": "IAMUser",

"principalId": "EX_PRINCIPAL_ID",

"arn": "arn:aws:iam::123456789012:user/Alice",

"accountId": "123456789012",

"accessKeyId": "EXAMPLE_KEY_ID",

"userName": "Alice"

},

"eventTime": "2015-03-24T21:11:59Z",

"eventSource": "iam.amazonaws.com",

"eventName": "CreateUser",

"awsRegion": "us-east-1",

"sourceIPAddress": ”55.55.55.55",

"userAgent": "aws-cli/1.3.2 Python/2.7.5 Windows/7",

"requestParameters": {

"userName": "Bob"

},

"responseElements": {

"user": {

"createDate": "Mar 24, 2015 9:11:59 PM",

"userName": "Bob",

"arn": "arn:aws:iam::123456789012:user/Bob",

"path": "/",

"userId": "EXAMPLEUSERID"

}

....

CloudTrail OFF"userIdentity": {

"type": "IAMUser",

"principalId": "AIDAI5WIMUDR2UZUI62VO",

"arn": "arn:aws:iam::000123456789:user/reinvent-sec308",

"accountId": "000123456789",

"accessKeyId": "AKIAIRAHHRD3PHLUFJLQ",

"userName": "reinvent-sec308"

},

"eventTime": "2015-09-23T00:41:45Z",

"eventSource": "cloudtrail.amazonaws.com",

"eventName": "StopLogging",

"awsRegion": "us-west-2",

"sourceIPAddress": “55.55.55.55",

"userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0",

"requestParameters": {

"name": "CloudTrail-Default"

},

"responseElements": null,

....

Amazon CloudWatch Logs

• Monitor, store, and access your log files from Amazon

EC2 instances, AWS CloudTrail, or other sources.

• Enable in the AWS Management Console, CLI, or via

AWS CloudFormation.

• Monitor and alarm for specific phrases, values, or

patterns.

http://docs.aws.amazon.com/AmazonCloudWatch/latest/

DeveloperGuide/WhatIsCloudWatchLogs.html

CloudFormation -> CloudWatch alarms

• Downloadable and editable example CloudFormation template from

AWS

• Contains predefined CloudWatch metric filters and alarms that

enable you to receive email notifications when certain security-

related API calls are made in your AWS account

• Amazon S3 bucket events, network events, Amazon EC2 events,

AWS CloudTrail, and AWS Identity and Access Management (IAM)

events

http://docs.aws.amazon.com/awscloudtrail/latest/userguide/use-

cloudformation-template-to-create-cloudwatch-alarms.html

CloudTrail OFF event – Detect

"CloudTrailStopMetricFilter": {

"Type": "AWS::Logs::MetricFilter",

"Properties": {

"LogGroupName": { "Ref" : "LogGroupName" },

"FilterPattern": ”{ ($.eventName = StopLogging) }",

"MetricTransformations": [

{

"MetricNamespace": "CloudTrailMetrics",

"MetricName": "CloudTrailEventCount",

"MetricValue": "1"

}

]

}

},

CloudTrail OFF event – Detect

"CloudTrailStoppedAlarm": {

"Type": "AWS::CloudWatch::Alarm",

"Properties": {

"AlarmName" : ”CloudTrailStoppedAlarm",

"AlarmDescription" : "Alarms when StopLogging API call is made",

"AlarmActions" : [{ "Ref" : "AlarmNotificationTopic" }],

"MetricName" : "CloudTrailEventCount",

"Namespace" : "CloudTrailMetrics",

"ComparisonOperator" : "GreaterThanOrEqualToThreshold",

"EvaluationPeriods" : "1",

"Period" : "300",

"Statistic" : "Sum",

"Threshold" : "1"

}

},

CloudTrail OFF event – Recover

CloudTrail OFF event – Investigate"userIdentity": {

"type": "IAMUser",

"principalId": "AIDAI5WIMUDR2UZUI62VO",

"arn": "arn:aws:iam::000123456789:user/reinvent-sec308",

"accountId": "000123456789",

"accessKeyId": "AKIAIRAHHRD3PHLUFJLQ",

"userName": "reinvent-sec308"

},

"eventTime": "2015-09-23T00:41:45Z",

"eventSource": "cloudtrail.amazonaws.com",

"eventName": "StopLogging",

"awsRegion": "us-west-2",

"sourceIPAddress": "55.55.55.55",

"userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0",

"requestParameters": {

"name": "CloudTrail-Default"

},

"responseElements": null,

....

CloudTrail OFF event – Protect

Deny permissions for CloudTrail in IAM groups or roles

{

"Sid": "Stmt0001",

"Effect": "Deny",

"Action": [

"cloudtrail:DeleteTrail",

"cloudtrail:StopLogging"

],

"Resource": [

"*"

]

}

Multi-Factor Authentication (MFA)

• Require unique authentication codes to access AWS

websites or services

• Hardware or virtual authentication device generates

codes

• Enter codes manually via AWS Management Console or

accompany API requests

• Configure via IAM

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_cr

edentials_mfa.html

MFA Deactivate Event

.....

"eventTime": "2015-09-20T18:53:02Z",

"eventSource": "iam.amazonaws.com",

"eventName": "DeactivateMFADevice",

"awsRegion": "us-east-1",

"sourceIPAddress": ”55.55.55.55",

"userAgent": "signin.amazonaws.com",

"requestParameters": {

"userName": ”bob",

"serialNumber": "arn:aws:iam::000019241430:mfa/bob"

},

"responseElements": null,

"requestID": "d1a9ebf8-5fc8-11e5-9d8f-1bc7c6757e61",

.....

MFA Deactivate Event – Detect

"MFADeactivateMetricFilter": {

"Type": "AWS::Logs::MetricFilter",

"Properties": {

"LogGroupName": { "Ref" : "LogGroupName" },

"FilterPattern": "{ ($.eventName=DeactivateMFADevice) }”,

"MetricTransformations": [

{

"MetricNamespace": "CloudTrailMetrics",

"MetricName": "MFADeactivateEventCount",

"MetricValue": "1"

}

]

}

},

MFA Deactivate Event – Recover

Reconfigure the MFA device

MFA Deactivate Event – Investigate

.....

"eventTime": "2015-09-20T18:53:02Z",

"eventSource": "iam.amazonaws.com",

"eventName": "DeactivateMFADevice",

"awsRegion": "us-east-1",

"sourceIPAddress": ”55.55.55.55",

"userAgent": "signin.amazonaws.com",

"requestParameters": {

"userName": ”bob",

"serialNumber": "arn:aws:iam::000019241430:mfa/bob"

},

"responseElements": null,

"requestID": "d1a9ebf8-5fc8-11e5-9d8f-1bc7c6757e61",

.....

MFA Deactivate Event – Protect

Use AWS Identity & Access Management to require MFA

http://blogs.aws.amazon.com/security/post/Tx2SJJYE082KBUK/How-to-Delegate-

Management-of-Multi-Factor-Authentication-to-AWS-IAM-Users

S3 object versioning

S3 object deletion event – Detect

• Bucket logging? Check.

• Bucket versioning? Check.

• Continuously reviewing logs …? NO

• We can enable push notifications for S3 events that

might concern us (for example, deletions)

• Configure S3 to detect events like ObjectRemoved

• S3 sends alert to the Amazon SNS topic of your choosing

• SNS topic sends message to subscribers, such as an email

to your security_team@yourcompany.com

S3 object deletion event – Recover

• Restore deleted file from previous version.

• Via AWS Management Console, just a couple clicks to

download/upload deleted version.

• Via CLI/API, just an S3 copy object request, specifying

version ID with copy source.

• If you enabled versioning AFTER initial object put,

version ID will be “NULL”. OK, you can still specify

“NULL” as a version to restore from.

Recover deleted S3 object – AWS CLI

aws s3api list-object-versions --bucket reinvent2015-sec308 --prefix prod

aws s3api copy-object --bucket reinvent2015-sec308 --copy-source reinvent2015-sec308/prod/important.txt?versionId=null --key prod/important.txt

Recover deleted S3 object (from backup) – AWS CLI

aws s3api copy-object --bucket reinvent2015-sec308 --copy-source reinvent2015-sec308/backup/important.txt?versionId=null --key prod/important.txt

S3 object deletion event – Investigate

S3 object deletion event – Protect

• Bucket versioning protects against inadvertent delete or

overwrite of objects.

• Consider more restrictive policies for credentials, such

as specifically disallow S3 object removal.

• Additional layer of protection; enable MFA Delete on a

versioned S3 bucket.

http://docs.aws.amazon.com/AmazonS3/latest/dev/Versi

oning.html#MultiFactorAuthenticationDelete

Log-in anomaly event – Detect

"ConsoleSignInAnomalyMetricFilter": {

"Type": "AWS::Logs::MetricFilter",

"Properties": {

"LogGroupName": { "Ref" : "LogGroupName" },

"FilterPattern": "{ ($.eventName = ConsoleLogin) && ($.sourceIPAddress != 55.55.*) }",

"MetricTransformations": [

{

"MetricNamespace": "CloudTrailMetrics",

"MetricName": "ConsoleSignInAnomalyCount",

"MetricValue": "1"

}

]

}

},

Log-in anomaly event – Recover

Add null IAM policy to the user (Deny all permissions):

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Deny",

"Action": [

"*"

],

"Resource": [

"*"

]

}

]

}

Log-in anomaly event – Investigate

Look in CloudTrail – Determine what events happened after the ConsoleLogin.

Log-in anomaly event – Protect

Add Condition statements to IAM

"Condition" : {

"IpAddress" : {

"aws:SourceIp" : [”55.55.0.0/16”]

}

}

Open security group

• 0.0.0.0/0 ingress has limited validity, but commonly used.

• Web server = Likely OK for the Internet to access 80/443.

• All of the web server’s OTHER ports? Likely NOT OK to

access the Internet.

• Policies can vary. No admin ports open to the world? OK.

• Creation and change velocity among security groups

should be LOW.

AWS Config

• AWS resource inventory, configuration history, and

configuration change notifications

• Discover existing AWS resources

• Export inventory of your AWS resources with all configuration

details

• Determine how a resource was configured at any point in

time

• Security geeks should LOVE it!

http://aws.amazon.com/documentation/config/

Open security group event – Detect

• Subscribe to AWS Config notification topic.

• Filter notifications for creation of security groups that

might be concerning. You could look for the following,

individually or combined:

• “SecurityGroup” and “Created” within subject

• changeType : “CREATE” within body

• resourceType: "AWS::EC2::SecurityGroup” within body

Open security group event – Detect

"groupId": "sg-7dc0d21a",

...

"ipPermissions": [

{

"ipProtocol": "-1",

"fromPort": null,

"toPort": null,

"userIdGroupPairs": [],

"ipRanges": [

"0.0.0.0/0"

],

"prefixListIds": []

}

],

...

Open security group event – Recover

• If responding soon enough to the creation of a new

security group and no instances, simply delete the

security group.

• Otherwise, assign running instances to another security

group, and then delete the offending security group.

• You can’t delete a default security group, but you can

change its rules back to something sane, including no

rules.

Delete open security group – AWS CLI

aws ec2 delete-security-group --no-dry-run --group-id sg-d3bda2b4

Open security group event – Investigate

• Revisit the AWS Config change notification.

• Note time, action, and security group ID to correlate to

principal and source IP of EC2 API call via AWS

CloudTrail.

• If possible, engage principal to understand intent or

determine if unexplained, such as by external actor and

potentially malicious.

Open security group event – Protect

• Appropriately constrain or deactivate associated

credentials as warranted.

• Security group changes, particularly within production,

should not be a frequent event, so maintain high

vigilance.

Unapproved AMIs

Amazon Machine Images

• Public AMI

• Marketplace AMI

• Private AMI

• Approved AMIs/“Golden” AMIs

Unapproved AMI event – Detect

• Compare launched EC2 instances against a whitelist.

• What is a good method to compare against a whitelist?

Let’s use AWS Lambda!

• Runs your code in response to events.

• Automatically manages compute resources for you.

• Create new back-end services where compute

resources are automatically triggered based on custom

requests.

• You can read CloudTrail events with AWS Lambda.

http://docs.aws.amazon.com/lambda/latest/dg/welcome.html

Unapproved AMI event - Recover

matchingRecords,

function(record, complete) {

var params = {

InstanceIds: []

};

// List each instance ID

for (var i = 0; i < record.responseElements.instancesSet.items.length; i++){

params.InstanceIds.push(record.responseElements.instancesSet.items[i].instanceId);

}

// Terminate the enumerated instances

ec2.terminateInstances(params, complete);

Unapproved AMI event – Investigate

Interrogate CloudTrail logs as before

• Who launched it?

• Where did the request come from?

• Which subnet was it being launched into?

Unapproved AMI event – Protect

Restrict access in IAM to specific AMIs IDs

Automate IR?

• Most, if not all, of the pieces to automate IR exist in AWS

• Automated IR = Even greater security agility

• Detect -> Protect programmatically

• Lambda-fy your IR!

Detecting events in Lambda

…

var EVENT_SOURCE_TO_TRACK = /cloudtrail.amazonaws.com/;

var EVENT_NAME_TO_TRACK = /StopLogging/;

var matchingRecords = records

.Records

.filter(function(record) {

return record.eventSource.match(EVENT_SOURCE_TO_TRACK)

&& record.eventName.match(EVENT_NAME_TO_TRACK);

});

…

Source: http://docs.aws.amazon.com/lambda/latest/dg/wt-cloudtrail-events-

adminuser.html

Responding to events in Lambda

…

if (matchingRecords.length >= 1) {

console.log(’StopLogging detected! Reverting...');

cloudtrail.startLogging(cloudtrailParams, function(err, data) {

….

Responding to events in Lambda

Building a “Lambda Responder”

CloudTrail S3

Lambda

Lambda

SNS

Building a “Lambda Responder”

1. Turn on AWS CloudTrail – Choose an S3 bucket.

2. Create an SNS topic.

3. Update the topic policy to allow event notifications from your

S3 bucket.

4. Configure your S3 bucket to send event notifications to the

SNS topic.

5. Create an IAM role for the Lambda functions.

6. Create the Lambda functions and process SNS messages.

https://aws.amazon.com/blogs/compute/fanout-s3-event-

notifications-to-multiple-endpoints/ by John Stamper

Building a “Lambda Responder”

• What could you automatically respond to?

Lambda – Automated S3 object recovery

...

var bucket = event.Records[0].s3.bucket.name;

var key = event.Records[0].s3.object.key;

var backup = ’your-backup-bucket/' + key;

var params = {

Bucket: bucket,

CopySource: backup,

Key: key,

};

s3.copyObject(params, function(err, data) {

// removed for brevity

});

...

Lambda – Automated open security group delete

var snsMsgString = JSON.stringify(event.Records[0].Sns.Message);

var snsMsgObject = getSNSMessageObject(snsMsgString);

if (snsMsgObject.configurationItemDiff.changeType == 'CREATE' && snsMsgObject.configurationItem.resourceType == 'AWS::EC2::SecurityGroup' && snsMsgObject.configurationItem.configuration.ipPermissions[0].ipProtocol == '-1' && snsMsgObject.configurationItem.configuration.ipPermissions[0].ipRanges == '0.0.0.0/0'){

var params = {

DryRun: false,

GroupId: snsMsgObject.configurationItem.resourceId,

};

ec2.deleteSecurityGroup(params, function(err, data) {

context.succeed(snsMsgObject);

});

}

AWS Config -> Lambda … IR aaS? AWS Config

Rules!

• Extends AWS Config with a powerful new rule system

• Use existing rules from AWS and from partners

• You can also define your own custom rules

• SEC314 - NEW LAUNCH! AWS Config/Config Rules:

Use AWS Config Rules to Improve Governance over

Configuration Changes to Your Resources

Practice makes perfect

• IR game day…YAY!

• Tabletop first…yay?

• See SEC316 – Harden Your Architecture with Security

Incident Response Simulations (SIRS), Jon Miller and

Armando Leite

AWS Partner, Dell SecureWorks, IR Support

• Customer IR case example

• Our IR preparedness “Wish List” for AWS customers

• How to contact us

IR Case Example – Background, Event

• Dell SecureWorks contacted by an AWS customer, a provider of cloud-

based collaboration software

• Customer investigated abnormally high CPU usage on Internet-facing

servers hosting their customers’ applications

• Customer’s review of system logs identified unauthorized logins from a wide

array of IP addresses using compromised credentials

• Threat actors leveraged the Customer’s compromised web app credentials

to gain unauthorized entry and propagate to a multitude of connected

resources within the Customer’s AWS environment

• Dell SecureWorks performed digital forensics on the Customer’s web

applications, AWS instances and snapshots, AWS CloudTrail logs, and

suspected on-premise systems

IR Case Example - Response

• Dell SecureWorks prepared forensic analysis environment:• Launched forensic EC2 instances within Dell SecureWorks’ VPC

• Created S3 bucket for event data storage and transfer of forensic artifacts

• Using IAM, Customer provided appropriate access for Dell

SecureWorks to:• Acquire snapshots of the affected Customer’s EC2 instances

• Transfer snapshots to Dell SecureWorks’ S3 bucket for forensic analysis

• Receive access to Customer’s CloudTrail logs for forensic analysis

• Using rapidly-deployed forensic toolsets, Dell SecureWorks

conducted forensic exam of:• File systems of the Customer’s Internet-facing EC2 instances

• Customer’s AMIs

• Customer’s AWS CloudTrail logs

• Dell SecureWorks provided comprehensive analysis of the incident

and affected AWS resources

IR Case Example - Takeaways

• AWS enables shorter response times for security events vs. on-premise• Time between engagement kickoff and commencing analysis was drastically reduced

• Security event data can be rapidly acquired, staged, and analyzed all within AWS

• Appropriate access can be quickly granted to security event responders via AWS IAM

• The ability to collaborate on configuration activities directly within AWS minimized time

taken for troubleshooting

• Creating effective environments for sharing incident response resources and

data within AWS is straight-forward

• Versus traditional IR, cost savings are also realized via IR within AWS

through reduction of the investigation timeline (minimized time to data

acquisition, resource setup, and initial analysis)

Our IR Prep “Wish List” for AWS Customers

• Take snapshots of all affected or suspected instances

• Collect network and instance metadata

• Create a restricted-access VPC, Security Group, and/or

separate AWS account

• Be ready to create temporary users / credentials via IAM

• Enable and centralize CloudTrail and CloudWatch logs

• Create a dedicated S3 bucket for sharing incident

response artifacts

How to Contact Dell SecureWorks

• Incident Response Hotline (24x7x365)

1-877-884-1110

• Website

http://www.secureworks.com/incident-response/

• Booth: #446 (next to Docker)

Flag me down and/or visit our booth to learn more about Dell

SecureWorks’ experience and capabilities and how we are partnered

with AWS to provide Incident Response for AWS customers!

AWS Security Best Practices whitepaper

• Help for designing security infrastructure and

configuration of your AWS environment

• High-level guidance for:

• Managing accounts, users, groups, and roles

• Managing OS-level access to instances

• Securing your data, OS, apps, and infrastructure

• Managing security monitoring, auditing, alerting, and incident

response

https://media.amazonwebservices.com/AWS_Security_Best_Practices.pdf

External resources – Reading, training

• SANS Reading Room, Incident Response

http://www.sans.org/reading-room/whitepapers/incident

• FIRST

http://www.first.org/resources/guides

• CERT, Incident Management

http://www.cert.org/incident-management/publications/

External resources – IR tools, frameworks

• Mozilla Investigator (MIG)

http://mig.mozilla.org/

• Netflix Fully Integrated Defense Operations (FIDO)

http://techblog.netflix.com/2015/05/introducing-fido-

automated-security.html

Other relevant talks this week

• SEC403 - Timely Security Alerts and Analytics: Diving

into AWS CloudTrail Events by Using Apache Spark on

Amazon EMR, Will Kruse

• SEC303 – Architecting for End-to-End Security in the

Enterprise, Hart Rossman and Bill Shinn

• If you miss(ed) any of them live, they will be on

YouTube, just like this talk.

• Don’t forget last year’s “Intrusion Detection in the Cloud”

and “Incident Response in the Cloud” that are already on

YouTube!

AWS Support for security concerns

• AWS Support is the one-stop shop for AWS customers,

for any concerns, including security related.

• If AWS Support cannot immediately address your

concerns, they will escalate internally to the appropriate

technical team, AWS Security included.

https://aws.amazon.com/support

AWS security resources

• AWS Security Blog

http://blogs.aws.amazon.com/security/

• AWS Security Center

https://aws.amazon.com/security

• Contact the AWS security team

aws-security@amazon.com

Summary

• Security agility with AWS

• Threat vs. policy-driven concerns, enumerate, create

signatures, detection mechanisms

• Automate IR where you can

• Two ways to get more practice: you only get to choose

one

• We (AWS and our technology partners) are here to help!

Remember to complete

your evaluations!

Thank you!