It's 10pm, Do You Know Where Your Access Keys

Are?Ken Johnson

Things to Mention

• Ask questions throughout presentation

• There will be no dedicated Q&A – so stick around after and find me if you want to chat

• This presentation will move fast. Slides will be available so don’t worry about minutia.

Background/About

• Ken Johnson, CTO and Partner at nVisium• Veteran, US Navy• I speak about:

– DevOps (In)Security– Exploiting Web Applications– Coding and Coding + Security– Node, Elixir, Python, Ruby, Go– AWS (clearly)

Background/About

• I’m the CTO of a security company• Naturally, I have some concerns as it

pertains to AWS• My Concerns

– Risk Assessments (Compliance)– Data security– Reputation

Background/About

1. AWS used to be just a “thing” we had

2. Then it became a little more important

3. Then it became business-critical4. Then I got worried…

Problem Statement

How can we prevent attacks?

How can we know if an attack is happening?

How can we recover if the worst case scenario somehow happens?

My Plan

• Harden – Make it difficult to reach our AWS environment

• Monitor – If our AWS environment is breached, we need to know and alert ourselves

• Restore – Have the ability to reconstruct data/configs after a “hack”

AWS’s Plan

• Took the AWS Security Fundamentals Course and…– Fortunately, our strategy lines up with AWS

recommendations– You are responsible for leveraging the tools

AWS provides (financially)– Your configuration… that is on you– https://aws.amazon.com

/training/course-descriptions/security-fundamentals/

AWS Hardening Basics

Making it difficult (for attackers) to reach our environment

Hardening Checklist

1. Don’t Use The Root Account!2. Disable Access Keys for Root

Account3. Multi-Factor Authentication4. API + MFA5. Strong Password Policy

Don’t Use Root Account

• Every AWS env has a root account, only necessary to use for very specific circumstances

• When these circumstances arise, notify your team that the account will be used

• We will discuss why this is important when we talk about CloudWatch metrics

Disable/Delete Root Account Access Keys

• Just delete them if they exist– Disable the access keys in the event you

are unable to delete them completely for some reason

• Make sure your admins have a (verbal/written) policy that states “we don’t create access keys for the root account”

MFA

• If credentials are stolen or guessed, we want a second layer of protection

• You can use apps or hardware to do this– Google Authenticator (Apps)– Gemalto (Hardware)

• Find the full list of MFA devices here:https://aws.amazon.com/iam/details/mfa/• This is so ridiculously easy to do, everyone

should

MFALet’s demonstrate enabling MFA using a virtual device (app) on an IAM account

MFA

Navigate to Identity & Access Management

MFA

Next, manage the MFA device

MFA

Choose a virtual device

MFA

Lastly, use Google Authenticator to take a snapshot of the QR code

MFA

• At this point, its worth mentioning that non-administrators or those without IAM privileges cannot enable MFA on their own account

• Why is this a problem? Well, they need to be able to enable MFA on their own device… not the administrator’s

• Fortunately, we have a solution!

MFA

MFA

• Okay so that wasn’t the easiest to read, so here is the link: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_delegate-permissions_examples.html#creds-policies-mfa-console

• Basically this IAM policy allows a user to manage their *OWN* MFA device

MFA (for Root Account)

• Need a shared MFA for root? TOTP!

• Recommend using something like 1password for teams, can share the TOTP code: https://support.1password.com/guides/mac/totp.htmlhttps://www.youtube.com/watch?v=eZyb-ArMK9g

API + MFA

• You have the ability to place a restriction where resources can only be interacted with if the user has authenticated with MFA

• This helps prevent (ab)use should someone steal access keys or credentials

API + MFA

1. At a minimum, apply to administrator & power user group policies… really any group that can do anything of importance

API + MFA

This entry requires MFA for Web/API

API + MFA

• Truth be told, doing this can be painful at first

• Things that used to work, might not (via the API)

• Fortunately, we have some answers for you

• Firstly, let’s discuss STS or SecurityToken Service

API + MFA

• Leverage STS in order to interact with the AWS API should this MFA restriction be placed on resources (and it should )

• Example of using STS:

https://gist.github.com/cktricky/127be4e431563a986f0f

API + MFA

Use this script to retrieve creds (from gist)

API + MFA

Output of script

API + MFA

Use the creds to leverage tools like ec2-api-tools(-O <access key id>–W <secret> and –T <session token>)

API + MFA

And in case you don’t like Ruby…

https://github.com/jimbrowne/aws-sts-helpers

API + MFA

• ElasticBeanstalk does not work with STS. Le Terrible.

• However, there is a workaround, use CodePipeline.

• Very simple process to setup but only works with:– GitHub– AWS CodeCommit– Amazon S3

API + MFA

• One final note of warning here, you may see oddities/restrictions when you go to use resources in the AWS web interface AFTER having been logged in for a bit… just reauthenticate

Password Policy

• Password policies are important because historically people do not choose complex passwords

• MFA should help, but we’re talking about a layered approach

• Again, making our AWS environment harder to reach

Example Password Policy

Hardening Recap

• Make credentials hard to guess• If guessed or stolen, we still have

MFA• Remember MFA only protects against

the web and NOT the API… unless you change your policies and use STS

• Root account is King, protect your King

Hardening Recap

• Things we did not (and won’t discuss)– S3 bucket policies– Security Group configurations– SSH Key Management– Encrypting Data (Volumes, S3 buckets)

• Trusted Advisor – Use it, because it catches a lot of “low hanging fruit” style issues

Hardening Recap

• Links to resources that discuss the items we’re not covering:– https://d0.awsstatic.com/whitepapers/compliance/

AWS_Auditing_Security_Checklist.pdf– http://aws-de-media.s3.amazonaws.com/images/Produkt

blaetter/AWS-Security-Check-List_eng.pdf

– http://www.slideshare.net/AmazonWebServices/masterclass-advanced-security-best-practices

• Frankly you can’t throw a rock without hitting some basic info regarding AWS Security Checklists

AWS Monitoring

Detecting malicious activity

AWS Monitoring

• Assuming hardening (prevention) has failed, how would we know?

• Luckily, AWS provides several services which alert to anomalies

• We will walk through examples of using these services, but ultimately decide what is right for you

• Fair warning, some of these services will provide a lot of noise

AWS Monitoring

4 important services:1. CloudTrail2. SNS3. Config4. CloudWatch

AWS Monitoring

• CloudTrail – Logs• SNS – Notifications• Config – Alerts for modifications &

noncompliance• CloudWatch – Alerts for specific types

of behavior

AWS Monitoring

CloudTrail

Config

CloudWatch

SNS

AWS CloudTrail

AWS Monitoring (CloudTrail)

• CloudTrail is primarily used for log collection

• Other services like CloudWatch, for example, use those logs to filter relevant data

AWS Monitoring (CloudTrail)

Pretty easy, first turn it on..

AWS Monitoring (CloudTrail)

Configure the log group

AWS Monitoring (CloudTrail)

Allow the creation of an IAM role by CloudTrail

AWS Monitoring (CloudTrail)

• At this point you are okay• Start configuring CloudWatch/Config

AWS SNS

AWS Monitoring (SNS)

• Fantastic offering, <3 it– Examples of ways to be notified by SNS

• SMS• Email• JSON Post to your Application’s API endpoint

AWS Monitoring (SNS)

• Receive SMS/Email/Slack notifications for important events

• ^ This is so you get immediate notifications

• You can have multiple subscribers, I’d suggest you use that functionality

• Basic gist? Receive immediate updates for things you want to see… immediately

AWS Monitoring (SNS)

Create a topic

AWS Monitoring (SNS)

Create Subscription

AWS Monitoring (SNS)

Create SMS (or whatever, but in this case, SMS)

AWS Monitoring (SNS)

Example of creating email subscription… bottomline you can have multiple ways of notifying people

AWS Config

AWS Monitoring (Config)

• Config:– Alerts owners to changes or

noncompliance with regards to AWS resources

– Can either design custom Config rules or use managed (pre-packaged) AWS Config rules

AWS Monitoring (Config)• Pre-packaged “Managed” AWS Rules

– CLOUD_TRAIL_ENABLED– EIP_ATTACHED– ENCRYPTED_VOLUMES– INCOMING_SSH_DISABLED– INSTANCES_IN_VPC– REQUIRED_TAGS– RESTRICTED_INCOMING_TRAFFIC

https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_use-managed-rules.html

AWS Monitoring (Config)

• Examples of things you can have alerts set for:– Change in Firewall (Security Group)

ports– Changes in VPC– Any change… at all

AWS Monitoring (Config)

Go to the Config service and choose resources to track

AWS Monitoring (Config)

Or choose to track everything

AWS Monitoring (Config)

Create a bucket, create an SNS topic (…we’ll discuss next)

AWS Monitoring (Config)

Allow the role to be created and you’re all set!

AWS CloudWatch

AWS Monitoring (CloudWatch)

• We can be very particular here about what it is we want to see

• Some very interesting things you can monitor

• Some examples:– Billing Alerts (Important for detection of

abuse or mistakes)– Track Root Account Usage– Failed login attempts

AWS Monitoring (CloudWatch - Billing)

• Used to prevent abuse or mistakes from costing your organization money

• Analyze and approximate your monthly spend• Configure via CloudWatch• Use SNS for instantaneous alerting

AWS Monitoring (CloudWatch - Billing)

Navigate to billing & cost management; enable billing alerts

AWS Monitoring (CloudWatch - Billing)

Create an SNS topic

AWS Monitoring (CloudWatch - Billing)

Subscribe to Topic

AWS Monitoring (CloudWatch - Billing)

Navigate to CloudWatch -> Metrics -> Billing

AWS Monitoring (CloudWatch - Billing)

Choose USD/EstimateCharges -> Create Alarm

AWS Monitoring (CloudWatch - Billing)

Set price point, SNS topic, and create alarm

AWS Monitoring (CloudWatch - Billing)

Exact steps to enable can be found here:http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier-alarms.html

AWS Monitoring (CloudWatch – Root Login)

• Remember how I said don’t use the Root account routinely?

• BUT… if this account is used, you should know about it

• This is the reason you’ll want to notify others (who receive SNS alerts) of the fact you are about to use the account

AWS Monitoring (CloudWatch – Root Login)

Choose log group, create metric

AWS Monitoring (CloudWatch – Root Login)

Define Logs Metric Filter

AWS Monitoring (CloudWatch – Root Login)

Assign/Create Filter

AWS Monitoring (CloudWatch – Root Login)

Click “Create Alarm”

AWS Monitoring (CloudWatch – Root Login)

Define Alarm and you’re good…

AWS Monitoring (CloudWatch – Root Login)

Exact steps (with pics) exist here:

https://blogs.aws.amazon.com/security/post/Tx3PSPQSN8374D/How-to-Receive-Notifications-When-Your-AWS-Account-s-Root-Access-Keys-Are-Used

AWS Monitoring (CloudWatch – Failed Logins)

• In the event someone is trying to break in, let’s alert ourselves to this!

• Failed logins typically suggest either someone forgot their password or… someone is trying to guess yours

AWS Monitoring (CloudWatch – Failed Logins)

Navigate to Logs, click “Create Metric Filter”

AWS Monitoring (CloudWatch – Failed Logins)

Enter the relevant filter pattern, click create

AWS Monitoring (CloudWatch – Failed Logins)

Fill out filter/metric/metric-namespace info

AWS Monitoring (CloudWatch – Failed Logins)

Click “Create Alarm”

AWS Monitoring (CloudWatch – Failed Logins)

Fill in relevant details and click “Create Alarm”

AWS Monitoring (CloudWatch – Failed Logins)

• Exact steps exist here:

http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudwatch-alarms-for-cloudtrail.html#cloudwatch-alarms-for-cloudtrail-signin

AWS Monitoring (CloudWatch) – Filter Patterns

• Create your own custom filter patterns, here is a resource for that:

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html

AWS + Splunk

AWS + Splunk

• Splunk is a pretty great resource for monitoring activity

• I’m fairly new to Splunk myself• Two separate plugins:

Splunk App for AWShttps://splunkbase.splunk.com/app/1274/Splunk Add-Onhttps://splunkbase.splunk.com/app/1876/

AWS + Splunk

• Examples of things you can view:– Billing– Topology– Usage– IAM Activity– SSH Key Pair Activity– User Activity– Network ACL(s)– VPC Activity

and a lot more…

AWS + Splunk

• Pretty Screenshot 1

AWS + Splunk

• Pretty Screenshot 2

AWS + Splunk

• Pretty Screenshot 3

AWS + Splunk

• Add plugins (apps) to Splunk

AWS + Splunk

• Splunk will need an AWS account in order to retrieve data

• Create account(s) for Splunk, grab the necessary permission policy from here:

http://docs.splunk.com/Documentation/AddOns/released/AWS/ConfigureAWSpermissions

AWS + Splunk

• Add the newly created account(s) to Splunk Add-on for AWS app - requires AWS access token id/secret

AWS + Splunk

• Configure AWS App for Splunk, add account(s), configure each input accordingly:

AWS + Splunk

• To view things like IAM Activity…– Subscribe to a cloudtrail log via SNS– Utilize SQS and subscribe SQS to an SNS

Topic

AWS Monitoring Recap

• Alert yourself when things change• This will get noisy, find a way to filter that which

is important– If it’s a high risk event, send an

SMS/Slack/Email blast• At a minimum, alert yourself when odd things

occur… like:– Billing increases past your normal spend– When somebody authenticates as Root– When someone has a login failure

AWS Monitoring Recap

• Interesting Quora thread: – https://www.quora.com/My-AWS-account-was-hacked-an

d-I-have-a-50-000-bill-how-can-I-reduce-the-amount-I-need-to-pay

• Highlights from the article:– AWS has “a review board of sorts” to determine if you

should be refunded– Bots are scouring GitHub searching for exposed access

keys– One of the more AWS-seasoned responders mentioned

doing part of what we discussed here today to avoid it– A decent number of the people posting on this thread

said “Yes, happened to me too”

AWS Restoration & Recovery

Plan to fail, just don’t fail to plan

AWS Restoration & Recovery – Basic Incident Response (IR)

• Understand who to contact if things go bad

• Understand how to communicate (ex: “speak only over the phone”)

• Understand what information to parse

• Understand where your backups are located and how they are secured

AWS Restoration & Recovery – Basic IR

• Do not USE AWS TO BACKUP YOUR AWS

• Offsite backups (meaning, off AWS site)

• Common things to back-up:– Databases/ Snapshots– S3 Buckets– EBS Volumes– CloudFormation Templates

AWS Restoration & Recovery – Basic IR

• Resources:– http://stackoverflow.com/questions/1708

7542/backup-solutions-for-aws-ec2-instances

– https://github.com/Scalr/installer-ng– http://www.n2ws.com/blog/3-ways-ec2-w

indows-backup-and-recovery.html

Presentation Recap

Summary

Recap

• Makes your environment harder to reach… for the bad guys– Limit what stolen or “otherwise

obtained” access keys or credentials could be used to do

– Prevent them being stolen in the first place

• Alert yourself to anomalies• Have a plan for if things go bad• Stay safe out there!

Contact Info

• My Info• Twitter: @cktricky• Email: ken@nvisium.com