It's 10pm, Do You Know Where Your Access Keys Are?
-
Upload
ken-johnson -
Category
Engineering
-
view
171 -
download
0
Transcript of It's 10pm, Do You Know Where Your Access Keys Are?
It's 10pm, Do You Know Where Your Access Keys
Are?Ken Johnson
Things to Mention
• Ask questions throughout presentation
• There will be no dedicated Q&A – so stick around after and find me if you want to chat
• This presentation will move fast. Slides will be available so don’t worry about minutia.
Background/About
• Ken Johnson, CTO and Partner at nVisium• Veteran, US Navy• I speak about:
– DevOps (In)Security– Exploiting Web Applications– Coding and Coding + Security– Node, Elixir, Python, Ruby, Go– AWS (clearly)
Background/About
• I’m the CTO of a security company• Naturally, I have some concerns as it
pertains to AWS• My Concerns
– Risk Assessments (Compliance)– Data security– Reputation
Background/About
1. AWS used to be just a “thing” we had
2. Then it became a little more important
3. Then it became business-critical4. Then I got worried…
Problem Statement
How can we prevent attacks?
How can we know if an attack is happening?
How can we recover if the worst case scenario somehow happens?
My Plan
• Harden – Make it difficult to reach our AWS environment
• Monitor – If our AWS environment is breached, we need to know and alert ourselves
• Restore – Have the ability to reconstruct data/configs after a “hack”
AWS’s Plan
• Took the AWS Security Fundamentals Course and…– Fortunately, our strategy lines up with AWS
recommendations– You are responsible for leveraging the tools
AWS provides (financially)– Your configuration… that is on you– https://aws.amazon.com
/training/course-descriptions/security-fundamentals/
AWS Hardening Basics
Making it difficult (for attackers) to reach our environment
Hardening Checklist
1. Don’t Use The Root Account!2. Disable Access Keys for Root
Account3. Multi-Factor Authentication4. API + MFA5. Strong Password Policy
Don’t Use Root Account
• Every AWS env has a root account, only necessary to use for very specific circumstances
• When these circumstances arise, notify your team that the account will be used
• We will discuss why this is important when we talk about CloudWatch metrics
Disable/Delete Root Account Access Keys
• Just delete them if they exist– Disable the access keys in the event you
are unable to delete them completely for some reason
• Make sure your admins have a (verbal/written) policy that states “we don’t create access keys for the root account”
MFA
• If credentials are stolen or guessed, we want a second layer of protection
• You can use apps or hardware to do this– Google Authenticator (Apps)– Gemalto (Hardware)
• Find the full list of MFA devices here:https://aws.amazon.com/iam/details/mfa/• This is so ridiculously easy to do, everyone
should
MFALet’s demonstrate enabling MFA using a virtual device (app) on an IAM account
MFA
Navigate to Identity & Access Management
MFA
Next, manage the MFA device
MFA
Choose a virtual device
MFA
Lastly, use Google Authenticator to take a snapshot of the QR code
MFA
• At this point, its worth mentioning that non-administrators or those without IAM privileges cannot enable MFA on their own account
• Why is this a problem? Well, they need to be able to enable MFA on their own device… not the administrator’s
• Fortunately, we have a solution!
MFA
MFA
• Okay so that wasn’t the easiest to read, so here is the link: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_delegate-permissions_examples.html#creds-policies-mfa-console
• Basically this IAM policy allows a user to manage their *OWN* MFA device
MFA (for Root Account)
• Need a shared MFA for root? TOTP!
• Recommend using something like 1password for teams, can share the TOTP code: https://support.1password.com/guides/mac/totp.htmlhttps://www.youtube.com/watch?v=eZyb-ArMK9g
API + MFA
• You have the ability to place a restriction where resources can only be interacted with if the user has authenticated with MFA
• This helps prevent (ab)use should someone steal access keys or credentials
API + MFA
1. At a minimum, apply to administrator & power user group policies… really any group that can do anything of importance
API + MFA
This entry requires MFA for Web/API
API + MFA
• Truth be told, doing this can be painful at first
• Things that used to work, might not (via the API)
• Fortunately, we have some answers for you
• Firstly, let’s discuss STS or SecurityToken Service
API + MFA
• Leverage STS in order to interact with the AWS API should this MFA restriction be placed on resources (and it should )
• Example of using STS:
https://gist.github.com/cktricky/127be4e431563a986f0f
API + MFA
Use this script to retrieve creds (from gist)
API + MFA
Output of script
API + MFA
Use the creds to leverage tools like ec2-api-tools(-O <access key id>–W <secret> and –T <session token>)
API + MFA
And in case you don’t like Ruby…
https://github.com/jimbrowne/aws-sts-helpers
API + MFA
• ElasticBeanstalk does not work with STS. Le Terrible.
• However, there is a workaround, use CodePipeline.
• Very simple process to setup but only works with:– GitHub– AWS CodeCommit– Amazon S3
API + MFA
• One final note of warning here, you may see oddities/restrictions when you go to use resources in the AWS web interface AFTER having been logged in for a bit… just reauthenticate
Password Policy
• Password policies are important because historically people do not choose complex passwords
• MFA should help, but we’re talking about a layered approach
• Again, making our AWS environment harder to reach
Example Password Policy
Hardening Recap
• Make credentials hard to guess• If guessed or stolen, we still have
MFA• Remember MFA only protects against
the web and NOT the API… unless you change your policies and use STS
• Root account is King, protect your King
Hardening Recap
• Things we did not (and won’t discuss)– S3 bucket policies– Security Group configurations– SSH Key Management– Encrypting Data (Volumes, S3 buckets)
• Trusted Advisor – Use it, because it catches a lot of “low hanging fruit” style issues
Hardening Recap
• Links to resources that discuss the items we’re not covering:– https://d0.awsstatic.com/whitepapers/compliance/
AWS_Auditing_Security_Checklist.pdf– http://aws-de-media.s3.amazonaws.com/images/Produkt
blaetter/AWS-Security-Check-List_eng.pdf
– http://www.slideshare.net/AmazonWebServices/masterclass-advanced-security-best-practices
• Frankly you can’t throw a rock without hitting some basic info regarding AWS Security Checklists
AWS Monitoring
Detecting malicious activity
AWS Monitoring
• Assuming hardening (prevention) has failed, how would we know?
• Luckily, AWS provides several services which alert to anomalies
• We will walk through examples of using these services, but ultimately decide what is right for you
• Fair warning, some of these services will provide a lot of noise
AWS Monitoring
4 important services:1. CloudTrail2. SNS3. Config4. CloudWatch
AWS Monitoring
• CloudTrail – Logs• SNS – Notifications• Config – Alerts for modifications &
noncompliance• CloudWatch – Alerts for specific types
of behavior
AWS Monitoring
CloudTrail
Config
CloudWatch
SNS
AWS CloudTrail
AWS Monitoring (CloudTrail)
• CloudTrail is primarily used for log collection
• Other services like CloudWatch, for example, use those logs to filter relevant data
AWS Monitoring (CloudTrail)
Pretty easy, first turn it on..
AWS Monitoring (CloudTrail)
Configure the log group
AWS Monitoring (CloudTrail)
Allow the creation of an IAM role by CloudTrail
AWS Monitoring (CloudTrail)
• At this point you are okay• Start configuring CloudWatch/Config
AWS SNS
AWS Monitoring (SNS)
• Fantastic offering, <3 it– Examples of ways to be notified by SNS
• SMS• Email• JSON Post to your Application’s API endpoint
AWS Monitoring (SNS)
• Receive SMS/Email/Slack notifications for important events
• ^ This is so you get immediate notifications
• You can have multiple subscribers, I’d suggest you use that functionality
• Basic gist? Receive immediate updates for things you want to see… immediately
AWS Monitoring (SNS)
Create a topic
AWS Monitoring (SNS)
Create Subscription
AWS Monitoring (SNS)
Create SMS (or whatever, but in this case, SMS)
AWS Monitoring (SNS)
Example of creating email subscription… bottomline you can have multiple ways of notifying people
AWS Config
AWS Monitoring (Config)
• Config:– Alerts owners to changes or
noncompliance with regards to AWS resources
– Can either design custom Config rules or use managed (pre-packaged) AWS Config rules
AWS Monitoring (Config)• Pre-packaged “Managed” AWS Rules
– CLOUD_TRAIL_ENABLED– EIP_ATTACHED– ENCRYPTED_VOLUMES– INCOMING_SSH_DISABLED– INSTANCES_IN_VPC– REQUIRED_TAGS– RESTRICTED_INCOMING_TRAFFIC
https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_use-managed-rules.html
AWS Monitoring (Config)
• Examples of things you can have alerts set for:– Change in Firewall (Security Group)
ports– Changes in VPC– Any change… at all
AWS Monitoring (Config)
Go to the Config service and choose resources to track
AWS Monitoring (Config)
Or choose to track everything
AWS Monitoring (Config)
Create a bucket, create an SNS topic (…we’ll discuss next)
AWS Monitoring (Config)
Allow the role to be created and you’re all set!
AWS CloudWatch
AWS Monitoring (CloudWatch)
• We can be very particular here about what it is we want to see
• Some very interesting things you can monitor
• Some examples:– Billing Alerts (Important for detection of
abuse or mistakes)– Track Root Account Usage– Failed login attempts
AWS Monitoring (CloudWatch - Billing)
• Used to prevent abuse or mistakes from costing your organization money
• Analyze and approximate your monthly spend• Configure via CloudWatch• Use SNS for instantaneous alerting
AWS Monitoring (CloudWatch - Billing)
Navigate to billing & cost management; enable billing alerts
AWS Monitoring (CloudWatch - Billing)
Create an SNS topic
AWS Monitoring (CloudWatch - Billing)
Subscribe to Topic
AWS Monitoring (CloudWatch - Billing)
Navigate to CloudWatch -> Metrics -> Billing
AWS Monitoring (CloudWatch - Billing)
Choose USD/EstimateCharges -> Create Alarm
AWS Monitoring (CloudWatch - Billing)
Set price point, SNS topic, and create alarm
AWS Monitoring (CloudWatch - Billing)
Exact steps to enable can be found here:http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier-alarms.html
AWS Monitoring (CloudWatch – Root Login)
• Remember how I said don’t use the Root account routinely?
• BUT… if this account is used, you should know about it
• This is the reason you’ll want to notify others (who receive SNS alerts) of the fact you are about to use the account
AWS Monitoring (CloudWatch – Root Login)
Choose log group, create metric
AWS Monitoring (CloudWatch – Root Login)
Define Logs Metric Filter
AWS Monitoring (CloudWatch – Root Login)
Assign/Create Filter
AWS Monitoring (CloudWatch – Root Login)
Click “Create Alarm”
AWS Monitoring (CloudWatch – Root Login)
Define Alarm and you’re good…
AWS Monitoring (CloudWatch – Root Login)
Exact steps (with pics) exist here:
https://blogs.aws.amazon.com/security/post/Tx3PSPQSN8374D/How-to-Receive-Notifications-When-Your-AWS-Account-s-Root-Access-Keys-Are-Used
AWS Monitoring (CloudWatch – Failed Logins)
• In the event someone is trying to break in, let’s alert ourselves to this!
• Failed logins typically suggest either someone forgot their password or… someone is trying to guess yours
AWS Monitoring (CloudWatch – Failed Logins)
Navigate to Logs, click “Create Metric Filter”
AWS Monitoring (CloudWatch – Failed Logins)
Enter the relevant filter pattern, click create
AWS Monitoring (CloudWatch – Failed Logins)
Fill out filter/metric/metric-namespace info
AWS Monitoring (CloudWatch – Failed Logins)
Click “Create Alarm”
AWS Monitoring (CloudWatch – Failed Logins)
Fill in relevant details and click “Create Alarm”
AWS Monitoring (CloudWatch – Failed Logins)
• Exact steps exist here:
http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudwatch-alarms-for-cloudtrail.html#cloudwatch-alarms-for-cloudtrail-signin
AWS Monitoring (CloudWatch) – Filter Patterns
• Create your own custom filter patterns, here is a resource for that:
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html
AWS + Splunk
AWS + Splunk
• Splunk is a pretty great resource for monitoring activity
• I’m fairly new to Splunk myself• Two separate plugins:
Splunk App for AWShttps://splunkbase.splunk.com/app/1274/Splunk Add-Onhttps://splunkbase.splunk.com/app/1876/
AWS + Splunk
• Examples of things you can view:– Billing– Topology– Usage– IAM Activity– SSH Key Pair Activity– User Activity– Network ACL(s)– VPC Activity
and a lot more…
AWS + Splunk
• Pretty Screenshot 1
AWS + Splunk
• Pretty Screenshot 2
AWS + Splunk
• Pretty Screenshot 3
AWS + Splunk
• Add plugins (apps) to Splunk
AWS + Splunk
• Splunk will need an AWS account in order to retrieve data
• Create account(s) for Splunk, grab the necessary permission policy from here:
http://docs.splunk.com/Documentation/AddOns/released/AWS/ConfigureAWSpermissions
AWS + Splunk
• Add the newly created account(s) to Splunk Add-on for AWS app - requires AWS access token id/secret
AWS + Splunk
• Configure AWS App for Splunk, add account(s), configure each input accordingly:
AWS + Splunk
• To view things like IAM Activity…– Subscribe to a cloudtrail log via SNS– Utilize SQS and subscribe SQS to an SNS
Topic
AWS Monitoring Recap
• Alert yourself when things change• This will get noisy, find a way to filter that which
is important– If it’s a high risk event, send an
SMS/Slack/Email blast• At a minimum, alert yourself when odd things
occur… like:– Billing increases past your normal spend– When somebody authenticates as Root– When someone has a login failure
AWS Monitoring Recap
• Interesting Quora thread: – https://www.quora.com/My-AWS-account-was-hacked-an
d-I-have-a-50-000-bill-how-can-I-reduce-the-amount-I-need-to-pay
• Highlights from the article:– AWS has “a review board of sorts” to determine if you
should be refunded– Bots are scouring GitHub searching for exposed access
keys– One of the more AWS-seasoned responders mentioned
doing part of what we discussed here today to avoid it– A decent number of the people posting on this thread
said “Yes, happened to me too”
AWS Restoration & Recovery
Plan to fail, just don’t fail to plan
AWS Restoration & Recovery – Basic Incident Response (IR)
• Understand who to contact if things go bad
• Understand how to communicate (ex: “speak only over the phone”)
• Understand what information to parse
• Understand where your backups are located and how they are secured
AWS Restoration & Recovery – Basic IR
• Do not USE AWS TO BACKUP YOUR AWS
• Offsite backups (meaning, off AWS site)
• Common things to back-up:– Databases/ Snapshots– S3 Buckets– EBS Volumes– CloudFormation Templates
AWS Restoration & Recovery – Basic IR
• Resources:– http://stackoverflow.com/questions/1708
7542/backup-solutions-for-aws-ec2-instances
– https://github.com/Scalr/installer-ng– http://www.n2ws.com/blog/3-ways-ec2-w
indows-backup-and-recovery.html
Presentation Recap
Summary
Recap
• Makes your environment harder to reach… for the bad guys– Limit what stolen or “otherwise
obtained” access keys or credentials could be used to do
– Prevent them being stolen in the first place
• Alert yourself to anomalies• Have a plan for if things go bad• Stay safe out there!