Secure Keystone Deployment:Lessons Learned and Best Practices

Priti DesaiSr. Software Engineer

Secure Keystone Deployment 1

The Symantec Team• Cloud Platform Engineering

– We are building a consolidated cloud platform that provides infrastructure and platform services for next generation Symantec products and services

• Me– In Security for over 6 years– Symantec Insight - Reputation Based Security– Symantec Data Analytics Platform– OpenStack Engineer - Keystone– OpenStack Security Group– Cop Open Source

Secure Keystone Deployment2

OpenStack Security Group

Secure Keystone Deployment3

security notes

Retrieved from http://www.openstack.orgRetrieved from http://docs.openstack.org

Secure Keystone Deployment

Why is Keystone security critical?

What is Keystone?

How is Authentication process implemented in Keystone?

How is Authorization mechanism implemented in OpenStack?

AuthN Overview

Secure Keystone Deployment5

Cloud User

Cloud User Identity(SQL/LDAP)Keystone

Token (SQL)Identity (SQL/LDAP)Keystone

Token (SQL)

Request sent withUsername and Password

Verify username and password (hash of

password)

Successful verification

Request metadata for user tenant relationship

Assignment(SQL)

Assignment (SQL)

User tenant relationshipinformation

Request to generate new token

Response with new token

Response with token

AuthZ Overview

Secure Keystone Deployment6

Cloud User

Cloud User KeystoneOpenStackService

KeystoneOpenStack Service

Request sent with session token

Verify session token

Successful verification

Is this token correct?Does it allow the service

usage?

Service executes the request

Response with success

Secure Keystone Deployment

Why is Keystone security critical?

Does it store/transmit any sensitive information?

What kind of cloud asset does it store?

Is any type of attack possible on Keystone? Can it bring down the entire cloud?

Keystone Security is Critical

Secure Keystone Deployment8

• Gatekeeper• Access to OpenStack Cloud

• Assets • Users• Passwords• Tokens• Roles• Catalog

• Vulnerable to DoS

Retrieved from http://internet.phillipmartin.info

Retrieved from http://blogs.citypages.com

Retrieved from http://assets.nydailynews.com

What was our approach to identifying key vulnerabilities?

Secure Keystone Deployment9

Security Risks

Secure Keystone Deployment10

• Global Security Office Threat Model Penetration Tests Traceability Matrix

Retrieved from http://www.technetics.com.au

Threat Model

Secure Keystone Deployment11

Secure Keystone Deployment12

Spoofing

Tampering

Repudiation

Information Disclosure

Denial of Service

Elevation of Privileges

Threat Model

What kind of security deficiencies did we discover?

Secure Keystone Deployment13

Secure Keystone Deployment14

Attack: Keystone user credential theft

Attack: Insecure file permissions on Keystone.conf

Keystone.conf

Attack: Access to cloud admin privileges for almost free

Attack: Leaking sensitive data in log messages

Attack: DoS – Authentication chaining - Havana

Attack: Unauthorized access to MySQL database

Many more …

Traceability Matrix

Secure Keystone Deployment15

✖

✖

✖

Keystone User Credential Theft

Secure Keystone Deployment16

Mitigate: Secure Communication - SSL

Secure Keystone Deployment17

Hardware Load Balancer Hardware Load Balancer

Keystone KeystoneKeystone

SSL Client

SSL Server

SSL Client

SSL Server

mod_ssl

35357/SSL 5000/SSL

mod_ssl

35357/SSL 5000/SSL

mod_ssl

35357/SSL 5000/SSL

Public API Admin API

Insecure file permissions on Keystone.conf

Secure Keystone Deployment18

Mitigate: • Restrict ownership to service user

- chown keystone:keystone /etc/keystone/keystone.conf

• Restrict to read and write by the owner - chmod 640 /etc/keystone/keystone.conf

hostnameabc

hostnameabc

hostnameabcuser

user

user

Access to admin privileges is almost free

Secure Keystone Deployment19

• Service Token• Bootstrap Keystone• Cloud admin privileges

• Register bad service/endpoints

Mitigate: Disable Service Token

• Comment out admin_token from /etc/keystone/keystone.conf:admin_token=e2112effd3ff05b8c88ad14e096e6615

• Remove admin token auth middleware from /etc/keystone/keystone-paste.ini:[filter:admin_token_auth]paste.filter_factory = keystone.middleware:AdminTokenAuthMiddleware.factory

Secure Keystone Deployment20

Who is the cloud admin now?

Secure Keystone Deployment21

Create Cloud Admin

• Leveraging Keystone Domain

• Before disabling service token:• Create a domain “cloud_admin_domain”• Grant “admin” role to appropriate user “Bob Smith”• Update keystone policy.json file:

• Replace:"cloud_admin": [["rule:admin_required", "domain_id:admin_domain_id"]],

• With: "cloud_admin”: [["rule:admin_required”,"domain_id:<cloud_admin_domain_id>"]],

Secure Keystone Deployment22

Leaking Sensitive Information in Log Messages

• Debug mode include plaintext request logging• Passwords• Tokens

• Mitigate:• Disable debug mode in keystone.conf with:

• With debug mode ON, upgrade keystone client:• python-keystoneclient >= 0.10.1 (OSSN-0024)

Secure Keystone Deployment23

[DEFAULT]

debug=False

Leaking Sensitive Information in Log Messages

Identity API V2 - INFO level logs contains auth tokens (OSSN-0023)

Mitigate:• Set the log level to WARN in logging.conf:

Secure Keystone Deployment24

[handler_file]class = FileHandlerLevel = WARN

Keystone DoS AttackIdentity API V3 – Authentication Chaining – CVE-2014-2828

Secure Keystone Deployment25

Keystone DoS Attack

Mitigate:• Impacted Versions: from 2013.1 to 2013.2.3• Patch applied during IceHouse rc2• Upgrade Keystone >= 2013.2.4

Secure Keystone Deployment26

Q&ALet’s talk…

Secure Keystone Deployment 27

Thank You

Priti DesaiPriti_Desai@symantec.com@pritidesai8

References• http://docs.openstack.org/developer/keystone/

• https://blog-nkinder.rhcloud.com/?p=7

• https://blueprints.launchpad.net/keystone/+spec/service-scoped-tokens

• http://docs.openstack.org/sec/

• http://www.florentflament.com/blog/setting-keystone-v3-domains.html

• https://wiki.openstack.org/wiki/Security_NotesSecure Keystone Deployment

29

References (Images)• Crime Identity Theft: http://internet.phillipmartin.info/crime_identity_theft.gif

• Computer Theft: http://blogs.citypages.com/blotter/Computer%20theft.gif

• Mickey Washington ID: http://assets.nydailynews.com/polopoly_fs/1.1864391!/img/httpImage/image.jpg_gen/derivatives/article_970/mickey13n-1-web.jpg

• Threat, Asset, and Vulnerability: http://www.technetics.com.au/images/easyblog_images/79/b2ap3_thumbnail_manage_your_risk_400_20140924-122014_1.jpg

• Openstack security Notes: http://www.openstack.org/assets/openstack-logo/openstack-one-color-alt.pdf

• OpenStack security Guide: http://docs.openstack.org/common/images/openstack-security-guide.jpgSecure Keystone Deployment

30