1

Hadoop Operations: How to Secure and Control Cluster AccessEric SammerEngineering Manager, Cloudera – Author, Hadoop Operations

2

We’re here to talk about…

•How common security constructs map onto services•How these constructs work in Hadoop•Security model and options for a few critical components

•A few DOs and DON’Ts

3

Warning

•Security in distributed systems is complicated•This is just a whirlwind tour – Do your homework•Assumptions

• You’re familiar with Hadoop’s architecture and functionality• You have a basic understanding of Kerberos

4

The Three Questions

•Identity: Who are you?•Authentication: Can you prove it?•Authorization: Are you allowed to do that?

5

Hadoop’s “Simple” Mode

•Identity: Usually the OS user of the client application•Authentication: Trust•Easy to impersonate other users•Stop good users from doing silly things•The default

6

Hadoop’s “Simple” Mode

•Use simple mode when:• No regulatory or compliance concerns• All users are trusted• Single purpose cluster (single-tenancy)

7

Hadoop’s “Secure” Mode

•Identity: Local part of the Kerberos principal•Authentication: Kerberos•User impersonation not possible except in specific (admin-configured) situations

8

Hadoop’s “Secure” Mode

•Use secure mode when:• Real regulatory concerns• Untrusted users• Running on untrusted infrastructure or in an untrusted

environment• Multi-purpose cluster (multi-tenancy)

9

Identity Management

•Always• Use a central user database/directory service for OS users• Wire up the Kerberos KDC to use the central directory

•Never• Use service users (e.g. hdfs, mapred) for anything other than

running services• Share accounts, even for admin purposes

10

Authentication

•Simple mode: Trust what the client provides•Secure mode: Kerberos

• Keytabs for services• Many options: Passphrase, M/TFA, X.509 for users• Depends on Kerberos implementation

11

Authorization

•Inherently service specific•Granularity of control varies by platform component•Examples

• Filesystem object-level, POSIX-style• Role-based access control (RBAC)• Access control lists (ACLs)• Deferral to underlying components

12

HDFS Security Model

•POSIX-style users and groups•Traditional Unix-style octal permissions

• Files: no execute, sticky, setuid, setgid• Directories: no setuid, always behave as if setgid is set

•Authorization checks performed by NameNode

13

HDFS User Levels

User Level Privileges Description and Notes

Cluster super user All User who started the daemons. Default: hdfs

Administrators AllConfiguration property dfs.permissions.supergroup specifies the name of the group of admins. Default: supergroup

Normal user Object-level All other users are beholden to the file and directory permissions, as specified.

14

MapReduce Security Model

•Configurable job queues•Queues have associated ACLs•ACLs control job submission and administrative ops•Authorization checks performed by JobTracker

15

MapReduce User Levels

User LevelPrivilege

sQueue Description and Notes

Cluster super user All All User who started the daemons. Default:

mapred

Cluster admins All AllConfiguration property mapred.cluster.administrators specifies the admin ACL.

Queue admins All SingleConfiguration property mapred.queue.queue-name.acl-administer-jobs specifies the admin ACL.

Job ownerSubmit,

Admin on own jobs

Queue containing

job

Configuration property mapred.queue.queue-name.acl-submit-job specifies the submission ACL.

16

Systems on top of MapReduce

•Hive/Impala are the most featureful today• Without Sentry: Defers to HDFS object permissions• With Sentry, fine-grained RBAC on logical constructs (New!)

• Scope: Server, database, table, view• Privileges: ALL, SELECT, INSERT, TRANSFORM• Removes direct access to files• Supports traditional techniques for controlling column-level access

(i.e. views without sensitive columns)

•Everything else: HDFS object permissions

17

A note on auditing...

•Winds up being service-specific•Cloudera Navigator handles this (and more)

18

What we didn’t talk about

•Configuration and deployment• Lots of options, lots of moving parts• Integration with existing infrastructure• Cloudera Manager turns days or weeks of work into minutes

or hours; built to handle exactly these challenges•The other 80%: YARN applications, ZooKeeper, Flume, Sqoop, Oozie, Hue, Cloudera Search (Solr), multi-tenant gateway services, all of the administrative web interfaces, encryption of data at rest and on the wire, network footprint and exposure, ...

19

Further reading and references

•Hadoop OperationsChapter 6: Identity, Authentication, and Authorization (E. Sammer, O’Reilly)

•Kerberos: The Definitive Guide(J. Garman, O’Reilly)

•CDH4 Security Guide•CDH4 Sentry Guide•Cloudera Manager•Cloudera Navigator

Submit questions in the Q&A panel

Watch on-demand video of this webinar and many more at http://cloudera.com

Follow Eric @esammer

Follow Cloudera @ClouderaU

Learn more at Strata + Hadoop World:

http://tinyurl.com/hadoopworld

Thank you for attending!