Apache Ranger

Comprehensive Security for Enterprise Hadoop

Developer Overview

Presenter: Tushar Dudhatra

Agenda

➔ Security Approach within Hadoop

➔ Core Features Of Ranger

➔ Technology Stack

➔ Ranger Architecture◆ High Level Overview

◆ Code flow

➔ How to build and run ?

➔ Typical things to watch out during development

➔ Questions !!!

Security Approach within HadoopFrom Slide No: #4To Slide No: #6

Security today in Hadoop with HDPHD

P 2.

3

Centralized Security Administration with Ranger

AuthenticationWho am I ?prove it !

AuthorizationWhat can I do?

AuditWhat did I do?

Data ProtectionCan data be encrypted at rest and over the wire?

Fine grain access control with Apache Ranger

Centralized audit reporting with Apache Ranger

➔ Wire encryption in Hadoop

➔ Native and partner encryption

➔ Kerberos➔ API security

with Apache Knox

Example Access Scenario

Access HDFS File/Folder

Access HBase Tables

Access Hive Tables

Example Access Scenario With Ranger

Access HDFS File/Folder

Access HBase TablesAccess Hive Tables

Plugins DownloadDefined Policies fromCentralized Policy store

Plugins Write to Centralized Audit store

Core Features Of RangerFrom Slide No: #8To Slide No: #8

Core Features Of Ranger>> Authorization Support for HDFS, Hive, HBase, Knox, Storm, Solr, Kafka, Yarn>> Audit destination support for HDFS, DB, SOLR (Including SolrCloud too)>> Support for High Availability (HA)>> Support for Transparent Data Encryption with KMS implementation

>> Dynamic policies with many other attributes>> Stack based implementation of Plugins

>> Internal permission model within Ranger Admin UI

>> Tighter integration with Apache Ambari

>> Policy Manager supports six different databases

Technology StackFrom Slide No: #10To Slide No: #11

Ranger Implementation - components

➔ Central Interface for security administration.

➔ Users can manage policies, view audit activities, manage users and permissions

➔ Sync utility to pull users and groups from Unix or LDAP or AD.

➔ User/group info is stored within Ranger Admin DB and used for policy definitions.

➔ Lightweight Java programs within Hadoop components. ➔ Pull in policies from policy admin and store them locally in cache. ➔ Act as Authorization module and evaluate user requests against security

policies before granting access. ➔ Collect data from user request and store this data into the audit store.

Ranger Policy-Admin Server Ranger User-Sync Server

Ranger Security Plugins

Ranger Policy-Admin Tool

Backbone.Model REST Interface

Client Server

DB➔ Backbone➔ jQuery➔ Underscore

js➔ Marionette➔ Bootstrap

➔ Spring Framework➔ Spring Security➔ EclipseLink➔ JPA➔ tomcat➔ Apache Commons➔ Jersey➔ JUnit➔ DB Connector➔ log4j

Database could be:➔ MySql➔ Oracle➔ MS-SQL Server➔ SQLAnywhere➔ Postgres

Ranger ArchitectureFrom Slide No: #12To Slide No: #17

Typical Flow of Ranger Admin

UI Server DB

Backbone.jshtmlcss

REST Mgr Service

DAOMySqlPostgresOracleSQLAMSSQL

3 Pillars of Ranger Admin (Policy Manager)Service Def

Service-1 Service-3Service-2

Policy-1

Policy-2

Policy-3

Policy-4

Policy-5

Policy-6

Ranger HDFS Service Def{"id":1,"name": "hdfs","implClass":"RangerServiceHdfs","label": "HDFS Repository","description": "HDFS Repository","guid": "8e9b-d5d377284b2d","resources": [

{"itemId": 1,"name": "path","type": "path","level": 10,"parent": "","mandatory": true,"lookupSupported": true,"recursiveSupported": true,"excludesSupported": false,

"matcher":"RangerPathReurceMatcher",

"matcherOptions": { "wildCard":true, "ignoreCase":false },

"validationRegEx":"","validationMessage": "","uiHint":"","label": "Resource Path","description": "HDFS file

path"}],

"accessTypes": [{

"itemId": 1,"name":

"read","label":

"Read"},{

"itemId": 2,"name":

"write","label":

"Write"},{

"itemId": 3,"name":

"execute","label":

"Execute"}],

"configs": [{

"itemId": 1,"name":

"username","type":

"string","subType":

"",

"mandatory": true,

"validationRegEx":"",

"validationMessage": "","uiHint":"","label":

"Username"}],

"enums": [{

"itemId": 1,"name":

"authnType","elements":

[{

"itemId": 1,

"name": "simple",

"label": "Simple"},{

"itemId": 2,

"name": "kerberos",

"label": "Kerberos"}],

"defaultIndex": 0}],

"contextEnrichers": [],"policyConditions": []

}

For complete JSON Please Click Here

Example Policy Creation{ "service": "hadoopdev",

"name": "policy-1",

"isEnabled": true,

"description": "securing /apps directory",

"isAuditEnabled": true,

"resources": { "path": { "values": ["/apps"], "isRecursive": true } },

"policyItems": [

{ "users": ["admin"], "delegateAdmin": true,

"accesses": [ { "type": "read", "isAllowed": true }, { "type": "write", "isAllowed": true }, { "type": "execute", "isAllowed": true }]}]

}

Continue...

Example Policy Creation

XXPolicyDaoBASE

DAO

XXPolicyItemDao

XXPolicyResDao

XXPolicyResMapDao

ServiceREST

ServiceDBStore RangerPolicyService

RangerPolicy → XXPolicy

DATABASE

RangerPolicy Object

Validation will be done at REST layer

Create RangerPolicy

Now Create Child Objectsi.e Create XXPolicyItemCreate XXPolicyResCreate XXPolicyResMap

How to Build and Run?From Slide No: #19To Slide No: #22

Prerequisites:

➔ JDK 7+

➔ Database you need anyone of these for Policy/Audit DB:

◆ MySQL (5.6+)

◆ ORACLE DB (11g+)

◆ Postgres (8.4+)

◆ MSSQL Server (2012)

◆ SQLAnywhere (17+) ➔ Maven (3.2 +)➔ Ranger Admin process requires approximately 1.5GB of RAM

Building Ranger From Source➔ Get the ranger source

◆ git clone -b master git@github.com:apache/incubator-ranger.git ranger

➔ Now build the source

◆ mvn clean compile package assembly:assembly -e

➔ Verify all the tar files under target dir i.e

◆ ranger-{version}-{component}-plugin.tar.gz

◆ ranger-{version}-admin.tar.gz

◆ ranger-{version}-usersync.tar.gz

Install/Configure Ranger Admin➔ Lay down the binaries into appropriate places.

➔ Untar ranger-{version}-admin.tar.gz

➔ Open install.properties and update it to set values DB name, DB host, DB username:password and other required details

➔ Once all the required properties are updated, execute setup.sh to run setup

➔ Start the Ranger Admin

➔ service ranger-admin start

➔ You can verify by visiting the external URL of the server using browser, for example :

◆ http://<Host Address>:6080

➔ Logs are in ews/logs folder. The path is relative to where you have installed ranger-admin. Check xa_portal.log and catalina.out files for ERROR and WARN log messages

Install And Configure Other Components of Ranger

➔ After installing ranger-admin, next step is to configure and install ranger-usersync.

➔ After installing ranger-usersync, depending upon your need you can enable required ranger-plugins.

➔ To configure and install usersync, you need to follow the same steps; i.e Update install.properties as per your setup and then run setup.sh and start ranger-usersync service.

➔ To configure plugins, you need update install.properties and run enable-{complonent}-plugin.sh and restart respective component to make ranger-plugin effective.

➔ For detailed documentation guide of installation, you can visit: https://cwiki.apache.org/confluence/display/RANGER/Ranger+Installation+Guide

Typical things to watch out during DevelopmentFrom Slide No: #23To Slide No: #23

Some Healthy tips for Ranger Developers➔ When you create a new file, don’t forget to add Apache Licence on top of the file.

Licence is required for all the files i.e .java, .js, .xml etc.➔ It is strongly recommended to write Unit Tests(JUnit) whenever you write a piece of

code or you create a new file or make changes in existing code.

➔ It is not at all recommended to use -DskipTests=true or similar command which skips Unit Tests while building ranger using maven command. Skipping unit tests is not advisable.

➔ If you are introducing a new functionality it is also recommended to write system tests(PyTests) as a proof of verification of new functionality.

➔ If you are changing existing core business logic; it is recommended to run System Tests before you commit or push your code to make sure that it’s not breaking the functionality.

➔ Clean code always Matters. It’s recommended to use camel case convention to keep code clean, consistent and readable as we have used that throughout the application.