Enterprise User Management - DOAG

Welcome

Enterprise User Management On A

Kerberos Secured Oracle BDA

Erkan ÜLGEY

2016-11-15

Who I am?

2010 Computer Eng. @ I.T.U2010 - DBA2012 - ~ M.Sc. @ BOUN2013 - ~ Blogger @ kamudba.blogspot.com.tr2013 - ~ 11g OCP & Perf. Tuning Cert. Expert2015 - DBA and BigData Administrator

Managing entire bigdata environment in companyAdministration of Exadata databasesOTechMag publication,Talks

2016 –~ Oracle BigData Cert. Impl. Specialist.

Currently enjoying with Big Data

Keep in touch

[email protected] tr.linkedin.com/in/erkanulgeytwitter.com/erkanulgey

Today in short

Establish an environment that lets• People feel working in a familiar environment

Oracle database• People work with standarts

Roles, grants , ..• DBA solve integration problems to company

EUM – What/why?

Security of your business Grouping people Managing access

actually just deny Custom rules

How we do that traditionally?

EUM – What/why?

Sample user management for an Oracle database

Person joins company … wish luck for him

Person added to Active Directory group

Define an OID user maps to Person

Groups premapped to database roles Give access grants to that database role

EUM – Big Data

How about Hadoop?

No OID Not mature Security mechanism Complicated integration of tools Problems of Integration to Company

Where to locate How to connect with other systems Role modeling

Developer/User access methods to Hadoop Define development area Via Commandline, via GUI

How to limit resources to people? – That is an issue for another event

DBA in chaos

EUM – Big Data

Lets start to integrate

GOAL : Design Big Data env. like working in Oracle asap.

First determine the needs. Enable security? ….. YES How users/developers access? How Oracle BDA connects with data sources Role/grant modeling of company What tools that Unix admins use

In next slides, we will see how hadoop admin maintains EUM in a fully secure environment

EUM – Big Data


That gateopens toBigData!

EUM – Big Data

Dev/User wants to access Big Data

!Secure!

EUM – Big Data

Lets zoom in detail.

Enable Kerberos Security on Oracle BDA.Prepare user access gates to hadoop

Locate BDA within your engineerd systems.

KerberosEnabledSecurity

Layer

EUM – Big Data

Keep zooming

- Suppose Dev/User wants to use GUI.

EUM – Big Data

Keep zooming

- Suppose Dev/User wants to use commandline.

Hive, Beeline, Java tools, spark jobs,etc

Hadoop fs

!ACL check!

EUM – Big Data

So far, we see the how a user interacts with hadoop , now lets go further to system level

Establish company role/grant modeling on hadoop side.

In this talk , our environment uses AD Kerberos.

The main step > Users must use a valid Kerberos ticket! to access hadoop.

> Ensure that all cluster and big data components are kerberos enabled on configuration

Assume we have following role modeling on company

Group 1 (members) user1, user2, user3 (access) dbtable1,dbtable2…Group 2 (members) user11, user22, user33 (access) dbtable11,dbtable22…

Remember, grants are given to groups not individual users.

EUM – Big Data

Developer uses shell [EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$beelineJava HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release16/09/11 20:44:57 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it.scan complete in 2msConnecting to jdbc:hive2://HIVEHOST:10000/default;principal=hive/HIVEHOST@HOSTREALM16/09/11 20:44:59 [main]: ERROR transport.TSaslTransport: SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]…Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

…Error: Could not open client transport with JDBC Uri: jdbc:hive2://HIVEHOST:10000/default;principal=hive/HIVEHOST@HOSTREALM: GSS initiate failed (state=08S01,code=0)Beeline version 1.1.0-cdh5.7.1 by Apache Hive0: jdbc:hive2://HIVEHOST:10000/default (closed)>

EUM – Big Data

Establish company «role/grant modeling » on hadoop side.

In this talk, user management on BDA done with Centrify.

For the authentication, use Sentry.

Sentry watches HIVE process, applies rules and allow/deny user.

EUM – Big Data


For Hive access,Define hive databasesCreate rolesGive access grants for table/db to rolesMap roles to groups.

0: jdbc:hive2://HIVEHOST:10000/default> show databases;..INFO : OK+------------------------+--+| database_name |+------------------------+--+| db1 || db2 || tmp1 || tmp2 |...+------------------------+--+27 rows selected (0.222 seconds)

0: jdbc:hive2://HIVEHOST:10000/default> show grant role role1;...INFO : OK+------------------------------------------------------+------------+------------+---------+-------------------------+-----------------+------------+---------------+-------------------+----------+--+| database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | grantor |+------------------------------------------------------+------------+------------+---------+-------------------------+-----------------+------------+---------------+-------------------+----------+--+| db1 | | | | role1| ROLE | * | false | 1469108056269000 | -- || hdfs://CLUSTER/HDFSPATH1| | | | role1| ROLE | * | false | 1448268203604000 | -- || db2 | table1 | | | role1| ROLE | select | false | 1466420085615000 | -- || tmp1 | table2 | | | role1| ROLE | select | false | 1466603319236000 | -- || hdfs://CLUSTER/HDFSPATH2| | | | role1| ROLE | * | false | 1471593510770000 | -- |+------------------------------------------------------+------------+------------+---------+-------------------------+-----------------+------------+---------------+-------------------+----------+--+6 rows selected (0.091 seconds)0: jdbc:hive2://HIVEHOST:10000/default>

0: jdbc:hive2://HIVEHOST:10000/default> desc database test;..INFO : OK+----------+----------+-------------------------------------------------+-------------+-------------+-------------+--+| db_name | comment | location | owner_name | owner_type | parameters |+----------+----------+-------------------------------------------------+-------------+-------------+-------------+--+| test | | hdfs://CLUSTER/HDFSPATH/test.db | hive | USER | |+----------+----------+-------------------------------------------------+-------------+-------------+-------------+--+1 row selected (0.078 seconds)0: jdbc:hive2://HIVEHOST:10000/default> show roles;...INFO : OK+-----------------------------+--+| role |+-----------------------------+--+| role_global_dba || role1 || role2 |...+-----------------------------+--+15 rows selected (0.074 seconds)

EUM – Big Data


You can execute grant/revoke statements if you/your group is added to Sentry admin groups

0: jdbc:hive2://HIVEHOST:10000/default> grant select on database tmp1 to role role2;...INFO : OKNo rows affected (0.411 seconds)

EUM – Big Data


You can execute grant/revoke statements if you/your group is added to Sentry admin groups

With Apache Senty, we cannot give grants to users . Only roles are supported.

beeline> grant role admin_role to user erkanul;Error: Error while compiling statement: FAILED: SemanticException Sentry does not allow grant/revoke on: USER (state=42000,code=40000)

beeline> SHOW ROLES;ERROR : Error processing Sentry command: Access denied to erkanul. Server Stacktrace: org.apache.sentry.provider.db.SentryAccessDeniedException: Access denied to erkanul…

EUM – Big Data


The main step is assigning roles to OS groups of user. Here we can map roles to groups.

Important! Sentry must get userid information when assinng role or determine user role whenhive access..

To maintain all users in company to bda , Centrify helps us- Do not define local users on BDA nodes, just make them LISTED- When you issue and ID command it must show like following

[BDANODE]~$id erkanuluid=1146673329(erkanul) gid=1119902816(dba_group) groups=1119902816(dba_group),1146698289(bda_allowed_groups),1119932894(db_bda_allowed_groups_centrify),1119916801(sys_dba_group_centrify)

EUM – Big Data


The main step is assigning roles to OS groups of user. Here we can map roles to groups.

0: jdbc:hive2://HIVEHOST:10000/default> Grant role role1 to group `os_group1`;...INFO : OK

0: jdbc:hive2://HIVEHOST:10000/default> SHOW ROLE GRANT GROUP `os_group1`;...INFO : OK+-----------------------------+---------------+-------------+----------+--+| role | grant_option | grant_time | grantor |+-----------------------------+---------------+-------------+----------+--+| role1 | false | NULL | -- |+-----------------------------+---------------+-------------+----------+--+2 rows selected (0.176 seconds)0: jdbc:hive2://gbbdap35:10000/default>

EUM – Big Data

Developer uses shell [EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$beeline….0: jdbc:hive2://HIVEHOST:10000/default> use tmp1;Error: Error while compiling statement: FAILED: SemanticException No valid privilegesUser dev1 does not have privileges for SWITCHDATABASEThe required privileges: Server=server1->Db=tmp1->Table=*->Column=*->action=insert;Server=server1->Db=tmp1->Table=*->Column=*->action=select; (state=42000,code=40000)0: jdbc:hive2://gbbdap35:10000/default>

EUM – Big Data

Developer uses shell [EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$beeline….0: jdbc:hive2://HIVEHOST:10000/default> use tmp1;...INFO : OKNo rows affected (0.125 seconds)0: jdbc:hive2://HIVEHOST:10000/default> show tables;...INFO : OK+------------------+--+| tab_name |+------------------+--+| table1 || table2 || table3 |..+------------------+--+13 rows selected (0.182 seconds)0: jdbc:hive2://HIVEHOST:10000/default>

EUM – Big Data


With Sentry, you can secure all your hive access.

When a user attempts to create an external table to a location which the user has access on HDFS, hive has also access that location.

Sentry stops «create table» statement.

Bug Alert SENTRY-1001 : Improve usability of URIs and URI privileges

CREATE EXTERNAL TABLE foo LOCATION '/path/to/table‘

(Ensure that Hive has access to that location - ACL)grant all on URI'hdfs://CLUSTER/HDFSPATH' to role role1;

https://issues.apache.org/jira/browse/SENTRY-1001

EUM – Big Data


For direct Hdfs access,Define ACLs

Best practise: Define solid paths for hive databases , some devs/apps may want direct access to hive db location

Use hadoop fs –setfacl / getfacl commands and remember subgroups

[EDGENODE]~$setbdaenv -testSetting for BDA Test Environment[EDGENODE]~$ kinit...[EDGENODE]~$hadoop fs -ls /Found 10 itemsdrwxrwxr-x - hdfs supergroup 0 2015-08-21 15:29 /appdrwxrwxr-x - oracle supergroup 0 2016-01-27 10:08 /datadrwxrwxr-x - hdfs supergroup 0 2015-07-28 09:50 /etldrwx------ - hbase hbase 0 2016-08-24 13:03 /hbasedrwxr-xr-x - hdfs supergroup 0 2015-07-08 10:03 /metadatadrwxrwxrwx - solr solr 0 2016-07-15 14:43 /solrdrwxrwxr-x - solr solr 0 2016-08-03 16:36 /solr2drwxr-xr-x - hdfs supergroup 0 2016-05-10 19:24 /systemdrwxrwxrwt - hdfs supergroup 0 2016-09-12 15:20 /tmpdrwxr-xr-x - hdfs supergroup 0 2016-09-01 12:47 /user[EDGENODE]~$

[EDGENODE]~$hadoop fs -getfacl /HDFSPATH/tmp1_dbpath <-- corresponds to hive db path# file: //HDFSPATH/dbpath# owner: hive# group: supergroupuser::rwxgroup::rwxgroup:os_group1:rwxgroup:hive:rwxmask::rwxother::---default:user::rwxdefault:group::rwxdefault:group:os_group1:rwx <-- default setting for newly created filesdefault:group:hive:rwxdefault:mask::rwxdefault:other::---

EUM – Big Data


For direct Hdfs access,Define ACLs

Centrify Alert ! ACL ( and also Sentry) must gather userid information to execute rules

[EDGENODE]~$hadoop fs -ls /HDFSPATH/tmp1_dbpath ls: Permission denied: user=user1, access=EXECUTE, inode="/HDFSPATH/":oracle:supergroup:drwxrwx---

[BDANODE]$id user1id: dev1: No such user

EUM – Big Data

How to entegrate users to HUE. ( we are all familiar with SQL)

Remember ! Never define all users localy on hadoop nodes

Configure HUE with LDAP, enable users to access with their LDAP account, so no need to maintain a HUE password.

In this talk, we want tofilter groups to access BDA and not define them all on HUE. Restrict the menus that they can access.

EUM – Big Data

How to entegrate users to HUE.

Approaches – HUE & LDAP Integration

- Provide LDAP settingsLDAP URL , LDAP Search BaseLDAP Bind User Distinguished NameCheck «Use Search Bind Authentication»

EUM – Big Data



Important properties: LDAP User FilterLDAP Username AttributeLDAP Group FilterLDAP Group Name AttributeLDAP Group Membership Attribute

Lets see some cases for HUE user management

You may need an Active Directory explorer tool

EUM – Big Data



Case 1: Admin can add all users to HUE.

Approach:Just Use a simple LDAP User Filter.

LDAP Username Filter: (objectClass=user)LDAP Username Attribute : sAMAccountName

- ADD the user with AD distinguished name

- Bug alert : HUE-1711 [core] LDAP username import lowercase

EUM – Big Data



Case 2: The groups who access HUE are fixed. But we want to limit the menus they access like security or users page..

Approach:Just Use LDAP User Filter and define groups on HUE and filter menus.

LDAP Username Filter: (&(objectClass=user)(|(memberof=CN=os_group1,OU=Distribution Groups,OU=Groups,OU=AllUsers,LDAPSEARCHBASE)(memberof=CN=os_group2,OU=Distribution Groups,OU=Groups,OU=AllUsers,LDAPSEARCHBASE)(memberof=CN=os_group3,OU=Distribution Groups,OU=Groups,OU=AllUsers,LDAPSEARCHBASE)))LDAP Username Attribute : sAMAccountName

- ADD the group with AD distinguished name

EUM – Big Data



Case 3: The groups who access HUE are incresingBut we want to limit the menus they access like security or users page..

Approach:Define a top AD group and add all the groups

BDA_ALLOWED_GROUPS : group1,group2,…groupN

LDAP Group Filter : (memberOf=CN=BDA_ALLOWED_GROUPS,LDAPSEARCHBASE)LDAP Group Name Attribute : sAMAccountNameLDAP Group Membership Attribute : member

Bug Alert HUE-2040 [useradmin] Synchronize nested ldap groups

[] access INFO 10.242.5.28 erkanul - "POST /useradmin/users/add_ldap_groups HTTP/1.1"[] views DEBUG Importing user CN=OS_Group1,LDAPSEARCHBASE[] views WARNING Could not get LDAP details for users with pattern CN=OS_Group1,LDAPSEARCHBASE

EUM – Big Data



Case 3: The groups who access HUE are incresingBut we want to limit the menus they access like security or users page..

Approach:Define a top AD group and add all the groups

BDA_ALLOWED_GROUPS : group1,group2,…groupN

- For adding user to HUE- Define LDAP User Filter :

(&(objectClass=user)(|(memberof:1.2.840.113556.1.4.1941:=CN=BDA_ALLOWED_GROUPS,LDAPSEARCHBASE)))

LDAP Rule 1.2.840.113556.1.4.1941 -> Recursive add all the users under groups and subgroups

Multiple domains in Company – HUE does not care cross-check domains

EUM – Big Data



For case 1 -> Add user one at a timeFor case 2 -> Refresh/sync LDAP Groups on HUE to get new members

+___________________________________________________________________For those scenarios, new members must be LISTED at Centrify.

For case 3 -> Add new bda user group to BDA_ALLOWED_GROUPSAdd user with * to get all new members from all subgroups

+____________________________________________________________________For this scenario, just add BDA_ALLOWED_GROUPS to Centrify and it will refresh to get new members.Less manuel action

EUM – Big Data



You can change attributes of groups for menu accessBecause we dont want them to see eveything

EUM – Big Data


First determine the needs. Enable security? ….. YES How users/developers access? How Oracle BDA connects with data sources Rol/grant modeling of company What tools that Unix admins use

In next slides, we will see how hadoop admin maintains EUM in a fully secure environment

EUM – Big Data

Yes, We integrated !

- We enabled Kerberos- We designed roles/grant modeling (wrt. Company)- We defined HIVE databases and hadoop file organizations- We gave necessary grants

EUM – Big Data

Note!

- Bug Alert SENTRY-1001 : Improve usability of URIs and URI privileges- Bug Alert HUE-2040 [useradmin] Synchronize nested ldap groups- Bug Alert HUE-1711 [core] LDAP username import lowercase-- Ensure users are LISTED on Centrify ( on BDA nodes )- Define top AD group : BDA_ALLOWED_GROUPS- Be careful about multiple domains

- Create HIVE databases different than /user/hive/warehouse- Ensure all hadoop components use Kerberos Security ( Keytab errors)

BONUS : Establish an EDGENODE and keep developers out of BDA NODES

EUM – Big Data

BONUS : Establish an EDGENODE and keep developers out of BDA NODES

- Create a unix/linux server ( maybe one node from BDA ) - Download client configs from your hadoop cluster- Make configuration settings ( dynamically for prod and test )- Define aliases to command line tools

- Do not use HIVE, it has no security

alias beeline='beeline -u "jdbc:hive2://HIVEHOST:10000/default;principal=hive/HIVEHOST@YOURREALM"'alias hive='echo '\''You can not run hive, disabled for security reasons'\'''alias impala-shell='impala-shell -k -i IMPALAHOST'

settestenv () {export JAVA_HOME=/usr/java/default/export BDAenv="test"export HADOOP_HOME=/usr/lib/hadoopexport HIVE_HOME=/usr/lib/hiveexport HBASE_HOME=/usr/lib/hbaseexport SPARK_HOME=/usr/lib/sparkexport HBASE_CONF_DIR=/home/scripts/oracle/testenv/hbase/hbase-conf/export HIVE_CONF_DIR=/home/scripts/oracle/testenv/hive/hive-conf/export HADOOP_CONF_DIR=/home/scripts/oracle/testenv/hdfs/hadoop-conf/export YARN_CONF_DIR=/home/scripts/oracle/testenv/yarn/yarn-conf/export SPARK_CONF_DIR=/home/scripts/oracle/testenv/spark/spark-confexport HADOOP_CONF_DIR=$HADOOP_CONF_DIR:$HIVE_CONF_DIR:$HBASE_CONF_DIRecho "Setting for BDA Test Environment"bash --init-file /etc/profile}

Enterprise User Management On A Kerberos Secured Oracle BDA

Q&ADanke

Enterprise User Management - DOAG

Documents

Transcript of Enterprise User Management - DOAG