CTD Enabling Secure Hadoop Environments -...

22
1 © Cloudera, Inc. All rights reserved. Fred Koopmans Sr. Director of Product Management Enabling Secure Hadoop Environments 1 © Cloudera, Inc. All rights reserved.

Transcript of CTD Enabling Secure Hadoop Environments -...

Page 1: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

1©Cloudera,Inc.Allrightsreserved.

FredKoopmansSr.DirectorofProductManagement

EnablingSecureHadoopEnvironments

1©Cloudera,Inc.Allrightsreserved.

Page 2: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

2©Cloudera,Inc.Allrightsreserved.©Cloudera,Inc.Allrightsreserved.

Thefutureofgovernmentisdatamanagement

What’syourstrategy?

Page 3: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

3©Cloudera,Inc.Allrightsreserved.

Cloudera’sEnterpriseDataHubmakesitpossible

Bringallyourdatatogether

Bringallyourknowledgeworkerstogether

Bringallyourdataapplicationstogether

Runanywhere

Page 4: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

4©Cloudera,Inc.Allrightsreserved.

But,anEDHcanalsomakeajuicytarget

Alldatainoneplace?

Provideeveryoneaccesstooneplatform?

HowdoIensuresecurity?

HowdoImaintainsecurityastheplatformgrows?

Page 5: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

5©Cloudera,Inc.Allrightsreserved.

4focusareasforsecuringyourHadoopenvironment

AccessDefiningwhatusersandapplicationscan

dowithdata

TechnicalConcepts:PermissionsAuthorization

DataProtectingdatainthe

clusterfromunauthorizedvisibility

TechnicalConcepts:Encryption,Keymanagement,Datamasking

VisibilityReportingonwheredatacamefromandhowit’sbeingused

TechnicalConcepts:AuditingLineage

PerimeterGuardingaccesstothe

clusteritself

TechnicalConcepts:Authentication

Networkisolation

Page 6: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

6©Cloudera,Inc.Allrightsreserved.

PerimeterSecurityRequirements

PreserveuserchoiceofHadoopservice

Conformtocentrallymanagedauthenticationpolicies

Implementwithexistingstandardsystems

ClouderaManager

PerimeterGuardingaccesstothe

clusteritself

TechnicalConcepts:Authentication

Networkisolation

Page 7: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

7©Cloudera,Inc.Allrightsreserved.

Authentication

• Kerberosauthentication• AutomationprovidedbyClouderaManagertoleverageActiveDirectory

CDHcomponents

• LDAPandSAMLauthentication

WebUIs

• LDAPandKerberosauthentication

SQLAccess

Userauthenticates

toAD

Authenticatedusergets

KerberosTicket

TicketgrantsaccesstoServices

e.g.ImpalaUser[ssmith]Password[*****]

Page 8: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

8©Cloudera,Inc.Allrightsreserved.

NetworkIsolation

EdgeNodes

WebServers

Onlyadmins permittedaccesstofullcluster

Mostusersonlypermittedaccesstogatewayservicesrunningonclusterperiphery

Page 9: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

9©Cloudera,Inc.Allrightsreserved.

AccessSecurityRequirements

Keeponlyonelogicalcopyofdata

Createonlyonepermissionsruleforallapplicationsandallcomputeframeworks

Enforcepermissionsatcolumnandrowlevelgranularity

AccessDefiningwhatusersandapplicationscan

dowithdata

InfoSecConcept:Authorization

ApacheSentry,RecordService

Page 10: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

10©Cloudera,Inc.Allrightsreserved.

FilesystemHDFS

STORA

GECO

MPU

TE

HIVE,IMPALA

SPARK,MR

EarlydaysofHadoop:Storagepermissionsonly

• Simple“AllorNothing”permissionsforeachfile/table

But...• Tablesoftencontain10s– 100sofcolumns• Notallusersareallowedtoseeallcolumnsandrows

APPS DATAMEER SASPLATFORA TABLEAU ETC...

Page 11: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

11©Cloudera,Inc.Allrightsreserved.

Usecasesforfine-grainedaccesscontrol

Columns• Differentusergroupsneedaccesstodifferentcolumns(ex:socialsecuritynumbers)

Rows• Differentusergroupsneedaccesstodifferentrecords(ex:bysecurityclearancelevel)

Page 12: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

12©Cloudera,Inc.Allrightsreserved.

FilesystemHDFS

STORA

GECO

MPU

TE

HIVE,IMPALA

SPARK,MR

Fewyearsago:Storagepermissions+SQLAuth.

• Addscolumnandrow-levelpermissions

But...• Createsduplicatedata,duplicatepermissionsrulestosupportSparkandMR

APPS DATAMEER SASPLATFORA TABLEAU ETC...

ApacheSentry

X

Page 13: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

13©Cloudera,Inc.Allrightsreserved.

FilesystemHDFS

STORA

GE

APACHESENTRY,RECORDSERVICE

COMPU

TE

HIVE,IMPALA

SPARK,MR

UpNext:ApacheSentry+RecordService*workingtogether

• ColumnandRow-levelPermissions• Onecopyofdata• Onesetofpermissions

APPS DATAMEER SASPLATFORA TABLEAU ETC...

*inbeta

Page 14: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

14©Cloudera,Inc.Allrightsreserved.

Fine-grainedaccesscontrolwithout Sentry&RecordService*

Date/time Accnt # SSN Asset Trade Country

09:33:11 16-Feb-2015

0234837823 238-23-9876

AZP Sell US

11:33:0116-Feb-2015

3947848494 329-44-9847

TBT Buy EU

14:12:3416-Feb-2015

4848367383 123-56-2345

IDI Sell UK

09:22:03 16-Feb-2015

3485739384 585-11-2345

ICBD Buy US

11:55:3316-Feb-2015

3847598390 234-11-8765

FWQ Buy US

10:22:55 16-Feb-2015

8765432176 344-22-9876

UAD Buy UK

13:45:24 16-Feb-2015

3456789012 412-22-8765

NZMA Sell EU

09:03:4416-Feb-2015

4857389329 123-44-5678

TMV Buy US

15:55:5516-Feb-2015

4756983234 234-76-9274

DRW Buy UK

Date/time Accnt # SSN Asset Trade Country

14:12:3416-Feb-2015

4848367383 123-56-2345

IDI Sell UK

10:22:55 16-Feb-2015

8765432176 344-22-9876

UAD Buy UK

15:55:5516-Feb-2015

4756983234 234-76-9274

DRW Buy UK

Date/time Accnt # SSN Asset Trade Country

11:33:0116-Feb-2015

3947848494 329-44-9847

TBT Buy EU

13:45:24 16-Feb-2015

3456789012 412-22-8765

NZMA Sell EU

Date/time Accnt # SSN Asset Trade Country

09:33:11 16-Feb-2015

0234837823 238-23-9876

AZP Sell US

09:22:03 16-Feb-2015

3485739384 585-11-2345

ICBD Buy US

11:55:3316-Feb-2015

3847598390 234-11-8765

FWQ Buy US

09:03:4416-Feb-2015

4857389329 123-44-5678

TMV Buy US

• SplittheHDFSpermissionsoriginalfile• Usetolimitaccess

Page 15: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

15©Cloudera,Inc.Allrightsreserved.

Fine-grainedaccesscontrolwith Sentry&RecordService*

• Sentry:Definepermissionsatthetable,columnandrowlevels• Sentry+RecordService:Enforcetheseacrossallaccesspaths

Date/time Accnt # SSN Asset Trade Country

09:33:11 16-Feb-2015

0234837823 238-23-9876

AZP Sell US

11:33:0116-Feb-2015

3947848494 329-44-9847

TBT Buy EU

14:12:3416-Feb-2015

4848367383 123-56-2345

IDI Sell EU

09:22:03 16-Feb-2015

3485739384 585-11-2345

ICBD Buy US

11:55:3316-Feb-2015

3847598390 234-11-8765

FWQ Buy US

10:22:55 16-Feb-2015

8765432176 344-22-9876

UAD Buy EU

13:45:24 16-Feb-2015

3456789012 412-22-8765

NZMA Sell EU

Column-LevelControls

Row-LevelCon

trols

Date/time Accnt # SSN Asset Trade Country

09:33:11 16-Feb-2015

0234837823 238-23-9876

AZP Sell US

11:33:0116-Feb-2015

3947848494 329-44-9847

TBT Buy group2

14:12:3416-Feb-2015

4848367383 123-56-2345

IBM Sell group3

09:22:03 16-Feb-2015

3485739384 585-11-2345

ICBD Buy US

11:55:3316-Feb-2015

3847598390 234-11-8765

FWQ Buy US

10:22:55 16-Feb-2015

8765432176 344-22-9876

UA Buy group3

13:45:24 16-Feb-2015

3456789012 412-22-8765

AMZN Sell group2

Column-LevelControls

Row-LevelCon

trols

XXX-XX

XXX-XX

XXX-XX

WhatU.S.BrokersSee

Hive,Impala,MR,Spark,Pig

SingleHDFSfile:

Page 16: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

16©Cloudera,Inc.Allrightsreserved.

VisibilitySecurityRequirements

Complywithpoliciesforaudit,dataclassification,andlineage

Centralizetheauditrepository

VisibilityReportingonwheredatacamefromandhowit’sbeingused

InfoSecConcept:AuditingLineage

ClouderaNavigator

Page 17: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

17©Cloudera,Inc.Allrightsreserved.

Audit&LineageTrustedforproduction• 100sofcustomerdeploymentsofClouderaNavigatoroverlast3+years

Compliance-ready• OnlyHadoopdistributiontopassPCIaudit

Detailed• Columnandrowlevelaccesstrail

Playsnicelywithothers• Integratedwiththeleadingpartnersolutions

Page 18: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

18©Cloudera,Inc.Allrightsreserved.

DataSecurityRequirements

Performanalyticsonregulateddata

Encryptdata,conformtokeymanagementpolicies,protectfromroot

IntegratewithexistingHSMaspartofkeymanagementinfrastructure

DataProtectingdatainthe

clusterfromunauthorizedvisibility

InfoSecConcept:Encryption,Keymanagement,Datamasking

NavigatorEncrypt&KeyTrustee

Page 19: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

19©Cloudera,Inc.Allrightsreserved.

ComprehensiveDataSecurity

Manager Navigator

Impala Hive

HDFS HBase

Sentry

NavigatorKeyTrustee

LogFiles

MetadataStore

EncryptedData

EncryptionKey

Legend

IngestPaths,Temp/Spillfiles

HSM(optional)

ALLdataonthewire• ALLdataatrest:HDFS,HBase,metadatadatabases,tempfiles,ingestpaths

• Automatedkeyreplication&backup• HSMbackedkeyprotection

• Sensitivedatainlogs• Passwordsinconfig files

Encryption

KeyManagement

DataMasking

Page 20: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

20©Cloudera,Inc.Allrightsreserved.

Cloudera’scomprehensive,compliance-readysecuritysolution

AccessDefiningwhatusersandapplicationscan

dowithdata

TechnicalConcepts:PermissionsAuthorization

DataProtectingdatainthe

clusterfromunauthorizedvisibility

TechnicalConcepts:Encryption,Keymanagement,Datamasking

VisibilityReportingonwheredatacamefromandhowit’sbeingused

TechnicalConcepts:AuditingLineage

ClouderaManager ApacheSentry&RecordService

ClouderaNavigator NavigatorEncrypt&KeyTrustee

PerimeterGuardingaccesstothe

clusteritself

TechnicalConcepts:Authentication

Networkisolation

Page 21: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

21©Cloudera,Inc.Allrightsreserved.

Beyondtraditionalsecuritycontrols

Automateddiscoveryandtaggingofsensitivedata• Automaticallyscanforprotectedattributetypes• Automaticallyapplyauthorizationandencryptionpolicy

“Followthedata”authorizationandprotectionpolicies• Leveragelineagedatataggingenforceauthorizationandencryptionpolicy• Eliminatemanualconfigurationofsecurityforeachnewtableandcolumn

Adminsfocusedonexceptionhandlingduetoinsufficientaccess

Page 22: CTD Enabling Secure Hadoop Environments - …cdn.govexec.com/media/ctd_enabling_secure_hadoop_environments.pdfCTD Enabling Secure Hadoop Environments Author: Kate Tong Created Date:

©Cloudera,Inc.Allrightsreserved. 22

ThankyouThankYouFredKoopmans


The DOG Gateway Enabling Ontology-based Intelligent Domotic Environments

The DOG Gateway Enabling Ontology-based Intelligent Domotic Environments

Enabling Java in Latency Sensitive Environments · 2020-04-24 · Enabling Java in Latency Sensitive Environments Matt Schuetze, Product Manager, Azul Systems DotCMS Bootcamp, Nashville,

Enabling Java in Latency Sensitive Environments · 2020-04-24 · Enabling Java in Latency Sensitive Environments Matt Schuetze, Product Manager, Azul Systems DotCMS Bootcamp, Nashville,

Data streaming in Hadoop - diva-portal.org927113/FULLTEXT01.pdf · data streaming in hadoop a study of real time data pipeline integration between hadoop environments and external

Data streaming in Hadoop - diva-portal.org927113/FULLTEXT01.pdf · data streaming in hadoop a study of real time data pipeline integration between hadoop environments and external

Enabling Multi-party 3D Tele-immersive Environments with ...

Enabling Multi-party 3D Tele-immersive Environments with ...

Enabling the Internet of Things with Real-time Hadoop

Enabling the Internet of Things with Real-time Hadoop

Space Robotics: Enabling Autonomy in Challenging Environments ·  · 2014-10-22Robotics Enabling Autonomy in Challenging Environments Ioannis Rekleitis Computer Science and Engineering,

Space Robotics: Enabling Autonomy in Challenging Environments ·  · 2014-10-22Robotics Enabling Autonomy in Challenging Environments Ioannis Rekleitis Computer Science and Engineering,

03. Distributed Environments - Hadoop Nghia Duong-Trung ...03. Distributed Environments - Hadoop Nghia Duong-Trung, Mohsan Jameel Information Systems and Machine Learning Lab (ISMLL)

03. Distributed Environments - Hadoop Nghia Duong-Trung ...03. Distributed Environments - Hadoop Nghia Duong-Trung, Mohsan Jameel Information Systems and Machine Learning Lab (ISMLL)

FORGING NEED -BASED, DEMAND-DRIVEN PARTNERSHIPS National Enabling Environments … · 2016-12-23 · FORGING NEED -BASED, DEMAND-DRIVEN PARTNERSHIPS National Enabling Environments

FORGING NEED -BASED, DEMAND-DRIVEN PARTNERSHIPS National Enabling Environments … · 2016-12-23 · FORGING NEED -BASED, DEMAND-DRIVEN PARTNERSHIPS National Enabling Environments

Enabling environments for fostering agriculture ... Masaba.pdf · Enabling environments for fostering agriculture investments conducive ... PPP for oil palm ... Overview of the Oil

Enabling environments for fostering agriculture ... Masaba.pdf · Enabling environments for fostering agriculture investments conducive ... PPP for oil palm ... Overview of the Oil

Machine Learning for Enabling ActiveMeasurements in IoT Environments …1295327/... · 2019. 3. 11. · Machine Learning for Enabling Active Measurements in IoT Environments Wenqing

Machine Learning for Enabling ActiveMeasurements in IoT Environments …1295327/... · 2019. 3. 11. · Machine Learning for Enabling Active Measurements in IoT Environments Wenqing

Enabling a Large Analytic Userbase on Hadoop & Co

Enabling a Large Analytic Userbase on Hadoop & Co

Assessing Business Enabling Environments: How Gender ...€¦ · Assessing Business Enabling Environments: How Gender Changes the Equation ... It has become clear that macro-, meso-

Assessing Business Enabling Environments: How Gender ...€¦ · Assessing Business Enabling Environments: How Gender Changes the Equation ... It has become clear that macro-, meso-

Enabling Engagement: Building Relationships Across Channels In Complex Environments

Enabling Engagement: Building Relationships Across Channels In Complex Environments

Enabling Environments

Enabling Environments

ENABLING ENVIRONMENTS COLLECTIONS Me do it!

ENABLING ENVIRONMENTS COLLECTIONS Me do it!

ScaleOut hServerv2: Enabling Real-Time Analytics Using Hadoop Map/Reduce

ScaleOut hServerv2: Enabling Real-Time Analytics Using Hadoop Map/Reduce

NetSolve: Grid Enabling Scientiï¬c Computing Environments

NetSolve: Grid Enabling Scientiï¬c Computing Environments

Enabling environments for agribusiness and agro-industries

Enabling environments for agribusiness and agro-industries

Design Guidelines for inclusive, enabling environments for ... · Design Guidelines for inclusive, enabling environments for adults with complex needs ... to incorporate innovative

Design Guidelines for inclusive, enabling environments for ... · Design Guidelines for inclusive, enabling environments for adults with complex needs ... to incorporate innovative

Enabling R on Hadoop

Enabling R on Hadoop