Taming the Digital Transformation DragonBig Data Governance Practices

Jay ZaidiNovember 14, 2016

ABOUT ME

“Succeeding with data isn’t just a matter of putting Hadoop in your machine room, or hiring some physicists with crazy math skills. It requires you to develop a data culture that involves people throughout the organization.”- DJ Patil, Chief Data Scientist of the U.S.

● Founded AlyData in 2014. We specialise in Data Management and Data Science. Our mission is to transform organizations, by helping them innovate and gain a competitive advantage by unleashing value from their data and information assets.

● 13 Years at Fannie Mae. Last 5 years was a direct report to the CDO.

● Have authored 2 books and over 80 articles on data management on LinkedIn.

● Email - jayzaidi@alydata.com● LinkedIn Profile -

https://www.linkedin.com/in/javedzaidi

● Twitter - @jayzaidi

#data-driven leadership2

3 MAJOR THEMES I WISH TO HIGHLIGHT -

1. Most important areas leaders focus on - Value Creation and Risk Management. “Data” is foundational to that.

2. We are living in the Fourth Industrial Revolution or the “Age of Data”. Data practitioners must focus on “What Data Does (value) and How to Govern It (risk)” not “What Data Is”. Value is generated by Data Science by unleashing the power of data via better insights and Risk is managed via robust Data Governance.

3. Data Governance is at the heart of Risk Management and Data Management (Master Data Management, Data Quality, Metadata, Analytics, and Data Science). Build it into the Big Data Lake, Data Science, and Small Data Repositories.

#create-value & #manage-risk 3

#data-science 4

#data-governance5

A FRAMEWORK FOR UNDERSTANDING DATA GOVERNANCE

6

BusinessImperatives

RiskManagement

RegulatoryCompliance

Legal

Data Security &Privacy

DataControls

DataOptimization

WHY

WHAT

Data democratization, data decentralization, and tougher regulatory environment are driving data governance as a critical risk management component.

“Governance is about changing culture and implementing processes to ensure proper oversight of data semantics and quality, transparency into data related metrics, accountability for data, and timely resolution of data-related issues or questions”

#manage-risk7

8

BRIDGING THE DATA GOVERNANCE DIVIDE - SMALL AND BIG DATA

- Big data training about unstructured data and containers - metadata, quality, master data, logs etc.

- Process training on changes made to governance, quality, MDM, metadata changes due to big data

- Analysis training on issue logs, interpreting them and working with stakeholders to address data issues

- Training on process changes due to big data

- Training on new Training about unstructured data and containers

- Training on process changes due to big data

- Training on new tools used to tag, profile, review big data

- Implement new tools used to tag, profile, analyze, review big data

- Implement activity log analyzer, automate and working with stakeholders to address data issues

- Training about unstructured data and containers

Small Data - Structured data is highly organized information that uploads neatly into a relational database (schema on write), lives in fixed fields, and is easily detectable via search operations or algorithms. Structured data is relatively simple to enter, store, query, and analyze, but it must be strictly defined in terms of field name and type (e.g. alpha, numeric, date, currency).

Big Data - Unstructured data resides in files. It is increasingly available in the form of complex data sources, such as web logs, multimedia content, email, customer service interactions, sales automation, and social media data. The fundamental challenge of unstructured data sources is that they don’t follow a predefined schema, are difficult for nontechnical business users and data analysts alike to unbox, understand, and prepare for analytic use. Beyond issues of structure, is the sheer volume of this type of data. Need additional contextual data to understand, different tools to process/analyze, and more automation due to volume and variety.

#small-big-data

This architecture supports various data access patterns and governance.

ARCHITECTING A DATA LAKE WITH GOVERNANCE IN MIND - THE HOW?

#data-classification

#data-access-controls

#data-lineage

#data-semantics

#data-quality #issue-log

9

Diagram Courtesy - “Enterprise Big Data Lake” a O’Reilly book by Alex Gorelik

#master-data

#people #process #tech

10

SOME EXAMPLES OF DATA LAKE GOVERNANCEClient: Fortune 50 Financial Services

Problem: CISO and Security Operation Center was able to pinpoint sensitive data being accessed via security logs but had no idea where it resided and who was accountable for it.

Solution: The Data Governance program had already classified data by sensitivity, identified enterprise critical data, and had a list of data stewards and custodians accountable for each. This facilitated the root cause analysis exercise to get to address the questions that the security team had.

Client: Fortune 100 Healthcare Client

Problem: Reports to CFO and corporate disclosures required significant effort and impacted time-to-value due to the quality of data acquired from upstream systems.

Solution: The Enterprise Data Quality group collaborated with CFO’s group and the three key upstream system staff to define data quality requirements for key data, agreed on the quality dimensions, implemented a data certification process and provided guidance on usage of open source data quality tools to ensure consistent delivery of high quality data.

Client: Federal Government Civilian Agency

Problem: Internal Audit team wasn’t able to quickly pinpoint issues with underlying data and how it could impact risk to the organization.

Solution: The Enterprise Data Quality team had already implemented a data quality framework that standardized the data quality dimensions, data profiling process and tools, and a consistent way to report this information. They collaborated with the Internal Audit staff to train them on these capabilities, so that they could independently validate data and process.

Client: Fortune 500 Hospitality Industry

Problem: Sales and Marketing campaigns weren’t delivering the desired ROI and there was no way to determine the root cause.

Solution: The Enterprise Data Governance team engaged the Stewards and Custodians for Customer, Sales, and Rewards system organizations to discuss the consistency and quality of data based on their specific quality reports and subject matter expertise and concluded that the lack of Customer 360 data and inconsistencies in customer and sales data across three siloed systems was the root cause. EDG facilitated sessions between various teams to address these issues. #data-lake-

governance

6-STEP GOVERNANCE PROCESS

1. Assess: Data Management Maturity Assessment focused on Governance, Quality and Master Data Management.

2. Process: Apply our proprietary Scope, Process Automation, Ownership, Cross-

functional Engagement, and Human Intelligence (SPOCH) framework.3. Agility: Use Agile Data Governance (DG), Data Quality (DQ) and Master Data

Management (MDM) processes.4. Standards/Policies: Define and monitor compliance.5. Alignment: Align Data Governance with IT governance.6. Automation: Implement tools to automate DG processes, DQ profiling, Tagging,

Discovery, etc.#data-governance11

APPENDIX

12

1. Data Governance Demystified – Lessons From The Trenches

2. Bridging the Data Governance Chasm

3. You Think You Know Data? Think Again

4. 6 Reasons Why Big Data Investments Aren’t Paying Off For Some Organizations

5. 5 Reasons More Companies Don’t Have Data Quality Programs

6. What’s An Information Supply Chain and Why You Should Care?7. The Dark Side of Big Data

8. Re-Thinking Information Security and Data Governance

SOME ARTICLES I’VE AUTHORED(link to over 80 articles)

#thought-leadership 13