Data Architecture for Data Governance

Data Governance

IASA is

a non-profit professional association

run by architects

for all IT architects

centrally governed and locally run

technology and vendor agnostic

The

use,

disclosur

e, reproduction

, modification

, transfer, or

transmittal of thi

s work

without

the written permissio

n of IA

SA is

strictly prohibited

. © IA

SA 2009

Information Architecture

Module 0: Course Intro, Architecture Fundamentals Introduction: Data, Information and Knowledge

Module 1: Information for Business Lesson 1 – Information as Strategy Lesson 2 – Relating Information to Value Lesson 3 – Information Scope and

Governance

Module 2: Information Usage Lesson 1 – Who Uses Your Information Lesson 2 – How, When, Where and Why

is Information Used Lesson 3 – Form Factors Lesson 4 – Usage Design 1 Lesson 5 – Quality Attributes for

Information Architecture Lesson 6 – Data Tools and Frameworks

Module 3: Data Integration Lesson 1: Integrating at the Company Level Lesson 2: Data Characteristics Lesson 3: Data Integration at the System level

Module 4: Data Quality and Governance Lesson 1 – Data and Information Quality Lesson 2: Data Compliance Lesson 3 – Data and Information Governance

Module 5: Advanced Information Management Lesson 1: Data Warehousing Lesson 2: Business Intelligence Lesson 3: Data Security and Privacy Lesson 4: Metadata and Taxonomy Management Lesson 5: Knowledge Management

Module 6: Architecture throughout the Lifecycle

Typical State of Affairs

The Problem

What is data governance?

Governance is Not a Popular Topic

Governance is Overcomplicated

It is not about politics

It is not about delivering verdicts

It is not about defining solutions

Step 1: Form a Team

Step 2: Establish Lineage

Step 3: Assess Maturity

Step 4: Create Governance Body

Step 5: Define Governance

Step 6: Define Charter

•Charter

Step 7: Define Scope

Step 8: Create a Plan

Career Path

Iasa Architect Services

Adopt Iasa Standards – Skills taxonomy, role descriptions, compensation models

Set your goals for value

Assess your current team – gap analysis

Implement processes

Develop and educate people

Certify employees and vendors

Build communities

•December 6th – 7th, 2012 // Austin, TX

•Training: Dec. 3rd – 5th, 2012•Certification: Dec. 7th – 9th, 2012

•

•www.iasaworldsummit.org•Leading Innovation in Architecture. Stay ahead of the technology curve and shape the future of architecture in your organization.

•Connect and share insights with the largest global network of technology and enterprise architect practitioners.

•Attend sessions on the most current breakthroughs, case studies andkey topics in architecture - presented by a mix of practicing architects

from top performing businesses and organizations.

•Participate in pre-conference workshops and training. Maximize your time

at the summit by taking part in one of six intensive training courses designed

to provide immediate solutions to benefit your everyday practice.

•Featuring Industry Leading Keynote Speakers:

•Sheila JeffreySenior Information Architect

Bank of America

•David Del GiudicePrincipal Architect

AstraZeneca

•Paul PreissPresident, Founder

Iasa Global

•Cat SuschEnterprise Architect

Microsoft

•Scott WhitmireEnterprise Architect

Nordstrom

•David ManningIT Enterprise ArchitectIdaho Power Company

End Module

Data Architecture for Data Governance

Presented by Malcolm Chisholm [email protected]

Telephone 732-687-9283www.refdataportal.com

www.bizrulesengine.com

© AskGet.com Inc., 2012. All rights reserved

November 8, 2012

• Introducing Enterprise Architecture (EA)

• Introducing Data Governance

• Data Architecture and Data Governance

• Data Architecture Patterns

• Data Architecture, Governance, and The Business

Agenda


Introducing Enterprise Architecture (EA)


Need for Enterprise Architecture (EA)

Myriad of Independent Projects in IT

Here is where we want to be in 5

years…

CxO’s

Strategy is Set at High Level

Operationalization

Enterprise Architecture

…Others…

• EA is responsible for ensuring overall business strategy is implemented by IT


The IT Mindset


IT CxO

That’s not a clear requirement…

Tell me what you want me to buildThen I will design itThen I will build itThen I will turn it over to youThen I will walk away

We have a high-level strategy…

• IT people just want to build stuff and hand it off to other people•They want business sponsors who will pay for this stuff and tell them exactly what is needed

•Strategy does not match this mindset

Design without Architecture


• The Winchester Mystery House• Lots of design – but no architecture• There is a difference between architecture and design

Rooms: 160; Doorways: 467; Doors: 950; Fireplaces: 47 (gas, wood, coal); Bedrooms: 40

Constructed 1884 – 1922 (38 continuous years); Cost: $5.5M

Blueprints: Never made; Individual rooms sketched out by Sarah Winchester on paper or other media (e.g. tablecloths)

All Design – No Architecture

“Circular” Stairway

Stairway to Ceiling

“…staggering amount of creativity, energy, and expense poured into each and every detail”

http://www.winchestermysteryhouse.com

Design vs. Architecture

• Architecture deals with many instances of a component type that must interact

• Design deals with one instance of a component type, without regard to interaction

• E.g. Perspective of Databases : Data Environment (BI or Integration Environment in this example)

DDL

Source Mirror

IntegrationChanged

Data Capture

Data Flow


Basic EA Taxonomy (“BAIT”)

Enterprise Architecture


Business Architecture

Application Architecture

Information Architecture

Technology Architecture

• EA is clearly not a monolithic discipline• It looks more like a collection of dissimilar components • Each component is a competency – so this matters• Yet there is no common agreement

Introducing Data Governance


What is Data Governance? Starter Definitions


”The exercise of decision-making and authority for data-related matters.”

”A system of decision rights and accountabilities for information-related processes, executed according to agreed-upon models which describe who can take what actions with what information, and when, under what circumstances, using what methods.”

www.datagovernance.com

From the Data Governance Institute

Data Governance is about Processes


Deciding who has what rights regarding data

In other Words…

Processes for…

Making decisions about data

Implementing decisions about data

Identifying the processes needed to manage data

Identifying the actors in these processes

Designing these processes

And designing the processes to design these processes

W R ON G

Data Governance – A Caution


The Business has…. IT has…

Business Processes Governance

Data Governance is the design, building, and operation of formal business processes to manage the enterprise data resource.

• The decision rights of actors are defined within the context of these business processes • There is no real distinction between the processes of “data governance” and the “business processes” of the business . [Compare HR].• A distinction that identifies the set of business processes that manage the enterprise data resource as “data governance” is valid. [Compare HR].• Use of the term data governance to further the pretence that IT is universe totally outside of the enterprise is invalid.• Therefore, the processes of data governance are truly business processes

Data Architecture and Data Governance


Architectural Levels

Reference Architecture

Conceptual Architecture

Logical Architecture

Physical Architecture

Assimilate best architectural

practices from the IT industry. All

relevant component classes and patterns are understood.

Match/select architectural

component classes and patterns to the

business needs, (including future

plans). Influencing actual physical

implementations to ensure

conformance with logical architecture

Detailed design of future state

architecture.

A reference architecture is the highest level of architecture. It can be undertaken prior to even thinking about the enterprise. Its advantages are that it can be very high level and can cover the entire enterprise. Useful for thinking about high level data layers


Reference Data Architecture - Example


• Example of a simple reference data architecture• Just data layers – no services or componentsBUT – just having a picture can be problematic

Need Definitions, Explanations – Not Just Picture

LayerCharacteristic Transactional

Application Data Warehouse Data Mart

Data produced via the automation of business processes

View of data across the enterprise. Supports dissemination, derivation of knowledge and history

Purpose

Data Life Cycle

Data Operations

Data Model

Data structured and filtered to support specific information needs of small groups of users.

All base (non-derived) data originates here

Derivations (including aggregations) produced here, and history is inferred

Data from Warehouse is transformed to support specific reporting

Create / Source / Read / Update / Delete / Archive

Extract / Transform / Load / Derive / Publish / Archive

Subscribe / Transform / Archive

Normalized to 3NFSubject Oriented / Snowflaked / Conformed Dimensions

Information Requirement Oriented / Snowflaked / Conformed Dimensions


•Much more is needed than the above•Definitions are a technical reference; explanations help stakeholders to understand the reference architecture

Subject Area Model

© Askget.com Inc., 2011. All rights reserved

• A Subject Area Model is a taxonomy of the major areas in the enterprise that are relevant to data management• A necessary starting point for MDM• 1 – A subject area should have stable definitions of entities• 2 – Production of master data should not cross subject area boundaries• Identify the major data concepts in each subject area – may will be master data entities.

A Subject Area Model is Not Costly


• 10-15 subject areas per model is the norm• Do not need elaborate data modeling tools• Much more about a standardized view of the enterprise• Can be done with 1-3 people• Gets a lot of bang for the buck• Is high-level and important

Data Management

Example of Use: Establish Vendor Data Profiles


• A Subject Area Model provides the basis for a common understanding of the data vendors provide

• Will need to go to major entity clusters within Subject Area

Data Management

Data Vendor 1

Data Vendor 2

Data Vendor 3

Data Vendor 4

Data Vendor 5

Subject Area Model(s)

Create & Manage Subject Area Model(s)

Users

Map Vendor’s Data to Subject

Areas

Map Subject Areas to User Requirements

Map Vendors to Uses

A Subject Area Model is a Taxonomy

CLIENT MARKETINGAND MANAGEMENT

CLIENT

CONTACT

MEETING

SERVICE REQUEST

PLANCONTRIBUTION /

WITHDRAWAL

ADDRESS


• People think a taxonomy is a way of unlocking the secrets of the universe• Taxonomies are a tools – you can have as many as you need to do your job • You can never expect the boundaries of a subject area model to be exactly precise and never changing

Your model does not tell me what is

financial data!

I don’t like your definition of this subject

area!

You mean to tell me every one of these

subject areas includes access security for data!

Create a Metadata Subject Area Model


The Fundamental Ontology of Metadata for the Enterprise

Metadata Subject Area Model


The Subject Area Model is itself Metadata and needs to be managed

Advantages

1. Standardizes the metadata in the enterprise at a high level.

2. Can be used to prioritize projects and programmes.

3. Useful in communications.

4. Shows what metadata is being produced by subject area. This is a high-level inventory.

Disadvantages

1. It is a taxonomy and can be argued over. No one taxonomy will satisfy every perspective. We use data categorization for that (see later). The Subject Area Model should be the most common and natural perspective.

2. The Subject Area Model often has more things expected of it than it can deliver. Managing expectations can be tricky.

5. Can distinguish data-related metadata from other data

6. Within each subject area, definitions should be constant. There may be variations across subject areas.

7. Subject areas are candidates for conceptual models.

Data Architecture Patterns


What is an Architectural Pattern?


• Patterns occur a lot in architecture• There are also styles, metaphors and antipatterns• Seem to have emerged from the application architecture

competency (e.g. rare to hear of dimensional modeling as a pattern or style)

“Pattern in architecture is the idea of capturing architectural design ideas as archetypal and reusable descriptions”- Christopher Alexander, quoted in Business Model Generation by Osterwalder and Pigneur (2010)

• An architectural style is a central, organizing concept for a system.

• An architectural pattern describes a coarse-grained solution at the level of subsystems or modules and their relationships.

• A system metaphor is more conceptual and it relates more to a real-world concept over a software engineering concept.

- Quoted from A Practical Guide to Enterprise Architecture by McGovern, Ambler et al, at

http://www.infoq.com/news/2009/02/Architectural-Styles-Patterns

Antipattern: A pattern that is commonly used, but which is ineffective or damaging. E.g. Analysis Paralysis. - Term created by Andrew Koenig, 1995

Example: Farm and Market Pattern


MDM HUB(“MARKET”)

Publish

App 1

App 2

App 3

App ...

TRANSACTION APPLICATIONS

App A App B App ...

OTHER CONSUMING APPLICATIONS

Other Functionality

Master Data Entity 1 App



Master Data Entity N App

MASTER DATA PRODUCTION (“FARMS”)

Reduced Cleansing

Reduced Integration

MDM HUB

Cleanse

Integrate

Publish

App 1

App 2

App 3

App ...

TRANSACTION APPLICATIONS

App A App B App ...

OTHER CONSUMING APPLICATIONS

Produce Master Data

Other Functionality

Steward

Traditional MDM Integration Hub

Farm and Market Pattern for MDM

Six Patterns for Data Hubs: 1 – Publish/Subscribe


• Publisher pushes data to hub• Subscriber pulls data from hub• No data integration• Publisher may not know who the subscribers are• Weak governance - may not even be SLA’s

Six Patterns for Data Hubs: 2 – ODS for Integrated Reporting


• Supports principle that transaction applications will not do operational reporting (adverse affect on performance)

• Some form of integration• Anti-pattern of real-time data warehousing• History (changed data capture) not considered

Six Patterns for Data Hubs: 3 – ODS for Data Warehouses


• No reporting in this kind of ODS• Probably evolved from ODS for reporting• Strong integration• Just to feed warehouses • At odds with Data Warehouse that has own staging, integration

Six Patterns for Data Hubs: 4 – Traditional MDM Hub


• Master Data only• Integration does happen (typically just Trust & Survivorship)• Some content management needed – typically to correct data

quality problems• Typically DQ functionality an afterthought• Distributes “Golden Copy” Master Data to enterprise

Six Patterns for Data Hubs: 5 – Message Hub


• Messages – real-time or near-real-time - not batch data• Message switch • Command and control for message orchestration• Not just “listening” – does routing• Messages also stored in database

Six Patterns for Data Hubs: 6 – Integration Hub


• Like message hub, but for batch data• Event data equivalent of MDM hub• Just one place where integration done• Both warehouse and transaction applications need integrated data• Removes need for each application to do integration by itself

Data Architecture, Governance, and The Business


Another View of EA

• Long ago, people had to know the business rules concerning what they did

• These are now in applications, and staff are more concerned with knowing how to make the applications work

• Data is an asset, but staff tend to know little about the data• Loss of business knowledge

Many Years Ago……”the business” knew the business

Today……it does so much less


Who Knows The Business?

• The business seems to be more fragmented than long ago (though that is difficult to prove)

• Managers who know the business know less of it and are so time-constrained they cannot help much on aligning projects to enterprise strategy


Populate EA with Business Experts

• Business experts provide better relations with rest of business• Keep architects with just being oriented to architecture • Immediate business credibility for EA• They get poached


A Different EA Vision:Architects + Business Experts

IT Tendency:Populate EA with Architects

Questions and Answers


Presented by Malcolm Chisholm Ph.D.Telephone 732-687-9283

[email protected]

Data Architecture for Data Governance

November 8, 2012

Data Architecture for Data Governance

Technology

Transcript of Data Architecture for Data Governance