Data-Ed Online: Let's Talk Metadata: Strategies and Successes
-
date post
14-Sep-2014 -
Category
Technology
-
view
2.190 -
download
3
description
Transcript of Data-Ed Online: Let's Talk Metadata: Strategies and Successes
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Welcome!
Date: September 11, 2012
Time: 2:00 PM ET
Presented by: Dr. Peter Aiken
18/14/2012
Let’s Talk Metadata:
Strategies and Successes
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Commonly Asked Questions
1) Will I get copies of the slides after the event?
YES*
2) Is this being recorded so I can view it afterwards?
YES*
28/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Get Social With Us!
Live Twitter FeedJoin the conversation!
Follow us:
@datablueprint
@paiken
Ask questions and submit
your comments: #dataed
38/14/2012
Like Us on Facebookwww.facebook.com/datablueprint
Post questions and
comments
Find industry news,
insightful content
and event updates.
Join the GroupData Management &
Business Intelligence
Ask questions, gain
insights and collaborate
with fellow data
management
professionals
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Meet Your Presenter: Dr. Peter Aiken
4
• Internationally recognized thought-leader in the data management field with more than 30 years of experience
• Recipient of the 2010 International Stevens Award
• Founding Director of Data Blueprint (http://datablueprint.com)
• Associate Professor of Information Systems at Virginia Commonwealth University (http://vcu.edu)
• President of DAMA International (http://dama.org)
• DoD Computer Scientist, Reverse Engineering Program Manager/Office of the Chief Information Officer
• Visiting Scientist, Software Engineering Institute/Carnegie Mellon University
• 7 books and dozens of articles• Experienced w/ 500+ data management practices in 20 countries
#dataed
8/14/2012
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060 EDUCATION
Let’s Talk Metadata:
Strategies and Successes
Let’s Talk Metadata: Strategies and Successes
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Abstract: Metadata Practices
604/07/23
This presentation describes how data management can be enhanced using meta-processing. Commonly described as metadata management, properly implemented metadata practices incorporate data structures into more abstract processing. By using data about the data to enhance its value, its understandability, its ease of use, and many other options – organizations have developed sophisticated ways to enhance their data management and especially their data quality engineering efforts.
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview
2. What is metadata and why is it important?
3. Types of metadata
4. Metadata for unstructured data
5. Strategy and implementation
6. Guiding Principles
7. Take Aways, References and Q&A
7
Tweeting now: #dataed
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
The DAMA Guide to the Data Management Body of Knowledge
804/07/23
Data Management Functions
Published by DAMA International•The professional association for Data Managers (40 chapters worldwide)
DMBoK organized around •Primary data management functions focused around data delivery to the organization
•Organized around several environmental elements
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
The DAMA Guide to the Data Management Body of Knowledge
904/07/23
Environmental Elements
Amazon:
http://www.amazon.com/DAMA-Guide-Management-Knowledge-DAMA-DMBOK/dp/0977140083
Or enter the terms "dama dm bok" at the Amazon search engine
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Data Management
1004/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Data Management
1104/07/23
Manage data coherently.
Manage data coherently.
Share data across boundaries.
Share data across boundaries.
Assign responsibilities for data.
Assign responsibilities for data. Engineer data delivery
systems.
Engineer data delivery systems.
Maintain data availability.
Maintain data availability.
Data Program Coordination
Organizational Data Integration
Data Stewardship
Data Development
Data Support Operations
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Data Management
1204/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Management
1304/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview
2. What is metadata and why is it important?
3. Types of metadata
4. Metadata for unstructured data
5. Strategy and implementation
6. Guiding Principles
7. Take Aways, References and Q&A
14
Tweeting now: #dataed
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata or metadata
• In the history of language, whenever two words are pasted together to form a combined concept initially, a hyphen links them.
• With the passage of time, the hyphen is lost. The argument can be made that that time has passed.
• There is a copyright on the term "metadata," but it has not been enforced.
• So, term is "metadata"
1504/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
DefinitionsMetadata is …
•… everywhere in every data management activity and integral to all IT systems and applications.
•… to data what data is to real life. Data reflects real life transactions, events, objects, relationships, etc. Metadata reflects data transactions, events, objects, relations, etc.
•… the data that describe the structure and workings of an organization’s use of information, and which describe the systems it uses to manage that information. [quote from David Hay's new book, page 4]
•Data describing various facets of a data asset, for the purpose of improving its usability throughout its life cycle [Gartner 2010]
•Metadata unlocks the value of data, and therefore requires management attention [Gartner 2010]
Metadata Management is …
•… the set of processes that ensure proper creation, storage, integration, and control to support associated use of metadata
1604/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Analogy: Card catalog in a library • Card catalog identifies what books
are stored in the library and where they are located in the building
• Users can search for books by subject area, author, or title
• Catalog shows author, subject tags, publication date and revision history of each book
• Card catalog information helps determine which books will meet the reader’s needs
• Without this catalog resource, finding books in the library would be difficult, time consuming and frustrating
• Readers may search many incorrect books before finding the right book if a catalog does not exist
1704/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Definition, cont’d• Metadata is the card catalog in a
managed data environment
• Abstractly, Metadata is the descriptive tags or context on the data (the content) in a managed data environment
• Metadata shows business and technical users where to find information in data repositories
• Metadata provides details on where the data came from, how it got there, any transformations, and its level of quality
• Metadata provides assistance with what the data really means and how to interpret it
1804/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Defining Metadata
Metadata is any combination of any circle and the data in the center of the spark!
1904/07/23
Adapted from Brad Melton
Data
WhereWhy
What How
Who
When
Data
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Who: Author
What: Title
Where: Shelf Location
When: Publication Date
Manage a large amount of data (the Library) with a small amount of metadata (Card Catalog)
2004/07/23
Data
WhereWhy
What How
Who
When
DataData
DataLibrary Book
Library Metadata ExampleLibraries can operate efficiently through careful use of metadata (Card Catalog)
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
2104/07/23
Data
WhereWhy
How
Who
When
Messages
What
Outlook Example
"Outlook" metadata is used to navigate and manage email
Imagine how managing e-mail (already non-trivial) would change if Outlook did not make use of metadata
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Who: "To" & "From"
What: "Subject"
How: "Priority"
Where: "USERID/Inbox", "USERID/Personal Folders"
Why: "Body"
When: "Sent" & "Received”
•Find the important stuff/weed out junk •Organize for future access/outlook rules
2204/07/23
Outlook Example, cont’d
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata practices connect data sources and uses in an organized and efficient manner
• What is the structure of metadata practices?– Storage: repository, glossary, models, lineage - currently multiple technologies are used– Engineering: identifying/harvesting/normalizing/administer evolving metadata structures– Delivery: supply/access/portal/definition/lookup search identify/ensure required
metadata supplies to meet business needs– Governance: ensure proper/creation/storage/integration/control to support effective use
• When executed, engineering and delivery implement governance
2304/07/23
Sources
Metadata Governance
Metadata Engineering
Metadata Delivery Uses
Metadata Practices
MetadataStorage
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Organized Knowledge 'Data'
Improved Quality Data
Data Organization Practices
Metadata Practices will be inextricably intertwined with Data Quality and Master Data and Knowledge Management, (among other EIM Functions)
Operational Data
Data Quality Engineering
Master Data Management
Practices
Suspected/Identified
Data Quality
Problems
Routine Data Scans
Master Data Catalogs
Routine Data Scans
KnowledgeManagement
Practices
ExtractionSources
Data that might benefit from Master Management
2404/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History 1990-2008The history of Metadata management tools and products seems to be a metaphor for the lack of a methodological approach to enterprise information management:
• Lack of standards and proprietary nature of most managed Metadata solutions cause many organizations to avoid focusing on metadata
• This limits organizations’ ability to develop a true enterprise information management environment
• Increased attention given to information and its importance to an organization’s operations and decision-making will drive Metadata management products and solutions to become more standardized
• More recognition to the need for a methodological approach to managing information and metadata
2504/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: The 1990s
• Business managers began to recognize the value of Metadata repositories
• Newer tools expanded the scope• Potential benefits identified during this period
include:– Providing semantic layer between company’s system
and business users– Reducing training costs– Making strategic information more valuable as aid in
decision making– Creating actionable information– Limiting incorrect decisions
2604/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: Mid-to late 1990s• Metadata becomes more relevant to corporations who were
struggling to understand their information resources caused by: – Y2K deadline
– Emerging data warehousing initiatives
– Growing focus around the World Wide Web
• Beginning of efforts to try to standardize Metadata definition and exchange between applications in the enterprise
• Examples of standardization:– 1995: CASE Definition Interchange Facility (CDIF)
– 1995: Dublin Core Metadata Elements
– 1994 – 1999: First parts of ISO 11179 standard for Specification and Standardization of Data Elements were published
– 1998: Common Warehouse Metadata Model (CWM)
– 1995: Metadata Coalitions’ (MDC) Open Information Model
– 2000: Both standards merged into CSM. Many Metadata repositories began promising adoption of CWM standard
2704/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: 21st Century• Update of existing Metadata repositories for deployment on the web
• Introduction of products to support CWM
• Vendors begin focusing on Metadata as an additional product offering
• Few organizations purchase or develop Metadata repositories
• Effective enterprise-wide Managed Metadata Environments are rare due to:
– Scarcity of people with real world skills
– Difficulty of the effort
– Less than stellar success of some of the initial efforts at some companies
– Stagnation of the tool market after the initial burst of interest in late 90s
– Still less than universal understanding of the business benefits
– Too heavy emphasis on legacy applications and technical metadata
2804/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Polling Question #1
What have been the driving factors in focusing on metadata within the last decade?
a. Recent entry of smaller vendors into the market
b. Challenges related to addressing regulatory requirements
c. Declination to the existing Metadata standards
297/10/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: Current Decade• Focus on need for and importance of metadata
• Focus on how to incorporate Metadata beyond traditional structured sources and include unstructured sources
• Driving factors:– Recent entry of larger vendors into the market
– Challenges related to addressing regulatory requirements, e.g. Sarbanes-Oxley, and privacy requirements with unsophisticated tools
– Emergence of enterprise-wide initiatives, e.g. information governance, compliance, enterprise architecture, automated software reuse
– Improvements to the existing Metadata standards, e.g. RFP release of new OMG standard Information Management Metamodel (IMM), which will replace CWM
– Recognition at the highest levels that information is an asset that must be actively and effectively managed
3004/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview
2. What is metadata and why is it important?
3. Types of metadata
4. Metadata for unstructured data
5. Strategy and implementation
6. Guiding Principles
7. Take Aways, References and Q&A
31
Tweeting now: #dataed
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Process Metadata• Process Metadata is...
– Data that defines and describes the characteristics of other system elements, e.g. processes, business rules, programs, jobs, tools, etc.
• Examples of Process metadata:– Data stores and data involved– Government/regulatory bodies– Organization owners and stakeholders– Process dependencies and decomposition– Process feedback loop and documentation– Process name
3204/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Business Process Metadata
Who: Created the documentation?
What: Are the important dependencies among the processes?
How: Do the business processes interact with each other?
3304/07/23
Data
WhereWhy
What How
Who
When
Data
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Business Metadata
• Business Metadata describe to the end user what data are available, what they mean and how to retrieve them.
• Included are:– Business names and definitions of subject and
concept areas, entities, attributes– Attribute data types and other attribute properties– Range descriptions, calculations, algorithms and
business rules– Valid domain values and their definitions
3404/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Technical & Operational Metadata
3504/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Technical and operational metadata provides developers and technical users with information about their systems
• Technical metadata includes…– Physical database table and column names, column properties, other
properties, other database object properties and database storage
• Operational metadata is targeted at IT operations users’ needs, including…– Information about data movement, source and target systems, batch
programs, job frequency, schedule anomalies, recovery and backup information, archive rules and usage
• Examples of Technical & Operational metadata:– Audit controls and balancing information
– Data archiving and retention rules
– Encoding/reference table conversions
– History of extracts and results
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Data Stewardship
3604/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Data stewardship Metadata is about...– Data stewards, stewardship processes, and responsibility
assignments
• Data stewards…– Assure that data and Metadata are accurate, with high quality
across the enterprise. – Establish and monitor data sharing.
• Examples of Data stewardship metadata:– Business drivers/goals– Data CRUD rules– Data definitions – business and technical– Data owners– Data sharing rules and agreements/contracts– Data stewards, roles and responsibilities
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Provenance
3704/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Provenance: – the history of ownership of a valued object or
work of art or literature" [Merriam Webster]– For each datum, this is the description of:
• Its source (system or person or department),
• Any derivation used, and
• The date it was created.
– Examples of Data Provenance:• The programs or processes by which it was created
• Its owner
• The steward responsible for its quality
• Other roles and responsibilities
• Rules for sharing it.
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview
2. What is metadata and why is it important?
3. Types of metadata
4. Metadata for unstructured data
5. Strategy and implementation
6. Guiding Principles
7. Take Aways, References and Q&A
38
Tweeting now: #dataed
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Subject Areas
3904/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
Subject Areas Components
1) Business Analytics Data definitions, reports, users, usage, performance
2) Business Architecture Roles and organizations, goals and objectives
3) Business DefinitionsBusiness terms and explanations for a particular concept, fact, or other item found in an organization
4) Business Rules Standard calculations and derivation methods
5) Data GovernancePolicies, standards, procedures, programs, roles, organizations, stewardship assignments
6) Data IntegrationSources, targets, transformations, lineage, ETL workflows, EAI, EII, migration/conversion
7) Data Quality Defects, metrics, ratings
8) Document Content Management
Unstructured data, documents, taxonomies, ontologies, name sets, legal discovery, search engine indexes
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Subject Areas, cont’d
4004/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
Subject Areas Components
9) Information Technology Infrastructure
Platforms, networks, configurations, licenses
10) Conceptual data modelsEntities, attributes, relationships and rules, business names and definitions.
11) Logical Data ModelsFiles, tables, columns, views, business definitions, indexes, usage, performance, change management
12) Process ModelsFunctions, activities, roles, inputs/outputs, workflow, timing, stores
13) Systems Portfolio and IT Governance
Databases, applications, projects, and programs, integration roadmap, change management
14) Service-oriented Architecture (SOA) information:
Components, services, messages, master data
15) System Design and Development
Requirements, designs and test plans, impact
16) Systems ManagementData security, licenses, configuration, reliability, service levels
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Benefits of Metadata
4104/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
1) Increase the value of strategic information (e.g. data warehousing, CRM, SCM, etc.) by providing context for the data, thus aiding analysts in making more effective decisions.
2) Reduce training costs and lower the impact of staff turnover through thorough documentation of data context, history, and origin.
3) Reduce data-oriented research time by assisting business analysts in finding the information they need in a timely manner.
4) Improve communication by bridging the gap between business users and IT professionals, leveraging work done by other teams and increasing confidence in IT system data.
5) Increased speed of system development’s time-to-market by reducing system development life-cycle time.
6) Reduce risk of project failure through better impact analysis at various levels during change management.
7) Identify and reduce redundant data and processes, thereby reducing rework and use of redundant, out-of-data, or incorrect data.
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata for Unstructured Data
4204/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
• Unstructured data = any data that is not in a database or data file,
including documents or other media data
• Metadata describes both structured and unstructured data• Metadata for unstructured data exists in many formats, responding
to a variety of different requirements• Examples of Metadata repositories describing unstructured data:
– Content management applications– University websites– Company intranet sites– Data archives– Electronic journals collections– Community resource lists
• Common method for classifying Metadata in unstructured sources is to describe them as descriptive metadata, structural metadata, or administrative metadata
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata for Unstructured Data: Examples
Examples of descriptive
metadata:• Catalog information• Thesauri keyword terms
Examples of structural
metadata• Dublin Core• Field structures• Format (audio/visual, booklet)• Thesauri keyword labels• XML schemas
4304/07/23
Examples of
administrative metadata• Source(s)• Integration/update schedule• Access rights• Page relationships (e.g. site
navigational design)
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Sources of MetadataPrimary Sources:
• Virtually anything named in an organization
Secondary sources:
• Other Metadata repositories, accessed using bridge software
• CASE tools, ETL tools
Many data management tools create and use repositories for their own use.
4404/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Specific ExampleFour metadata sources:
1.Existing reference models (i.e., ADRM)
2.Conceptual model created two years ago
3.Existing systems (to be reverse engineered)
4.Enterprise data model
4504/07/23
ADRM
}
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview
2. What is metadata and why is it important?
3. Types of metadata
4. Metadata for unstructured data
5. Strategy and implementation
6. Guiding Principles
7. Take Aways, References and Q&A
46
Tweeting now: #dataed
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Strategy • Metadata Strategy is…
– … a statement of direction in Metadata management by the enterprise
– … a statement of intend that acts as a reference framework for the development teams
– …driven by business objectives and prioritized by the business value they bring to the organization
• Build a Metadata strategy from a set of defined components
• Primary focus of Metadata strategy: gain an understanding of and consensus on the organization’s key business drivers, issues, and information requirements for the enterprise Metadata program
• Need to understand how well the current environment meets these requirements now and in the future
• Metadata strategy objectives define the organization’s future enterprise Metadata architecture and recommend logical progression of phased implementation steps
4704/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
4804/07/23
Metadata Strategy Implementation Phases
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Management
4904/07/23
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Goals and Principles
5004/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
1. Provide organizational understanding of terms and usage
2. Integrate Metadata from diverse sources
3. Provide easy, integrated access to metadata
4. Ensure Metadata quality and security
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Activities1) Understand Metadata requirements
2) Define the Metadata architecture
3) Develop and maintain Metadata standards
4) Implement a managed Metadata environment
5) Create and maintain metadata
6) Integrate metadata
7) Management Metadata repositories
8) Distribute and deliver metadata
9) Query, report and analyze metadata
5104/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Activities: Metadata Standards Types
5204/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Two major types exist:
1) Industry or consensus standards
2) International standards
• High level framework shows how standards are related and how they rely on each other for context and usage:
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Common Warehouse Metadata (CWM):• Specifies the interchange of Metadata among data warehousing, BI, KM,
and portal technologies.
• Based on UML and depends on it to represent object-oriented data constructs.
The CWM Metamodel
5304/07/23
Activities: Noteworthy Metadata Standards Types
Warehouse Process Warehouse Operation
Transformation OLAP Data Mining
Information Visualization
Business Nomenclature
Object Model Relational Record Multidimensional XML
Business Information Data Types Expression
Keys and Indexes
Type Mapping
Software Deployment
Object Model
Management
Analysis
Resource
Foundation
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Information Management Metamodel (IMM)
• Object Management Group Project to replace CWM• Concerned with:
– Business Modeling• Entity/relationship metamodel
– Technology modeling• Relational Databases• XML• LDAP
– Model Management• Traceability
– Compatibility with related models• Semantics of business vocabulary and business rules• Ontology Definition Metamodel
5404/07/23
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
The Information Management Metamodel...
5504/07/23
• Based on Core model.• Used to translate from one model to another.
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Primary Deliverables• Metadata repositories
• Quality metadata
• Metadata analysis
• Data lineage
• Change impact analysis
• Metadata control procedures
• Metadata models and architecture
• Metadata management operational analysis
5604/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Roles and Responsibilities
5704/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
Suppliers:– Data Stewards– Data Architects– Data Modelers– Database Administrators– Other Data Professionals– Data Brokers– Government and Industry
Regulators
Participants:– Metadata Specialists– Data Integration Architects– Data Stewards– Data Architects and Modelers– Database Administrators– Other DM Professionals– Other IT Professionals– DM Executives– Business Users
Consumers:• Data Stewards• Data Professionals• Other IT Professionals• Knowledge Workers• Managers and Executives• Customers and Collaborators• Business Users
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Technology• Metadata repositories• Data modeling tools• Database management systems• Data integration tools• Business intelligence tools• System management tools• Object modeling tools• Process modeling tools• Report generating tools• Data quality tools• Data development and administration tools• Reference and mater data management tools
5804/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview
2. What is metadata and why is it important?
3. Types of metadata
4. Metadata for unstructured data
5. Strategy and implementation
6. Guiding Principles
7. Take Aways, References and Q&A
59
Tweeting now: #dataed
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Guiding Principles1) Establish and maintain a Metadata strategy and
appropriate policies, especially clear goals and objectives for Metadata management and usage
2) Secure sustained commitment, funding, and vocal support from senior management concerning Metadata management for the enterprise
3) Take an enterprise perspective to ensure future extensibility, but implement through iterative and incremental delivery
4) Develop a Metadata strategy before evaluating, purchasing, and installing Metadata management products
5) Create or adopt Metadata standards to ensure interoperability of Metadata across the enterprise
6) Ensure effective Metadata acquisition for internal and external metadata
7) Maximize user access since a solution that is not accessed or is under-accessed will not show business value
6004/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Guiding Principles, cont’d
6104/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
8) Establish and maintain enterprise-wide business involvement in data stewardship, assigning accountability for metadata
9) Define and monitor procedures and processes to ensure correct policy implementation
10) Include a focus on roles, staffing, standards, procedures, training, and metrics
11) Provide dedicated Metadata experts to the project and beyond
12) Certify Metadata quality
8) Understand and communicate the necessity of Metadata and the purpose of each type of metadata; socialization of the value of Metadata will encourage business usage
9) Measure content and usage
10) Leverage XML, messaging and web services
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Using metadata descriptions of Bluetooth devices
6204/07/23
Attributes/FieldsKeyboard VCUMotorola S9Peter's i4Trackpad CGLTrackpad IDRTrackpad VCU
Data ColumnCGL TrackpadIDR TrackpadMotorola S9Peter's i4VCU KeyboardVCU Trackpad
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
6304/07/23
Example: iTunes Metadata
• Example: – iTunes Metadata
• Insert a recently purchased CD
• iTunes can:– Count the number
of tracks (25)
– Determine the length of each track
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
6404/07/23
Example: iTunes Metadata
• When connected to the Internet iTunes connects to the Gracenote(.com) Media Database and retrieves:– CD Name– Artist– Track Names– Genre– Artwork
• Sure would be a pain to type in all this information
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
6504/07/23
Example: iTunes Metadata
• To organize iTunes – I create a
"New Smart Playlist" for Artist's containing "Miles Davis"
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Example: iTunes Metadata
6604/07/23
• Notice I didn't get the desired results
• I already had another Miles Davis recording in iTunes
• Must fine-tune the request to get the desired results– Album contains "The
complete birth of the cool"
• Now I can move the playlist "Miles Davis" to a folder
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Example: iTunes Metadata
6704/07/23
• The same: – Interface– Processing– Data Structures
• are applied to – Podcasts
– Movies
– Books
– .pdf files
• Economies of scale are enormous
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview
2. What is metadata and why is it important?
3. Types of metadata
4. Metadata for unstructured data
5. Strategy and implementation
6. Guiding Principles
7. Take Aways, References and Q&A
68
Tweeting now: #dataed
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Summary
6904/07/231/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
References & Recommended Reading
7004/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
References, cont’d
7104/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
References, cont’d
7204/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
References, cont’d
7304/07/23
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Questions?
74
It’s your turn!
Use the chat feature or Twitter (#dataed) to submit
your questions to Peter now.
+ =
8/14/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
04/07/23 © Copyright this and previous years by Data Blueprint - all rights reserved!
Upcoming Events
75
October Webinar:Engineering Solutions to Data Quality Challenges
October 9, 2012 @ 2:00 PM – 3:30 PM ET
(11:00 AM-12:30 PM PT)
November Webinar:Get the Most Out of Your Tools:
Data Management Technologies
November 13, 2012 @ 2:00 PM – 3:30 PM ET
(11:00 AM-12:30 PM PT)
Sign up here:•www.datablueprint.com/webinar-schedule •www.Dataversity.net
Brought to you by:
8/14/2012