DQS & MDS in SQL Server 2016
-
Upload
sebastien-notebaert -
Category
Technology
-
view
121 -
download
3
Transcript of DQS & MDS in SQL Server 2016
Data Quality Services and Master Data Services in SQL Server 2016
2017-03-15
Agenda
• Introduction : 5 min.• Reality check, do you have these issues?• What is master data management?
• Master data management in SQL Server 2016 : 15 min.• Data Quality Services (DQS)• Master Data Services (MDS)• What’s new in SQL Server 2016 for MDS?
• Data Quality Services demo : 15 min.• Cleansing & Matching data using Excel MDS Add-in• Cleansing data using SSIS
• Master Data Services demo: 20 min.• Creating model, entities, attributes and business rules• Load data using SSIS
33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7
Sébastien NotebaertSenior Consultant
Microsoft Data Platform expert with 15+ years experience in deploying data warehousing, analytics, data integration and MDM solutions.
• Services :• Projets BI• Conseil BI
• Fondée en 2010• 35 experts• Clientèle :
• Grande entreprise• PMEs
• Services :• Infra Cloud (Azure)• Solutions Cloud
• Nouvelle pratique• Issue de l’acquisition
GTO• 2 experts, recrutement
actif• Clientèle :
• Grande entreprise• PMEs
• Services gérés• Cloud privé• Cloud Public
• Fondée en 2006• 20 experts• Clientèle :
• PMEs• Domaine Médical
• Produits :• Solution BI
Financier• Solution processus
budgétaire• Reporting ERO
• Fondée en 2010• 5 experts• Clientèle :
• PMEs
Who we are…
Reality check: do you have these issues?
• Do you have instances of invalid data impacting business processes?
• Do you wish your business users could manage data themselves such as Customer and Product?
• Do you have IT resources spending time investigating data quality issues and/or applying data fixes?
• Do you have the need for consolidation and distribution of data to other systems?
• Do you have an environment of heterogeneous systems which all could benefit from a single view of domain data such as Customer or Product?
-> You need master data management!
What is master data management?
• The technology, tools, and processes required to create and maintain consistent and accurate master data
• Master data is a set of data objects that are at the center of business activities like: Customers, Products, Cost Centers, Locations, …
• Master data is not transactional data like a sale or inventory movement
• To succeed an MDM initiative must include business processes, people AND data
MDM Architecture Style
Style Description Which
System is
Master
Update
Source
Systems
Comple
xity
REGISTRY Master data is not consolidated, but is maintained as a set of “stub”
records mapped to attributes stored in the source systems. There
is little data movement other than create or delete global stub
records in the registry. The golden record is assembled dynamically
using complex distributed queries. The upside is that a real time
central reference can be made with little or no infrastructure
investment. The downside is that without central governance of the
data the golden record isn’t highly reliable.
Source
system
Not
applicable
Data stays
in source
systems
1
CONSOLIDATED Master data is consolidated from multiple source systems into a
physical golden record for downstream consumption; however, any
updates made to the master data are not returned to the original
sources. Consolidated MDM hubs are quick and inexpensive to set
up, and offer a big return by enabling reliable enterprise-wide
reporting. Data movement is typically tolerant of inter-day latency
and managed through inexpensive batch processes, but it’s a one-
way street from source systems.
Source
system
No 2
MDM Architecture Style (continued)
Style Description Which
System is
Master
Update
Source
Systems
Comple
xity
COEXISTENCE Just like consolidated, coexistence harmonizes multiple sources into a
physical golden record for downstream consumption. Coexistence
adds the important step of updating the source systems. These
requirements are typically high latency and can usually be met at an
acceptable time and cost through bi-directional batch processes.
Coexistence is a natural evolution of the consolidated architecture
with the added benefit of linking centrally governed data back to the
source systems. Interfacing with complex data sources, such as ERP
systems, can become a costly drawback.
Source
system
Yes 3
TRANSACTIO
OR
CENTRALIZED
While this approach also creates a physical golden record, the key
differentiator from the consolidated and coexistence styles is the
MDM hub extends enterprise governance over the source systems.
This introduces shorter latency requirements that are typically
addressed with a combination of web services integration and an
authoring application that bridges centralized and application-
specific governance needs. In most cases, this is a big step up in cost,
complexity and implementation time. The pay-off is more
comprehensive governance in real-time; however, too many complex
sources can push cost and complexity beyond the value created by
the architecture.
MDM
system
Yes 4
Master Data Management Entity Examples
People Things Places Abstract
Customers
Vendors
Sales People
Employees
Partners
Patients
Products
Business Units
Bill of Materials
Parts
Storage Bins
Equipment
Locations
Stores
Wells
Power Lines
Geo Areas
Warehouses
Accounts
Warranties
Time
Metrics
Securities
Contracts
Master Data Management Solution Areas
Data Quality
Improve Efficiency
Compliance
Retain Customers
Merger & Acquisition
Cross Reference
Golden Records
• Human typographical errors; incomplete information; spreadsheet data management• Mergers and consolidation; ERP implementations, consolidation or migration• New purposes for old data; retire old applications such as mainframe applications• Single point of data maintenance; BI reporting
• Different types of customer accounts• Accurate view of data by implementing MDM and DQ • Single point of data maintenance• Cross sell and upsell
• Tracking spends by customer • Province and federal legislation
• Single view of customer spend, channels, cross sell and upsell
• Cross reference of same customers across multiple systems• Survivorship of best consolidated data across multiple systems
• Single view of anything that has attributes that can be matched• Cleanup of source systems with business rules and golden records pushed back
• Merging chart of accounts; consolidate financial reporting • Single view of product• Single view of customers
Master Data Management in SQL Server 2016
Master Data Management in SQL 2016
MDS & DQS in a Consolidated MDM Architecture
Data Quality Services
Data Quality Services
• SQL Server Data Quality Services (DQS) is a knowledge-driven data quality tool.
• DQS enables you to build a knowledge base and use it to perform a variety of critical data quality tasks including: correction, enrichment, standardization, and de-duplication of your data.
• DQS is like a spell checker for your data
• DQS was shipped in SQL Server 2012 (ent. & BI ed.)
Data Quality Services (continued)
• DQS consists of Data Quality Server and Data Quality Client, both of which are installed as part of SQL Server 2016.
• Data Quality Server is a SQL Server instance feature that consists of three SQL Server catalogs with data-quality functionality and storage
Data Quality Services Building Blocks
• DQS building blocks• Knowledge base
• Domains• Knowledge discovery• Matching policy
• Data Quality projects• Cleansing project• Matching (deduplication)
project
Using Data Quality Services
Create DomainsKnowledge Discovery
Clean Data Match Data
• Create a domainlike product brand or category
• Set it properties • Set it values• Set it rules• Set its term
based relations
• Train your domains by loading data
• Accept / reject / enrich / correct your domains
• Clean data setsusing your trained knowledge base
• Match and deduplicateyour data usinga matching policy
Data Quality Services Features & Tasks
Knowledge Bases and Domains•Building a KB•Importing and Exporting Knowledge•Managing a domain•Managing a composite domain
Data Quality Projects•Create a Data Quality project•Open, Unlock, … a DQ project•Open an SSIS Project in DQ client
Data Cleansing•Cleanse Data Using DQS•Cleanse Data in a Composite Domain
Source: https://msdn.microsoft.com/en-us/library/hh213042.aspx
Data Matching•Create a matching policy•Run a Matching Project
Reference Data Services in DQS•Configure DQS to use reference data•Attach domain to reference data•Cleanse data using reference data
Data Profiling and Notifications
DQS Administration
DQS Security
-> Topics in bold will be covered during the demo
Master Data Services
Master Data Services
• SQL Server Master Data Services is an Master Data Management (MDM) tool to define and manage non-transactional lists of data, with the goal of compiling maintainable and validated master lists for the enterprise.
• An MDM project generally includes an evaluation and restructuring of internal business processes along with the implementation of MDM technology. The result of a successful MDM solution is reliable, centralized data that can be analyzed, resulting in better business decisions.
• MDS was shipped in SQL Server 2008 R2 (datacenter & ent. ed.)
Master Data Services (continued)
Master Data Services has the following components:• Master Data Services Configuration Manager, a tool you use to create and
configure MDS databases and web applications.
• Master Data Manager, a web application you use to perform administrative tasks (like creating a model or business rule), and that users access to update data.
• MDSModelDeploy.exe, a tool you use to create packages of your model objects and data so you can deploy them to other environments.
• Master Data Services web service (WCF), which developers can use to extend or develop custom solutions for Master Data Services.
• Master Data Services Add-in for Excel, which you use to manage data and create new entities and attributes.
Master Data Services Building Blocks• The model is the most fundamental object in an MDS solution• Models are the containers that encapsulate all other MDS objects (i.e. entities, attributes and business rules)
Interacting with Master Data Services
• Master Data Manager Web User Interface• MDS Web Services API (WCF API)• Subscription views via T-SQL• Staging stored procedures• MDS Add-in for Excel
SSIS package that calls MDS stored procedures ->
Master Data Manager Web Application
Work with master data
Work with models, entities, attributes, business rules
Master Data Services Excel Add-in• The Excel Add-in for MDS offers users functions that can be found in the Master Data
Manager web app
• Users can update and view master data, as well as modify or create entities
• A major benefit of the Excel Add-in is the ability to quickly bulk load data into MDS
• The Excel Add-in provides users the ability to use Data Quality Services to clean data
before it moves into MDS
Using Master Data Services
Create Entities Load Members Validate Review Version
• Createentities and attributes to hold your master data
• Define business rules to ensure quality
• Load members using the staging process, Excel add-in or the API
• Validate the model’s data loaded to ensure it complies to all business rules
• Correct and enrich data that fails validation
• Version your model so that subscribing apps access only valid data
Massive improvements to performance and scalability
15x
MDS: What’s new in SQL Server 2016?
Transaction log retentionConfigurable settings for retaining the MDS transaction history table to enable automatic truncation. New member retention mode.
Display name for each objectGives more control over the names displayed for a given object – including the Code and Name attributes.
Entity Sync
Modeling and Management
MDS: What’s new in SQL Server 2016?
Type 2 subscription viewsSee attribute change history in an easy to consume format.
Custom and compound IndexesAdd a custom index on commonly used attributes to improve overall performance.
Business rules supporting custom SQL Scripts
One level approval workflows
Modeling and Management
MDS: What’s new in SQL Server 2016?
Granular security permissionsAllows permissions to be set around read, write, create, and delete.
Multiple administrator rolesSupport for Super User and Model Admin roles allows for multiple system administrators, and model level admins.
MDS: What’s new in SQL Server 2016?
User Experience Improvements
MDS: What’s new in SQL Server 2016?
-> Silverlight dependency implies using IE, no Chrome or Safari support
Master Data Services Features & Tasks
Create Structures to Contain Data•Models•Entities•Custom Index•Attributes•Domain-Based Attributes •Attribute Groups
Maintain Master Data•MDS Add-in for Microsoft Excel•Members•Business Rules•Transactions•Annotations•Hierarchies
Source: https://msdn.microsoft.com/en-us/library/hh231022.aspx
Improve Data Quality•Validation•Versions•Notifications •Security
Move Data•Importing Data •Exporting Data •Deploying Models
Develop a Custom Application•Developer's Guide •Microsoft.MasterDataServiceNamespace
-> Topics in bold will be covered during the demo
Demo
References
Master Data Services• Microsoft Master Data Services in SQL Server 2012 by James Serra: http://bit.ly/QW6kpQ• Master Data Overview: https://msdn.microsoft.com/en-us/library/ff487003.aspx• Master Data Features and Tasks: https://msdn.microsoft.com/en-us/library/hh231022.aspx• Master Data Developer’s Guide: https://msdn.microsoft.com/en-us/library/hh230994.aspx• What’s New in MDS 2016 : https://msdn.microsoft.com/en-us/library/ff929136.aspx
Data Quality Services• Data Quality Services: https://msdn.microsoft.com/en-us/library/ff877925.aspx• DQS Matching Transform for SSIS: https://ssisdqsmatching.codeplex.com/• MS DQS Blog using DQS Matching Transform:
https://blogs.msdn.microsoft.com/dqs/2013/06/25/automating-the-data-matching-process-in-sql-server-data-quality-services-dqs/
We are recruiting!
PROFILES
BI and Analytics Architect
Data Architect
Microsoft BI Developper
Analytics and Data Visuzlization Specialist
2
TECHNOLOGIES
Microsoft BI (SSAS, SSIS, SSRS)
Power BI
Tableau
MDM
SQL Server
THANK YOU FOR YOUR TIME!
THANK YOU TO THE DATA SCIENCE INSTITUTE FOR THIS OPPORTUNITY!
DRIVE SAFELY ON YOUR WAY HOME!
☺
33, Prince Street, Suite 284, Montreal (Quebec) H3C 2M7