MarkLogic Overview Clark D. Richey, Jr. – Technical Director, MarkLogic [email protected] @crichey.
Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior...
Transcript of Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior...
![Page 1: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/1.jpg)
13 June 2019© MARKLOGIC CORPORATION
Pete AvenSenior Principal Solutions Engineer
@peteaven
Intro to MarkLogic Data Hub Architecture
![Page 2: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/2.jpg)
Data is (still) in silos
![Page 3: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/3.jpg)
MarkLogic Data Hub
ADVANCED SECURITY
SMART CURATION
UNIFIED PLATFORM
LOAD DATA AS IS SIMPLE DEVELOPMENT
![Page 4: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/4.jpg)
KEY PRINCIPLES
Agility in Action Data Services First
Business Answer First
Expect and Embrace Change
Governed by Default
Deploy Anywhere
![Page 5: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/5.jpg)
DATA AGILITY
Data Services First Minimize up-front work by
focusing on business valueand working back to data
Reduce execution risk with aggressive scoping, frequent iterations, continuous feedback
Increase returns on cumulative data
Direct analogue and enabler for Agile software development
![Page 6: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/6.jpg)
Focus on Ease and Impact (ROI)Ea
se o
f Exe
cutio
n
Business Impact
Find Orders
High impact
Predict fraud
LEADING WITH BUSINESS VALUE
Flag Fraud
![Page 7: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/7.jpg)
The Data Hub In Action
ERP
CRM
Taxonomy
Inventory
places
includes
Customer
Order
Product
wheresMyOrder
Flag Fraud
Segregate Cohorts
DATA SERVICES FIRSTI
![Page 8: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/8.jpg)
MarkLogic Architecture
STORAGE LAYERScalability and Elasticity
ACID Transactions
INTERFACE LAYER
Data ServicesJSON, XML, RDF, Geo, Text, Binaries
RESTAPI
Graph / SPARQL
QUERY LAYER
JS XQuery SPARQL
JavaScript XQuery SPARQLSQL
INDEXES Universal Index
Geospatial Index
Triple Index
AutomatedFailover
ReverseIndex
DATA LOGIC
![Page 9: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/9.jpg)
Expect and Embrace Change
Upstream
Quality and Meaning
New Sources
Messy or unexpected data
Ambiguous or conflicting definitions
Downstream
Business Requirements
New opportunities enabled by creative reuse
Get value sooner
Experiment with less cost
Everywhere
Compliance and Governance
New regulations and enforcement
Increased threats
Sharing not hoarding
![Page 10: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/10.jpg)
ERP
/wheresMyOrder
Customer
Collection:/acme/customers
![Page 11: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/11.jpg)
Hierarchical, sparse, high cardinality
Precise structure to free text
Change the data, change the schema
Standard JSON or XML, text, binary
Documents Represent Data More Naturally
DATA MODEL
![Page 12: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/12.jpg)
ERP
/wheresMyOrder
Customer
Collection:/acme/customers
Document Data Model Load as is Universal indexing
Values Full text Structure Scalar ranges Geospatial
Schema on read Organize by collections,
directories
![Page 13: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/13.jpg)
ERP
/wheresMyOrder
CRM eCommerce
Customer
![Page 14: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/14.jpg)
Customer
prov:derivedFrom
prov:generatedBy
rdf:type
rdf:type
prov:wasRevisionOf
/wheresMyOrder
e7e9879a…
?
![Page 15: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/15.jpg)
Customer
Order
Product
Purchased
Places
Includes
RelationshipsGRAPHS
Entities are documents
Relationships are triples
- Entities related to Entities
- Entities related to Facts
- Facts related to Facts
Infer new relationships
![Page 16: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/16.jpg)
Derived fromType
Type
PII
SSN
Semantic RelationshipsGRAPHS
Entities are documents
Relationships are triples
- Entities related to Entities
- Entities related to Facts
- Facts related to Facts
![Page 17: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/17.jpg)
Semantic RelationshipsGRAPHS
Entities are Documents
Relationships are triples
- Entities related to Entities
- Entities related to Facts
- Facts related to Facts
73fa4dc0…
Is Concept
Same As
SKOS
![Page 18: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/18.jpg)
Customer
30d623ff…
,
Order Product
acme:includesacme:places
73fa4dc0…
isConcept
rdf:type
rdf:type
acme:purchased
e7e9879a…
acme:powerOfAttorney
0.
1.
rdf:type
prov:generatedBy
![Page 19: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/19.jpg)
Governed by DefaultPUTTING THE “MS” BACK IN DBMS
Manage policy along with the data and metadata that it governs
Query that policy just like data to make enforcement model-driven
Automatically enforce policy in the database
Track lineage as data and policy change
![Page 20: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/20.jpg)
Developer
Ummm,Can you repeat that please?
Domain Expert
![Page 21: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/21.jpg)
Domain Expert
MODEL-DRIVEN
Data, Metadata, and Policy Model important business concepts as
needed (and not before)
Manage policy along with the data and the metadata it governs
Drive business processes and configuration from queryable policy definitions
![Page 22: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/22.jpg)
Customer
e7e9879a…Secure by DesignGOVERNED BY DEFAULT
Confidentiality: Role-based access control and encryption at rest, in motion
Integrity: Transactional consistency and auditable trustworthiness
Availability: Elastic scale out and HA/DR
![Page 23: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/23.jpg)
Deploy AnywhereAGILE INFRASTRUCTURE
Align infrastructure costs with SLAs using elastic scaling
Avoid lock-in with flexible cloud and on premise deployment
Reduce risk with automation and componentization
![Page 24: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/24.jpg)
MarkLogic in Any CloudCLOUD NEUTRAL
• Proven in the cloud
• Private, hybrid, or public cloud
• AWS, Azure, and Google Cloud (and others)
• Deployment automation
![Page 25: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/25.jpg)
CUSTOMER LDAP
CUSTOMERVPC
VPCPEERING
INGESTION & CURATION ACCESS
SERVICE VPC
(CUSTOMER ISOLATED)
LOAD BALANCER
LOAD BALANCER
Data
D-NODES
Operational AnalyticalCuration
LOAD BALANCER(8010-13-8000)
MarkLogic Tools
DMSDK MLCP REST API
MarkLogic Data Hub Service
![Page 26: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/26.jpg)
Much More Than a Database as a Service
On-Premises
DATA CENTERS
NETWORKING
STORAGE
SERVERS
VIRTUALIZATION
OS
DOCUMENT DB
GRAPH DB
RELATIONAL DB
SEARCH
ETL
MDM
SECURITY
APPS
DATA CENTERS
NETWORKING
STORAGE
SERVERS
VIRTUALIZATION
OS
DOCUMENT DB
GRAPH DB
RELATIONAL DB
SEARCH
ETL
MDM
SECURITY
APPS
DATA CENTERS
NETWORKING
STORAGE
SERVERS
VIRTUALIZATION
OS
DOCUMENT DB
GRAPH DB
RELATIONAL DB
SEARCH
ETL TOOLS
MDM
SECURITY
APPS
DATA CENTERS
NETWORKING
STORAGE
SERVERS
VIRTUALIZATION
OS
DOCUMENT DB
GRAPH DB
RELATIONAL DB
SEARCH
ETL TOOLS
MDM
SECURITY
APPS
IaaS DBaaS Data Hub Service
The most comprehensive out-of-the-box cloud service stack
HARMONIZATION HARMONIZATION HARMONIZATION HARMONIZATION
![Page 27: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/27.jpg)
Tool Chain
![Page 28: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/28.jpg)
KEY PRINCIPLES
Agility in Action Data Services First
Business Answer First
Expect and Embrace Change
Governed by Default
Deploy Anywhere
![Page 29: Intro to MarkLogic Data Hub Architecture · 13 June 2019© MARKLOGIC CORPORATION Pete Aven. Senior Principal Solutions Engineer. @peteaven. Intro to MarkLogic Data Hub Architecture](https://reader034.fdocuments.in/reader034/viewer/2022042419/5f364e6cde41ea03022eac0d/html5/thumbnails/29.jpg)
Thank you