5 Fundamental Strategies for Building a Data-centered Data ...€¦ · At the end of a 3-year life...

5 Fundamental Strategies for Building a Data-centered Data Center

June 3, 2014 Ken Krupa, Chief Field Architect Gary Vidal, Solutions Specialist

Last generation

Warehouse

Data Marts Archives

“Unstructured”

“ ”

Video Audio

Signals, Logs, Streams

Social

Documents, Messages

{ } Metadata

Search 🔍

Reference Data

Summary – The Data-centered Data Center

Elastic: flexible, shared-nothing, scale-out architecture

Cost competitive: low-cost commodity hardware, lower TCO

Converged: single data layer for operational and analytical workloads

Managing data life-cycle in real-time: prioritize your data storage

Governed, not renegade: customizable, transparent, secure

ELASTIC

Data organization in MarkLogic

Data inserted into stands One stand is in-memory Many other stands are on disk A collection of stands is a forest Each forest is an atomic unit and

can be managed and moved

Servers have Multiple Forests

Scale out

Clustering

Migration

Two forests on one node Bring a second node online Replicate a forest Disable the forest on the

original node Original forest on original

node fails over Enable the original forest as

a replica

Migration in one step $ cat forest-migrate.json

{ "operation": "forest-migrate”, "forest": [”forest-in-database", ”another-forest-in-database"], "host": ”destination-host” }

$ curl --anyauth --user user:password -X PUT -d @./forest-migrate.json \ -i -H "Content-type: application/json" \ http://anyhostinthecluster:8002/manage/v2/forests

Cluster topologies XA

Knowing where you’re going - and where you’ve been

Business context

What are your SLAs? How many requests per second does the application

have to support? How will the business grow? What will drive growth - and how fast will it go?

As-Built Capacities

How does your system perform under different usage profiles (e.g., QPS tests)?

How often do you hit the cache? What is your peak storage I/O? What is end-to-end recovery objective/capability?

Performance History

To handle more requests: • Fix Configuration • Add Disk IO via Volumes or Nodes • Add Ram to decrease Disk IO • Rewrite Query

Scaling out: questions to consider

How do you know when you need to add a node?

CPU/Memory/IO: when you get close to hardware limits, time to grow

High Performance: SLA’s may drive forest sizes; more docs, time to grow

High Capacity: running low on storage, time to grow

Easy (temporary?) fix—add RAM

Cheaper alternative

Increases cache hits for better performance

Fewer than three hosts, local forests MUST move across hosts

Use forest migrate to move forests from one host to another

Faster than backup/restore

Follow distribution pattern:

Don’t just swap masters/replicas on two: if one goes down, load is not split evenly across cluster

Adding a node - or RAM Migrating a forest

LOWERING TCO THROUGH

COMMODITY HARDWARE

Kryder’s Law: The density of hard drives

increases by a factor of 1,000

every 10.5 years.

(doubling every 13 months)

Moore’s Law: The density of transistors on

integrated circuits doubles

every 18 months.

The laws in action At the end of a 3-year life cycle, one new server can do the job of four old

servers.

At 1.5 Years, you can add 100% more capacity for 50% of original spend

For the cost of storing 1TB in 1996, we will be able to store 1PB in 2016.

Commodity hardware will reduce costs

Hardware/sizing recommendations

2U 25 SFF Chassis 2 Socket

8 Core/2.8Ghz

128GB – 256GB RAM

10GB Network

2 2GB RAID Cards

22 10K 900-1200GB Data Drives

VMWare NetApp recommendations (preliminary)

1U 8SFF Chassis 2 Socket

8 Core/2.8Ghz

128GB – 256GB RAM

10GB Network

1 10GB iSCSI 12-16 Spindles per Server, 10K SAS

Storage Economics

SAN/Scale-up

$2 - $10/Gigabyte

$1M gets: 0.5Petabytes

200,000 IOPS 1Gbyte/sec

NAS Filers

$1 - $5/Gigabyte

$1M gets: 1 Petabyte

400,000 IOPS 2Gbyte/sec

Local Compute

$0.20/Gigabyte

$1M gets: 10 Petabytes

5,000,000 IOPS 40 Gbytes/sec

SAN (Scale Up) Commodity (Scale Out)

Public cloud

$0.04/gb/month

$100K/month: 1.25 Petabytes (HA)

1,500,000 IOPS 150 GB/Sec

Signs of war: cloud prices have dropped recently

Google Cloud: - $0.04 GB-month for 1000GB

Amazon EBS: - $.055 GB-month (standard) - $.138 GB-month (provisioned)

Leveraging Scale-out Economics Run on existing Infrastructure today

Leverage Scale-Out Commodity Hardware as you grow

Leverage Cloud today or tomorrow

SAME DATABASE, SAME CODE

DATA LAYER CONVERGENCE

FEWER MOVING PARTS =

MORE AGILITY

Last generation

Warehouse

Data Marts Archives

“Unstructured”

“ ”

Video Audio

Social

Documents, Messages

{ } Metadata

Search 🔍

Reference Data

RDBMS: One Tool, Many Contortions OLTP

3rd normal form, updates, simple query Reporting DB

Because the OLTP app slowed down during heavy query use Enterprise Data Warehouse

Because we needed a unified view of the enterprise – Star schema enters the picture

Data Marts Because the EDW didn’t have everything – Also star schema

Federated Because it took too long to agree on a standard model

Hybrid Because Federated is too slow

If I run analytics in my OLTP DB then.... Won’t meet my SLA’s Too expensive No common data model Cache won’t ever be right Too expensive to keep around

context necessary for analytics

If I run transactions in my Analytical DB...

Transaction locks will block aggregate reads

Too expensive Why constrain ad-hoc

query? We need to investigate

The old consensus: mixing is bad

The new wisdom: mixing is good Operational with Analytics Risk calculations Underwriting Compliance Content Discovery Fraud

Analytics with Operations Operational BI Archival/E-Discovery Personalization Situational Awareness

SINGLE SOURCE OF TRUTH

Mixing workloads in MarkLogic - how it works

ML as an analytic database - examples and possibilities

Range indexes: in-memory columnar Query load separation Tiered storage and real-time replication Hadoop MapReduce and HDFS Transactions and ACID help manage and

prioritize data - better performance, lower TCO

Operations and analytics in MarkLogic

COPIES, NOT ETL

INFORMATION LIFE-CYCLE MANAGEMENT (FOR REAL)

Understanding the life cycle The older your database,

the more data you have

The older the data, the less likely you will reuse it

Storage requirements increase, but much of what is stored will go untouched

Data life cycle management, in three easy steps

1. Move data off active system to cheaper system.

2. Keep track of what you moved.

3. Provide facility for getting it back.

CERN: implementation is hard in the RDBMS world DBAs / database developers cannot easily

implement these policies by themselves.

Database admins, application developers, and application owners must work together to: Reduce amount of data produced Allow for database structure that can

facilitate archiving Define data availability requirements for

online data and archive Identify how to leverage database features

CERN: archiving RDBMS data is also difficult

The DBA removes old partitions from the production database and moves them to the archive. One option: use partition exchange to table Post-move jobs can implement compression, drop indexes

Sticking points: Set of data must be consistent Must build support in the application Have to validate access to archived data Archived data must remain readable in future

versions

Tiered Storage With Tiered Storage, you can… Define data tiers based on a range index

Have content balanced into forests by tier

Move an entire tier to different storage

Attach a tier to a different database

Query one database on one tier…

…or the other database on the other tier…

…or both at once All with no downtime, and 100% consistency

Tier 1 SAN Exadata ML usingDAS

Tier-1

Effective Cost/TB for Production Storage (all copies)

0100020003000400050006000700080009000

FlexPod/VCE NetApp ML usingDAS

Tier-2

GOVERNANCE + PROVENANCE

Data Governance Considerations

Security

Privacy

Security

Privacy

Provenance

Security

Privacy

Provenance

Retention

Security

Privacy Continuity

Provenance

Retention

Security

Privacy Continuity

Provenance Compliance

Retention

Last Generation

Warehouse

Data Marts Archives

“Unstructured”

“ ”

Video Audio

Social

Documents, Messages

{ } Metadata

Search 🔍

Reference Data

New Generation

Application

Summary Elastic systems let you respond rapidly to changing loads - and let you keep costs

in line with usage.

Scale-out systems on commodity hardware are much less expensive and more powerful than scale-up systems.

Converging transactional and analytical workloads into single data layer is not only possible - it is often a great idea. A single data layer can increase agility.

Managing information throughout its life cycle means more than choosing the cheapest storage possible - it means being able to manage and query data in real time.

Proper data governance is simpler in an enterprise NoSQL system.

5 Fundamental Strategies for Building a Data-centered Data ...€¦ · At the end of a 3-year life...

Documents

Transcript of 5 Fundamental Strategies for Building a Data-centered Data ...€¦ · At the end of a 3-year life...

My Data - A Nordic Model for human-centered personal data management and processing

Serial studies of Future of Consumer II Big data-driven ...€¦ · Big data-driven consumer insights Chapter I "Consumer-centered" strategies 2 To develop "consumer-centered" strategies,

Reliability Centered Maintenance From a Data Center Perspective

Automating User-Centered Design of Data- Intensive Processes · 2015. 7. 16. · RPR - Doctoral Colloquium, IT4BI-DC (eBISS 2015) Automating User-Centered Design of Data-Intensive

Google Fusion Tables: Web-Centered Data Management …web.engr.illinois.edu/~adityagp/courses/cs598/papers/fusion-tables.pdf · Google Fusion Tables: Web-Centered Data Management

Resource centered RDF data management - ISWC 2011iswc2011.semanticweb.org/fileadmin/iswc/Papers/Workshops/SSWS… · Resource centered RDF data management Ralf Heese1 and Martin Znamirowski2

Building the Data Capacity for Patient-Centered Outcomes ...2018 PORTFOLIO REPORT | 1 . Building the Data Capacity for Patient-Centered Outcomes Research: The 2018 Annual Report Office

Charting a Course for an Equity-Centered Data System

Resource centered RDF data management

2017 ANNUAL REPORT - MHIN · 2018-02-26 · This concept is called “Patient Centered Data Home.” With a Patient Centered Data Home, linked-HIEs in the Midwest can alert the home

How Tech Can Foster Person-Centered Connection During ... · How Tech Can Foster Person-Centered Connection During Social Isolation Jack York, President and Co-Founder – iN2L Kristi

5 Fundamental Strategies for Building a Data-centered Data ...

A COMPUTATIONAL FRAMEWORK FOR HUMAN-CENTERED MULTIMODAL DATA ANALYSIS

Patient Centered Care & the New CODA Standards€¦ · Patient-Centered Care Patient centered approach …norm for teaching & patient care …CAN have these characteristics . 1.Patient

Reliability Centered Maintenance From a Data Center Perspective March 2013.

How can Innovative, People-Centered Data Collection contribute … · 2018. 7. 16. · through biosensor dataMonitoring Network Air Quality ... vehicular traffic, highways, high speed,

BGPeep: An IP-Space Centered View for Internet Routing Data

A User-centered Design Methodology for Students’ data-focused InfoVis

Consumer-Centered Data and Strategies to …...Consumer-Centered Data and Strategies to Advance Evidence-Based Advocacy in Child Health Christina Bethell, PhD, MBA, MPH AAP Chapter

USER CENTERED DESIGN OF INTERACTIVE DATA VISUALIZATIONS