Netezza pure data
-
Upload
hossein-sarshar -
Category
Data & Analytics
-
view
238 -
download
7
Transcript of Netezza pure data
IBM PureData System for Analytics
Powered by Netezza
Hossein Sarshar
Agenda• What is PureData and Netezza
o History
o Characteristics
o Product chain
• PureData Hardware Architectureo Introduction
o Hardware architecture
o Paralleled structures
• Analytics with PureDatao Introduction
o In-database analytics tools
• Demo
IBM® PureData™ for Analytics 2
What is PureData and Netezza
PureSystems
PureFlex PureApplication
IBM® PureData™ for Analytics 3
In 2010, IBM bought a new analytics platform called
Netezza. It was founded in 2000 at Marlborough, CA.
IBM later rebranded it to PureData.
What is PureData and Netezza
PureSystems
PureFlex PureApplication PureData
IBM® PureData™ for Analytics 4
PureSystems Product Family
PureFlex: o Combines and optimizes compute, storage, networking and virtualization
capabilities under a single, unified management console into an
infrastructure system.
PureApplication:o Is a platform system designed and tuned specifically for transactional web
and database applications.
PureData:o Based on Netezza technology, PureData is all data experts need in a
single well tuned appliance.
IBM® PureData™ for Analytics 5
PureData
Operational Analytics
Transactions Analytics
PureSystemsCharacteristics
• Built-in Expertso No indexing/tuning/partitioning
o Fully parallel, optimized in-Database Analytics.
o No storage administration.
o No software installation.
• Integration by Design:o Server, Storage, Database in one easy to use package.
o Automatic parallelization and resource optimization to scale economically
o Enterprise-class security and platform management
• Simplified Experience:o Up and running in hours.
o Minimal ongoing administration.
o Standard interfaces to best of breed Analytics, BI, and data integration tools.
o Built-in analytics capabilities allow users to derive insight from data quickly.
o Easy connectivity to other Big Data Platform components
IBM® PureData™ for Analytics 6
Each of these come as an appliance equal to simplified yet strong private clouds with
minimal administration
PureData Introduction• It is a datawarehousing and data analytics
appliance that is fast enough to process terabytes of data in seconds. It is a fully parallel machine.
• Netezza’s main technology is using FPGA (Field Programmable Gateway Array) to filter unnecessary files in parallel manner.
• PureData uses Netezza technology to perform deep analytics on huge amount of data in a reasonable time.
• It is purpose-built for high performance analytics.
• It supports all DB structures (3NF, Star, De-Normalized table)
IBM® PureData™ for Analytics 7
PureData Architecture
IBM® PureData™ for Analytics 8
Disk storage
RAID 1 disksHigh speed data
streams
SMP Host
Redhat linuxservers
OptimizerCompiler
A gateway to the system
Snippet-Blades
Query accelerator using FPGAs
S-Blades (SPU)
IBM® PureData™ for Analytics 9
S-Blades
IBM® PureData™ for Analytics 10
Intel Quad-Core
Dual-Core FPGADRAM
IBM BladeCenter Server Netezza DB Accelerator
SAS Expander
Module
SAS Expander
Module
S-Blades Overview• There are 8 intel core on IBM Blade-Center Server
and 8 FPGA on Netezza DB accelerator.o FPGA has similar dimensions a CPU has, consumes 5 times less power and
clock speed is about 5 times less
o More caching capability
o Low latency and high throughput
• Each of these S-Blades takes ownership of 6-8 disks.
• The queries are divided into subqueries that are
processed by S-Blades.
IBM® PureData™ for Analytics 11
PureData AMPP (Shared-Nothing) Architecture
12
Advanced Analytics
Loader
ETL
BI
Applications
FPGA
Memory
CPU
FPGA
Memory
CPU
FPGA
Memory
CPU
Hosts
SMP
Host
Disk
EnclosuresS-Blades™
Network
Fabric
Netezza Appliance
FPGA Secret Sauce
IBM® PureData™ for Analytics 13
FPGA Core CPU Core
Uncompress Project Restrict,
Visibility
Complex ∑
Group by, …
select DISTRICT,
PRODUCTGRP,
sum(NRX)
from MTHLY_RX_TERR_DATA
where MONTH = '20091201'
and MARKET = 509123
and SPECIALTY = 'GASTRO'
Slice of table
MTHLY_RX_TERR_DATA
(compressed)
where MONTH = '20091201'
and MARKET = 509123
and SPECIALTY = 'GASTRO'
sum(NRX)
select DISTRICT,
PRODUCTGRP,
sum(NRX)
Using FPGA reduces a tremendous among of
unnecessary data movement
PureData System Configuration
14IBM® PureData™ for Analytics
PureData System Configuration
IBM® PureData™ for Analytics 15
PureData System Configuration
IBM® PureData™ for Analytics 16
Single Rack System Multi Rack System
Specs N3001-
002
N3001-
005
N3001-
010
N3001-
020
N3001-
040
N3001-
080
Racks 1 1 1 2 4 8
Active S-Blades 2 4 7 14 28 56
CPU Cores 40 80 140 280 560 1120
FPGA Cores 32 64 112 224 448 896
User Data in TB 32 98 192 384 768 1536
N3001 is the newest IBM PureData
What is Achievable• Having agile analytics platform.
• No administration effort to install/manage
• Scalability in petabyte level
• Linear speedup scalability by adding additional
racks.
• Big Data Meets Deep Analytics => No need to
sample
IBM® PureData™ for Analytics 17
High Performance Analytics Architecture
IBM® PureData™ for Analytics 18
PureData Analytics Modules
IBM® PureData™ for Analytics 19
Netezza In-Database Analytics Options
Classification Time Series Clustering
Associate Rules
Simulation and Monte
Carlo AnalysisGeospatial
IBM® PureData™ for Analytics 20
Demo• Installation
• Client Tool Exploration
• Command Execution
IBM® PureData™ for Analytics 21
Summary• A system for analytics
• Out-of-the-box solution
• It uses FPGA technology to boost query execution
• It uses nothing-shared approach.
• PureData uses open standards to communicate to
outside world
• It has many NZ in-database and 3rd party in-
database options to enrich our analytics
IBM® PureData™ for Analytics 22
References• http://www-01.ibm.com/software/data/netezza/
• http://www.ibm.com/ibm/puresystems/ca/en/
IBM® PureData™ for Analytics 24