SAP IQ 16 Product Annoucement


Agenda

Welcome

Agenda• Introduction to Dobler Consulting• Introduction to SAP Sybase IQ, Challenges and Trends, Key Capabilities• What’s New In Version 16• Q&A

Presenters• Dan Lahl - Vice President of Product Marketing, SAP• Joydeep Das - Vice President Product Management, SAP Database Management

Portfolio• Peter Dobler - CEO Dobler Consulting• Allan Roeder - VP of Sales Dobler Consulting

Closing


Introduction to Dobler Consulting

Dobler Consulting is a leading information technology and database services company, offering a broad spectrum of services to their clients; acting as your Trusted Adviser, Provide License Sales, Architectural Review and Design Consulting, Optimization Services, High Availability review and enablement, Training and Cross Training, and lastly ongoing support and preventative maintenance. Founded in 2000, the Tampa consulting firm specializes in SAP/Sybase, Microsoft SQL Server, and Oracle.

Visit us online at www.doblerconsulting.com, or contact us at 813 322 3240, or [email protected].

http://www.doblerconsulting.com/

mailto:[email protected]


Your Data is Your DNA, Dobler Consulting Focus Areas

Strategic Database Consulting

DBA Managed Services

Database Training Programs

SAP VAR D&T

Cross-Platform Expertise SAP Sybase® SQL Server® Oracle®

Dan Lahl9/19/2013

SAP Sybase IQ 16Tap Into Big Data at the Speed of Business


Today’s topics

Introduction to SAP Sybase IQ Challenges and Trends Key Capabilities:

– Exploit the value of Big Data– Transform businesses through deeper insights– Extend the power of analytics across the entire enterprise

Customer and Ecosystem Success


SAP Sybase IQ: smarter answers now

Up to 1,000x faster than traditional DBMS

Handles volumes of ad hoc queries in real time

Simple and complex analytics on structured and unstructured data

Speed & Flexibility

2x as many customers as the next leading provider

96%+ customer satisfaction rates

Consistently ranked as EDW market leader by leading analysts*

Mature and Proven

Compression reduces data volume by up to 70%

Less storage and maintenance required

Easy to lean and use –requires minimal training

Lower TCO

Source: Magic Quadrant for Data Warehouse Data Management System, Gartner, January 31, 2013

Business Challenges and Trends


What’s happening in the marketplace…

Exploding Data Volumes

The Need for Speed

Rising IT Costs and Complexity


Today’s challenges

Lost Revenues; Lack of Insight

High Costs & Complexities

Data Management Challenges

Security

Slow Performance


• Marketing analyticsDigital channelsTrack visits, discover best channel mix: e-mail, social media, and search

• Sales analyticsDeep correlationsPredict risks based on deal DNA (e-mails and meetings) and pattern match

• Operational analyticsAtomic machine dataAnalyze RFID, blogs, Short Message Service (SMS), and sensors — for continuous operational inefficiency

• Financial analyticsDetailed simulationsLiquidity and portfolio simulations — for stress tests and error margins

Competitive advantage is now data-driven


The data explosion continues

“Petabyte is the new Terabyte” - ForbesData volumes and applications in analytics environments are “growing up” to Big Data

Source: Wikibon, Feb 2013, Big Data and Data Warehousing, www.wintercorp.com

$4.6B

Big Data Apps

$6.9B

http://www.wintercorp.com/

Key Capabilities


SAP Sybase IQ 16

SAP Sybase IQ transforms the way companies compete and win through actionable intelligence delivered at the speed of business to more people and processes.


Key CapabilitiesThe Value of SAP Sybase IQ 16

Extend the power of analytics across the entire enterprise

Exploit the value of Big Data

Transform businesses through deeper insights

1

2

3


Exploit the value of Big Data

Unconstrained, unlimited, fast access

Flexible, expandable grid architecture

Integral native frameworks

Cost-effective compression


Transform business through deeper insights

Unified platform for all data

Low latency for immediate updates

Comprehensive answers using massive volumes


Extend the power of analytics across the entire enterprise

User driven self-service at “Google speed”

Highly available enterprise data

Fast and efficient MPP and query optimization

Secure data and systems

Customer & Partner Success


SAP Sybase IQGlobal Presence in a Range of Data-Driven Industries

Since adding Sybase IQ, the demand for reports has risen dramatically as users realize just how much more helpful the information is that we can now provide. We receive requests for 10-20,000 more reports per month, and ad hoc queries have grown from 5,000 to 10,000 per month.

Marc Guillard, DBA , BNP Paribas Securities Services

“

”

“

”

“

”

SAP Sybase technology added a new dimension to our business – we can now report on any data our executives need within minutes. Previously, reports were impossible or took a full day to generate.

Bonjan Sovilj, Research and Development Manager, Mozzart

…in North America …in EMEA …in EMEA

I would venture to say SAP Sybase IQ is the fastest in the industry. We load almost 10 billion rows a month, a terabyte a quarter. With SAP Sybase IQ, it just flies.

Craig Silver, Senior Database Architect, Nielsen Media Research


Measurable benefits for our customers

75% 75% 800%Reduced daily DB administration by 75%

Faster data loads -increase productivity during maintenance window

Reduced BI storage requirements by 75% a ratio of 4 to 1

http://www.digiturk.com.tr/



2,500 + Customers

96% Customer satisfaction rating

4,500 Installations

SAP Sybase IQ Market-Leading RDBMS Platform for EDW

http://www.comscore.com/

http://www.comscore.com/



Transforming companies through new insights.

SAP Sybase IQ 16: What’s NewJoydeep DasVice President, Product Management,SAP DBMS Portfolio


2009

Very large database (VLDB) platform foundation

2011

MapReduce API

2011

PlexQ MPP Foundation

2010

Full text search; Web 2.0 API

2009

In-database analytics

1H 2013

XLDB Analytics

SAP SYBASE IQ 16PATH TO ACTIONABLE INTELLIGENCE


Ecosystem

SAP SYBASE IQA COMPREHENSIVE ANALYTICS PLATFORM

Most mature column store

Comprehensive lifecycle tiering

MPP queries, virtual marts, and user scaling

High-speed loads

Structured and unstructured store

Comprehensive SQL with OLAP

Built-in full-text search

In-database analytics with MapReduce

Web 2.0 APIs

Big Data open source APIs

Optimized BI and EIM

Development and administration tools Predictive analytics Packaged ILM

applications

Bradmark, Symantec,

Whitesands, Quest, and

ZEND

SAS, SPSS, KXEN, Fuzzy

Logix, Zementis, and Visual Numerics

BMMSoft, SOLIX, and PBS

Sybase PowerDesigner,

Sybase Replication Server,

SAP and Panopticon

Application services

DBMSHadoop and R

SAP Sybase IQ 16Architectural Details


SAP Sybase IQ 16 Engine

Multiplex grid architecture

Storage area network

Loadingengine

In-database analytics

Text search

Web-enabled analytics

Information lifecycle m

anagement

Low latency, write optimized store

Role-based access control

Communications and security

N-bit and tiered indexing

Column indexing subsystem

Query engine

Column store

Aggressive scale out

Fullyparallel

Resilient

Web based adm

inistration and monitoring

LDAP authentication

Hash partitioned tables and data affinity

New Generation PETABYTE SCALE store

New indexing technology for increased compression and fast, incremental batch loads

SAP SYBASE IQ 16WHAT’S NEW!

High performance, fully parallel bulk loading

Enterprise-grade RBAC authorization and LDAP authentication

Comprehensive web-based monitoring and administration

Low latency, concurrent write optimized delta store

Re-engineered column store for extreme data volumes

Resilient Multiplex grid withstands network interrupts and server failures

Aggressively scaled out query engine

Intelligent query engine with data affinity for “shared nothing” performance on a flexible “shared everything” architecture


SAP SYBASE IQ 16INNOVATIONS FOR EXTREMELY LARGE DATABASES (XLDB)

Storage Architecture

• New generation column store• New partitioning and compression

Query Processing

• Data affinity• Aggressively parallel and distributed

Loading Engine

• Fully parallel bulk loading• Real-time loading into delta store

System Reliability

• Grid resiliency• LDAP and role-based security

SAP Sybase IQXLDB

Analytics Petabytes Real-time


SAP SYBASE IQ 16NEW COLUMN STORE ARCHITECTURE

Value propositionEnhanced compressionStorage savingsImproved I/O bandwidth

Architectural considerationsSupport variable number of cells per pageSupport various page formats within a columnHigh performance access pathsEven with variable length data, insert/update/delete efficiently into an existing pageRicher metadata

Logical page of variable sized cells

cell values

directory

Logical page of fixed sized cells

Before

NOW…


SAP SYBASE IQ 16N-BIT DICTIONARY COMPRESSION

Value propositionReduced memory footprintImproved effective I/O ratesMore efficient table scansReduced disk space

Architectural considerationsN-bit Fast Project (FP) Indexes, instead of 1, 2 and 3-byte FP IndexesDifferent data pages for same column can have different values of “N” for N-bit No more requirement to rollover FP format for all column dataColumn is N-bit by default, unless otherwise specified to be flatOptions provided to set threshold for rollover to flat (to prevent large dictionaries)Options provided to prevent rollover to flat (to prevent long rollover time)Compatibility mode allows the database to mimic IQ 15 rollover behavior

Raw Data = 400 MB; 1 Billion 4-byte integer values fn (N)

N Token Size Savings

2 (1B * 2 ) / 8 = 250MB 93.75%3 (1B * 3 ) / 8 = 375MB 90.6%4 (1B * 4 ) / 8 = 500MB 87.5%

…….24 (1B * 24 ) / 8 = 3000MB 25%

2->33->44->55->66->88->1010->1212->1616->2121->24


SAP SYBASE IQ 16FULLY PARALLEL BULK LOAD

Value propositionImproved performanceMaximize use of existing cores on the machineDynamic load balancing

Architectural considerationsLoad an index/column concurrently with multiple threadsRemove all bottlenecks which contribute to inefficient use of CPU and storageDynamically scale up and down degree of parallelism depending on the workload

Before…

NOW…


SAP SYBASE IQ 16SMALL BATCH LOAD PERFORMANCE

Value propositionImproved performance of frequent, small batch loadsPredictable performance of small batch loads: performance is proportional to the size of the

data being loaded, not the table being updated

Architectural considerationsInserting into a large High Group (HG) b-tree index is costlyHG index will have a tiered structure with a small tree component and a large tree componentSmall, batch loads into the HG index are written to the small tree component quickly and synchronouslyThe small tree component is periodically merged into the large tree component as a background task

Small tree component

Btree-based index

Trickle insert

PeriodicMerge


SAP SYBASE IQ 16HIGH VELOCITY DATA LOADING

Value propositionContinuous analytics over operational dataHigh velocity, concurrent data modificationsExploit large memory and core footprints

Architectural considerationsWrite optimized in-memory In-memory RLV (Row-level versioned) storeRow level locking, and statement snapshot isolationLow latency micro operationsIn-memory RLV store has reduced compression, no sorting, no indexingFully recoverable with dedicated transaction logAsynchronous data transfer from In-memory RLV store to IQ main storeUsers choose which tables are In-memory RLV tables

In-memory

RLV Store

RLV log on shared store

IQ main on shared store

Hybrid Table Storage

Write ahead/commit

periodic merge

IQ query engine

IQ DML engine

RLV rows

main rows

Transaction Manager

Consistent View Manager Version Manager

Lock Manager

Insert/Update/Delete

Consistent Viewglobal row

space


SAP SYBASE IQ 16QUERY SCALE OUT – HASH PARTITIONING

Value propositionGives best of both worlds of shared everything and shared nothingDecreases hardware needs and localizes processing

Architectural considerationsData is automatically partitioned during loading with built-in hash algorithmsData is divided into persistent subsets Reduces results sharing More efficient CPU usage Reduces instantaneous temp usageOptimizer will use hash partitions for join and group by when available


SAP SYBASE IQ 16QUERY SCALE OUT – DATA AFFINITY

Value propositionProvides efficient utilization of cluster-wide cache resources such as shared tempAchieves ultra low-latency data access while preserving elastic multiplex capabilities Group by and Order by benefit

Architectural considerationsAffinity is automatically configured and used in multiplexAdapts to query workloads and self managesData must be hash or logically partitioned Each partition is assigned to a specific node

Fact Dim

Cache Cache


SAP SYBASE IQ 16QUERY SCALE OUT – Query Runtime and DQP Optimization

Value propositionEliminates SMP (single node) and DQP (distributed query processing) bottlenecksLeverages large memory and scale outLowers shared temp and interconnect bandwidth

Architectural considerationsTakes place automatically as optimizer selects best plan based on costFor non-partitioned data, new Join and Group algorithms reduce the amount of intermediate results exchanged

Group ByQuery operator specialized for med cardinality

processing

Eliminate single-threaded hash

group-by bottlenecks

Improve DQP scale-out

Join New scale-out DQP merge join

Predicate Optimize in-memory/cached

processing

Eliminate intermediate

result bottlenecksImprove DQP

scale-out

SubqueryNew query

operators for improved

efficiency and scale-out


SAP SYBASE IQ 16LDAP AUTHENTICATION

Value proposition: Reduced TCO and improved securityEnable customers to hook into existing enterprise infrastructures for managing users and passwordsEnable central management of password complexity policiesMultiple domains and multiple LDAP servers

Architectural considerations:Secure communication with LDAP server using TLSDeployable across various vendor’s Directory Service that supports Lightweight Directory Access Protocol (LDAP)Support 24x7 operation: Automatic failover and failbackEfficient design for frequent, short-lived connectionsNo client side changes neededSQL Anywhere, Sybase IQ, and ASE can share common user repository

WebServer


SAP SYBASE IQ 16ROLE BASED ACCESS CONTROL

Value PropositionSupport separation of duties and principle of least privilege

Breakdown privileged operations into fine grained sets that can be individually granted

Control over propagation of privilegesWho can grant which privileges

Complete backwards compatibility and clean migration

Stay competitive

Architectural ConsiderationsSupport ANSI SQL role semantics, system defined roles and user defined roles

Minimum number of role administrators

Grantable system privileges for privileged database operationsSecure system stored procedures with SQL SECURITY INVOKER

Minimize performance impact by adding connection level cache mechanism

Same in-memory representation of RBAC structures: upgraded and non-upgraded DBsRestrict impersonation through SET USER to adhere to RBAC model


SAP SYBASE IQ 16SCALE OUT CLUSTER (MULTIPLEX) ENHANCEMENTS

• Shared System Temp- Reduces the size of local temporary store- Simplifies sizing requirements for temp stores- Opens up support for DQP on temp tables in future

• Logical Server – Login redirection- Zero changes on client side on dynamic changes to logical servers

- Single point to connection “redirector”- HA with multiple servers in the connection string to prevent a single point of failure

• Cache Ejection Policy improvements for better cache hits- Better infrastructure to track on disk changes and maximize cache hits to increase performance

• Global Transaction Resiliency- Suspend/resume global DML transactions, with a timeout, during INC disconnect/reconnect, coordinator downtime


SAP SYBASE IQ 16SCALE OUT CLUSTER (MULTIPLEX) – GBL TRANSACTION RESILIENCY

Offline Coordinator

Writer1

Heartbeat

Client Reconnect

Reader2

Global R/W transactionssuspend untilINC connectionsreestablishedor timed out

Global R/W INC Connectionsautomatically reconnect and resume transactionafter a coordinator failover. In most cases long-running loads will transparently resume.

Storage

Failover node restarts as Coordinator


Summary

• Market-Leading product with tremendous momentum

• 96%+ customer satisfaction rates

• Pioneering Column-store with 10+ patents

• World record holder in data loading

• SAP Sybase IQ is used by twice as many companies as

the next leading pure column store DBMS provider

• Focused sales and support teams

• SAP commitment to product leadership

SAP Sybase IQ in the Context of RTDP


Requirements for Real-Time Data-Driven ApplicationsDefining a Big Data-enabled Logical Data Warehouse

Business Applications

Endlessly expanding repository for all kinds of data

Deep reference and trend analysis

Instant classification of streaming data

Real-time processing, correlation, transactions, and analysis

Data movement, cleansing, and placement throughout the architecture

Business Events

Metadata Management, MDM, and Governance


Enabling The Logical Data Warehouse With RTDPEngineered to Work Together – even with Big Data

HANA

IQ

Business Applications

ESP fast driver for HANA + integration of ESP and Hana Studios

Real-time Replication + ETL to HANA and IQ

SQOOP to HANA and IQ from/to Hadoop

Data Virtualization: SAP Sybase IQ to Hadoop to HANAQuery Integration: IQ to Hadoop

SAP BW

Tightly integrated EDW management with SAP HANA and SAP Sybase IQ

Business Events

Data Services

ESP

High Performance data processing with tight integration with SAP BW, SAP Sybase IQ, ESP and Data Services


Isn’t it time to rethink your EDW strategy?Benefits of a complete Big Data Architecture

Accelerate AnalyzeAcquireEffective, Rapidly and Efficiently acquire and consolidate massive amounts of diverse and arbitrary data

Real-time response time regardless of data volume or data location manage and integrate massive volumes of data

Achieve real results to get the insights you need with a variety of means alone or in combination


Empowering you to Deliver Data Management for the New Real-time World

Open APIs and Protocols

SAP Real-time Data Platform

Data Services Information ServicesCommon Services

TransactionReal-time

EDW

Integration Quality

MDM ILM

Mobile

Stream

Model

Manage

Metadata

Engineered to Work Together

Custom Mobile Analytics/BI Cloud Big Data

Thank you


Offer from Dobler Consulting

For Qualified IQ installed base customers we will provide 4 hours of health check / upgrade consulting

For Qualified New Customers we will provide 4 days of design and optimization consulting

Please contact Allan Roeder at Dobler Consulting for more details.

Call (813) 322 3240 or email [email protected]


Q & A

Please submit your questions through the questions panel on the webinar control menu.

For more information on SAP Sybase IQCall 813 322 3240 or email: [email protected]

SAP IQ 16 Product Annoucement

Software

Transcript of SAP IQ 16 Product Annoucement