New Functions and Capabilitiesdugi.molaro.be/wp-content/uploads/2016/04/Novità... · •...

© 2016 IBM Corporation

Accelerator V5 & Accelerator Loader V2

New Functions and Capabilities

Francesco Borrello

Technical Sales and Solutions

Milan, 19th April 2016Rome, 20th April 2016

Dugi2016

© 2016 IBM Corporation2

Legal Disclaimer

• © IBM Corporation 2016. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify

the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any

kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are

subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or

otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall

have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms

and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all

countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at

any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment

to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the

effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other

results.


Agenda

� Accelerator-only tables

• Introduced as foundation in V4.1 PTF 5 for DB2 for z/OS 10 only

• Now available for DB2 for z/OS 10 and DB2 for z/OS 11

� Analytics Server, SPSS, INZA

� Incremental Update Enhancements

� Temporal Tables

� Encryption of Data-in-motion

� New IBM DB2 Analytics Accelerator Loader V2.1

�Miscellaneous


Agenda






� Temporal Tables



�Miscellaneous


Introducing Accelerator-only table type in DB2 for z/OSCreation (DDL) and access remains through DB2 for z/OS in all cases

Non-accelerator DB2 table

• Data in DB2 only

Accelerator-shadow table

• Data in DB2 and the Accelerator

Accelerator-archived table / partition

• Empty read-only partition in DB2

• Partition data is in Accelerator only

Accelerator-only table (AOT)

• “Proxy table” in DB2

• Data is in Accelerator only

Table 1

Table 4

Table 3

Table 2Table 2

Table 4

Table 3


In-database transformation and multi-step processing

� DB2 Analytics Accelerator introduces Accelerator-Only tables to store

intermediate or final results of data transformation or multi-step processes

� Accelerator-Only tables enable complex data transformations taking place on DB2

Analytics Accelerator, called “In-database transformation”

� Accelerator-Only tables enable intermediate and final results of reporting or other

processes to be generated and stored on DB2 Analytics Accelerator, called

“In-database multi-step processing”

� Advantages:

� Accelerate in-database data transformations and data movement processes

� Reduced need of data movement processes to other platforms for data transformation

purposes

� Enables multi-step reporting on the Accelerator

� Saves disk space and CPU cost on z Systems currently used for transformations and

reporting steps

� Allow data preparation steps for data mining and other advanced analytics to execute

on the Accelerator

� Store a set of data in DB2 Analytics Accelerator only, not on DB2 for z/OS, without

using the High Performance Storage Saver functionality


Accelerator-only tables – Technical basics

� AOTs are created and dropped using DB2 DDL statements (CREATE; DROP)

• Accelerator must be started

• QUERY ACCELERATION behavior may have any value during CREATE/DROP

• Syntax:

� CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1>;

� DROP TABLE MYTABLE;

� Recommended to create a database in DB2 to be used for the AOTs

• CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1> IN

DATABASE MYDB;

• Usual authorization necessary to create objects in database

� SELECT and INSERT/UPDATE/DELETE operations using AOTs can only run on the

Accelerator

• QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL

• Accelerator-shadow tables, Accelerator-archived tables and other AOTs can be used in

the same statement

� Dynamic and static SQL can be used with AOTs


Accelerator-only tables – Limitations

� Column type and SQL query restrictions of accelerated tables apply to AOTs, too

� An AOT is created in one Accelerator only, no HA or workload balancing support

� An AOT cannot be selected for archiving

� An AOT cannot be enabled for Incremental Update

� INSERT INTO AOT (SELECT FROM db2-table) is not supported

� The source table in the SELECT must be an accelerated table or another AOT

� INSERT: only single-row or insert from sub-select

� UPDATE: not possible to update values in distribution key columns

� No automatic Backup/Recovery mechanisms

� 2 phase commit is not supported on AOTs. Multiple I/U/D statements in one commit

scope are only possible if all statements target AOTs on the same Accelerator


Accelerator-only tables – Data Studio

� How do AOTs appear in Data Studio?

� Different icon in front of the table to distinct AOTs

� Less operations possible for AOTs (Load, Switch Acceleration, Storage Saver =)

� The ‘Last Load’ column currently shows “Operational”


Multi-step reporting applications with DB2 for z/OS BEFORE Accelerator-only tables: Report processing on DB2, source data might

reside on the Accelerator already

Temporary results

2

Temporary results

1Temporary results

Reports and Dashboards

n

Multi-Step Report

n

1

2

Reporting Application

Credit Card

Transaction History

Customer

Summary Mart

Credit Card

Transaction History

Customer

Summary Mart


Multi-step reporting applications with DB2 for z/OSWith Accelerator-only tables: Temporary objects and processing on the Accelerator

Temporary results

2Temporary results

1Temporary results

Reports and Dashboards

n

Multi-Step Report

n

1

2

Reporting Application

Credit Card

Transaction History

Customer

Summary Mart

Credit Card

Transaction History

Customer

Summary Mart

Data for transactional and analytical processing


In-database transformationUsing Accelerator-only tables and ELT logic in the Accelerator

Transaction Processing

Systems (OLTP)

Analytics

Advantages: • Simpler to manage

• Better performance and reduced latency Data for transactional and analytical processing

Customer

Transactions

Customer

Data

Customer Transaction Summary and History AOTs

Customer Summary

Mart AOTs

Customer

Transactions

Customer

Data

ELT logic


Data scientist work areaUsing Accelerator-only tables for ad-hoc analysis


Systems (OLTP)


Customer

Transactions

Customer

Data

Customer

Transactions

Customer

Data

Work database John

Work Area AOTs

Work database Bob

Work Area AOTs

Data Scientist John

Data Scientist Bob


Integrate more data sources for analyticsLoad external data to the Accelerator and save combined analytic results in

Accelerator-only tables


Systems (OLTP)

AnalyticsData for transactional and analytical processing

Customer

Transactions

Customer

Data

Combined result in AOTs

Customer

Transactions

Customer

Data

Related data from

other sourcesExternal

Files

Imagecopy

DB2 Analytics Accelerator Loader


Agenda






� Temporal Tables



�Miscellaneous


In-database analytics

Enable acceleration of predictive analytics applications

In-database analytics enables SPSS/Netezza

Analytics (INZA) data mining and in-database

modeling to be processed within the Accelerator

� Accelerates SPSS/Netezza Analytics (INZA) data mining and in-database

modeling

� Allows frequent model refreshes to enable adequate scoring

� Reduced need of data movement processes to other platforms for predictive

analytics purposes

� Supports the full lifecycle of a real-time analytics solution on a single,

integrated system, combining transactional data, historical data, and

predictive analytics


In-database analyticsData preparation (using AOTs) and SPSS modeling in the Accelerator

Transaction Processing Systems

(OLTP)

With embedded scoring

Advantages: • Allows fast model refreshes

• Ensures adequate scoring

• Better performance and reduced latency


Customer

Transactions

Customer

Data

Customer Txn

Data Prep AOTs

Customer

Transactions

Customer

Data

Modeling

ModelModelSPSS Modeler


SPSS Modeler Integration


Agenda






� Temporal Tables



�Miscellaneous


Incremental update improvements in 2015

• V4.1 PTF 5 (May 2015)

• Suspending faulty tables from replication

• Improved incremental update performance

– Parallel Apply

– Automatic definition of distribution and organization keys

• V4.1 PTF 6 (4Q 2015)

• Continuous operation of replication

• V5.1 (4Q 2015)

• Disable query acceleration for suspended faulty tables


DB2 Log

Data

Table T1Table T1

Table T2

Table T3

I/U/D Statements

The subscription with three

defined tables is started and the

tables are replicated

Data

T1 T2 T3

Subscription

Capture

Agent

Replication

Engine

Table T2

Table T3

Isolation of Tables in Error: Initial Situation


Isolation of Tables in Error: Error Situation

DB2 Log

Data

Table T1Table T1

Table T2

Table T3

I/U/D Statements

After the error occurred the subscription is stopped and automatically restarted. The faulty table is suspended and

the remaining tables are replicated.

Data

T1 T2 T3

Subscription

Capture

Agent

Replication

Engine

Table T2

Table T3

Capture Agent Error.e.g. decompression

error

Apply Error.e.g. invalid UTF-8

character

1 2

X

An error can occur on

either capture or

apply site1

2


Without Parallel Apply

DB2 Log Data

<T1; I>

<T2; D>

<T3; U>

<T4; I>

<T1; I>

<T2; I>

Table T1

Table T2

Table T3

Table T4

T1

<T1; I>

<T1; I>

T2

T3

T4

<T2; D>

<T2; I>

<T3; D>

<T3; I>

<T4; I>

I/U/D images per TableTable T1

Table T2

Table T3

Table T4

Build Apply

I/U/D Statements

The Build-Thread takes

arriving log data and creates

Insert/Update/Delete data

images per table

Every minute one Apply-

Thread sequentially applies

the data images to the tables.

Table by table...

Log data sent by Capture

Agent and received by

Replication Engine

se

qu

en

tial


With Parallel Apply

Table T1

Table T2

Table T3

Table T4

T1

<T1; I>

<T1; I>

T2

T3

T4

<T2; D>

<T2; I>

<T3; D>

<T3; I>

<T4; I>

I/U/D images per TableTable T1

Table T2

Table T3

Table T4

Build Apply

Apply

Apply

Apply

I/U/D Statements

Four Build-Threads take

arriving log data and create

Insert/Update/Delete data

images per table. Better

performance....

Every minute four Apply-

Threads apply the data

images to the tables. Much

faster....

Build

Build

Build

DB2 Log Data

<T1; I>

<T2; D>

<T3; U>

<T4; I>

<T1; I>

<T2; I>

Log data sent by Capture

Agent and received by

Replication Engine

4x

pa

ralle

l


Continuous Replication: Adding/Removing/Loading Tables (I)

25

T1

T2

T3

T4

Time

Replication

runningLoad/Reload of T1

Replication continues

running for all tables

Replication

stops for all

tables,

potentially for

hours

T1

T2

T3

T4

Time

Replication

runningLoad/Reload of T1

Replication continues

running also for T1

Replication stops for all tables, but only for some seconds

V3

V4


Continuous Replication: Adding/Removing/Loading Tables (II)

� Since v4.1.5 (beta), 4.1.6 (GA): no impact to other tables when adding, removing, loading tables

− T1 is re-loaded --> no impact on the replication progress of T3 and T4

− T2 is added to IDAA, loaded and replicated --> no impact on replication for T3 and T4

− T4 is removed from IDAA --> no impact on T1, T2 and T3

26

T1

T2

T3

T4

Time

Reload of T1

Add+Initial Load of T2

T4 is removed

from IDAA

Note: this feature also improves replication throughput significantly.

Stop+restart are quite expensive operations.

Note: this feature also improves replication throughput significantly.

Stop+restart are quite expensive operations.


Continuous Replication: No locks on DB2 tables during load

� The Accelerator will allow to load (or reload) a table without taking a S-lock on the source

table

− Introduced new ACCEL_LOAD_TABLES lock-mode “ROW” for

replicated tables

• UNLOAD with SHRLEVEL CHANGE ISOLATION CS

• SKIP LOCKED DATA will not be used (lock-mode NONE only)

� In-flight changes will be kept on per-table spill queues on the appliance

− Spill queues are in target memory, backed by disk

27


Disable query acceleration for suspended faulty tables

� Available since V4.1 PTF5: “Suspend faulty table”

• Replication of tables may be suspended & mirroring continuous w/out table

• By default: “Suspend Faulty Tables” is on

• Issue: queries still accelerated while faulty table is not replicated anymore

� New capability

• Automatically disable query acceleration for faulty tables

• Generate DSNX message: notify query acceleration is disabled for table

• Configuration

• Off per default

(to avoid breaking existing

automation)

• Turn on in configuration

console per subsystem /

database system


Agenda






� Temporal Tables



�Miscellaneous


Temporal Tables Introduction

� Concept: DB2 supports time-based data management that allows you to insert, update,

delete, and query data in the past, the present, and future while keeping a complete history of

what you knew and when you knew it.

� 3 Types of temporal tables

• System-period temporal tables

• Where DB2 transparently keeps a history of old rows that have been updated or

deleted over time. With new constructs in the SQL language standard, users can go

back in time and query the database at any chosen point in the past.

• Application(Business)-period temporal tables

• Where applications supply dates or timestamps to describe the business validity of

their data. Already supported in DB2 Analytics Accelerator V4.1 PTF 5

• Bitemporal tables

• Manage system time and business time. Bitemporal tables combine all the capabilities

of system-period and application-period temporal tables. This combination enables

applications to manage the business validity of their data while DB2 keeps a full

history of any updates and deletes.


System-period Temporal Tables Introduction

� Two tables:

• Active table

• History table: contains the historial versions of the rows in the active table.

� Three columns

• Begin column(timestamp(12)): represents the time at which row data begins to be valid

• End column(timestamp(12)): represents the time at which row data is no longer valid

• Transaction ID column(timestamp(12)): record the transaction id that resulted in creating a row

of data.

CREATE TABLE policy_info

(policy_id CHAR(10) NOT NULL,

coverage INT NOT NULL,

sys_begin TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW BEGIN,

sys_end TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW END,

create_id TIMESTAMP(12) GENERATED ALWAYS AS TRANSACTION START ID,

PERIOD SYSTEM_TIME(sys_begin, sys_end));

CREATE TABLE hist_policy_info ( )...

ALTER TABLE policy_info

ADD VERSIONING USE HISTORY TABLE hist_policy_info;


Timestamp(12) support in DB2 Analytics Accelerator

Netezza NPS doesn't support timestamp(12).

Workaround: MAPPING timestamp precision 12 to timestamp precision 6 on the Accelerator.

• With DB2 Analytics Accelerator V5.1 tables with timestamp(12) columns can be added

and loaded to the Accelerator with the truncation of timestamp(12) to timestamp(6).

• Impact: Queries running on the Accelerator return different results than on DB2 for

z/OS for these columns

• Introduction of two new options 5 and 6 for ZPARM QUERY_ACCEL_OPTIONS to

control the truncation of timestamp precision > 6.

• Option 5: queries referencing system temporal tables are eligible.

• Option 6: queries referencing an expression or regular (non system temporal) table

columns with a TIMESTAMP(12) data type are eligible.

• These new options only affect DML(SELECT, INSERT, UPDATE, DELETE), no impact

on DDL

• Accelerator-only tables can be created with timestamp(12) columns regardless of the

QUERY_ACCEL_OPTION 6 setting.


Agenda






� Temporal Tables



�Miscellaneous


Encryption of data 5

� Encryption of data “at rest”

� Already available by default with N3001 self-encrypting disks (SEDs)

� Addresses risk of stolen disks

� Encryption of “data-in-motion”

� Encrypt network traffic between z Systems and the Accelerator using

Internet Protocol Security (IPsec)

� Network traffic includes:

� DRDA traffic (e.g. queries and results, table loads)

� Configuration Console traffic (passwords)

� Incremental Update traffic

� Requires configuration and enablement on z Systems and on the

Accelerator

� Addresses network sniffing (interception of TCP/IP packets)


Encryption of data in DataStudio

Icon Description

Disks are not encrypted.

Network traffic is not encrypted

Disks are not encrypted.

Network traffic is encrypted for at least one z/OS system.

Disks are encrypted.

Network traffic is not encrypted.

Disks are encrypted.

Network traffic is encrypted for at least one z/OS system.

One of the certificates used for network encryption is expiring in 17 days.

or

One of the certificates used for network encryption has expired.

Click on details link

shows details about certificates and keys

Click on details link

shows details about certificates and keys


Agenda






� Temporal Tables



�Miscellaneous


IBM Insight Conference, Las Vegas, Oct 2013

� Announced the IBM DB2 Analytics Accelerator Loader for z/OS

− High speed, zIIP enabled accelerator loading with additional functions

� Addressed challenges in loading the Accelerator

− Elongated two-step process

− Fast load from data in a file to accelerator only

− Related set of tables put into read-only state

− Inability to load to desired point-in-time

− Plus

• zIIP-eligible data preparation to load data faster

• Load stand-alone backups directly to accelerator


Existing Accelerator Loader Customers

� Identified challenges in loading non-DB2 data− Complexity in converting non-relation data to relational

− Manual / ETL process

− Slow, due to intermediate write to disk


Loading non-DB2 Data Sources into Accelerator

Product Comparison

� IBM DB2 Analytics Accelerator Loader – The user must:

• Extract data from source (IMS, VSAM, Oracle, SMF, etc.)

• Convert extracted data to DB2 external load file format

• DataStage or other tooling can be used

• Create a DB2 table that matches format of extracted data

• Add newly created table to the Accelerator

• Construct a DB2 Load utility field specification that describes the input data

• Run Accelerator Loader batch job to load data to accelerator

� Accelerator Loader V2.1– Automates entire process:

• User builds a select statement from data source(s) (IMS, VSAM, SMF, Oracle, =)

• Automatically creates the DB2 table

• Automatically adds table to Accelerator

• Automatically extracts specified source data

• Automatically converts data to necessary DB2 format (in memory)

• Automatically loads data to Accelerator

• Automatically enables table for acceleration

Single batch job!


Other Accelerator Loader v2.1 Improvements

� LOAD RESUME support

− Avoid the need for full LOAD replace

− New function to append data from a file to existing data in a table

• Replaces the need for full data replace

− LOAD RESUME can provide significant CPU / elapsed time savings vs. REPLACE

− LOAD RESUME to both DB2 table and Accelerator table in parallel

− LOAD RESUME to Accelerator only and to Accelerator-only tables

� Accelerator-only tables (AOTs) – new in Accelerator V4.1 PTF5

− New table type in DB2 for z/OS

• Data resides only in Accelerator version of table, not in front-end DB2

− All queries targeting these tables are routed to Accelerator

• All SELECT statements automatically routed to Accelerator

• INSERT/UPDATE/DELETE also supported

• No logging or point in time recovery

− Without tooling, data must be inserted to load

• Insert directly to Accelerator-only table

• If data sourced from outside DB2, multi-step process to load


DB2 Analytics Accelerator Loader V2.1 Architecture

Access to non z/OS data sources

through DB2 LUW Federation Server


Agenda






� Temporal Tables



�Miscellaneous


New system monitoring with V5 – for free in Data Studio

!


System utilization


Current Tasks


Finished Tasks


Data distribution - Uneven in this example


Incremental Update Monitor


� DB2 now caches accelerated queries in the dynamic statement cache

� Improves performance of PREPARE statements for accelerated dynamic

queries

� New query preparation process:

− For a new arriving query DB2 searches the DSC whether the query has been

prepared before

− The ACCELERATED column value in the DSC defines what happened to the

query when it was prepared before

• Values: NO, YES, NEVER

− The query acceleration behavior together with the ACCELERATED column in

the DSC defines whether the query can be accelerated

• Before the query is accelerated DB2 checks whether the Accelerator is

active and whether all involved tables are enabled

− The DSC can have multiple entries for the same query depending on the

query acceleration behavior on when the statement was prepared previously

− If there is no matching record, the query is treated like a new one, goes

through full preparation and a new record is added to DSC

� DB2 APARs only, no additional accelerator PTFs required:

− DB2 10: PI22531 (PTF UI24359)

− DB2 11: PI28660 (PTF UI28976)

Dynamic Statement Cache support for accelerated queries


� New set of statement level monitoring counters

− Report SQL statement's eligibility to be routed to the Accelerator

• DSNZPARM or special register for Accelerator modeling must be enabled

− Report SQL statement’s execution statistics for queries that are routed to the

Accelerator

• Execution statistics include:

• Elapsed, execution and wait times

• CPU time

• Number of rows and bytes returned

� Exposed via IFCID 316 (dynamic SQL), IFCID 401 (static SQL) and in the

DSN_STATEMENT_CACHE_TABLE table

� DB2 APARs/PTFs, no additional accelerator PTFs required:

− DB2 10: PI23083/UI24713

− DB2 11: PI30005/UI29032

Dynamic Statement Cache support for accelerated queries


Dynamic Statement Cache support

Query acceleration behavior is specified by either the QUERY_ACCELERATION subsystem parameter, the

QUERYACCELERATION bind option, or the CURRENT QUERY ACCELERATION special register


Dynamic Statement Cache support


QMF for z/OS V11.2 enhancements to support AOTs

� New global variables

• DSQEC_SAV_ALLOWED – Controls whether users save data to a new table in the

database or in an Accelerator

• 0 – Disable Save Data

• 1 – Enable Save Data to database tables only

• 2 – Enable Save Data to AOT only

• 3 – Enable Save Data to either database or AOTs (database default)

• 4 – Enable Save Data to either database or AOTs (accelerator default)

• DSQEC_SAV_ACCELNM – Contains the default name of the Accelerator to be used

when creating AOTs from QMF commands (e.g. SAVE DATA)

� Supported in TSO Client only so far


QMF for z/OS V11.2 enhancements to support AOTs (Contd.)

Command with syntax enhancement Description

SAVE DATA AS tabname (ACCELERATOR

accelname

Saves data as accelerator-only table

tabname in accelerator accelname

SAVE DATA AS tabname (SPACE name Saves data as database table tabname in

database and table space specified by

name

IMPORT TABLE tabname FROM

datasetOrFile (ACCELERATOR accelname

Imports table data into accelerator-only

table tabname in accelerator accelname

IMPORT TABLE tabname FROM

datasetOrFile (SPACE name

Imports table data to database table

tabname in database and table space

specified by name

RUN QUERY qname (TABLE tabname

ACCELERATOR accelname

Runs a query and saves the result directly

into accelerator-only table tabname in

accelerator accelname

RUN QUERY qname (TABLE tabname

SPACE name

Runs a query and saves the result directly

into database table tabname in database

and table space specified by name

New syntax enhancement supported in TSO client only so far. For other clients create the AOTs

separately before using the SAVE DATA, RUN QUERY or IMPORT commands.


Dugi2016

New Functions and Capabilitiesdugi.molaro.be/wp-content/uploads/2016/04/Novità... · •...

Documents

Transcript of New Functions and Capabilitiesdugi.molaro.be/wp-content/uploads/2016/04/Novità... · •...