Post on 20-Jan-2016
© 2014 IBM Corporation
Simply Powerful – Informix 12.10.xC1 & xC2 - Recap
Scott PickettWW Informix Technical SalesFor questions about this presentation, email to: spickett@us.ibm.com
Agenda
New Packaging & Pricing Table Compression features Index Compression features New SQL Informix Warehouse Accelerator Time Series HA, ER, SDS, Flexible Grid, Connection Manager
© 2014 IBM Corporation2
Informix Packaging & Pricing Changes
© 2014 IBM Corporation
Informix Edition Comparison
* Limits are hard-coded
Components Developer Innovator-C Express WorkgroupAdvanced Workgroup Enterprise
Enterprise Hypervisor
Advanced Enterprise
Pricing Metric Free FreeAUSI, PVULUVS
AUSI, PVULU Socket
PVU AUSI, PVU PVU PVU
Processor/CPU VP Limit
1 core 1 core* 4 cores* 16 cores* 16 cores* Unlimited Unlimited Unlimited
Install Memory Limit 1 GB 2 GB* 8 GB* 16 GB* 16 GB* Unlimited Unlimited Unlimited
Install Data Limit 8 GB 8 GB* Unlimited Unlimited Unlimited Unlimited Unlimited Unlimited
Warehouse Accelerator Memory Limit
N/A N/A 48 GB N/A N/A Unlimited
Replication Limits Unlimited 2 root nodes*
Unlimited Unlimited Unlimited Unlimited Unlimited
HDR/SDS/RSS Limits Unlimited 1 secondary*
2 secondary*
2 secondary*
Unlimited Unlimited Unlimited
Other Limits Parallel Operations*,Distributed Operations,Private Memory Cache for VP,HPL, Compression*,Partitioning*
Parallel Operations*,Distributed Operations,Private Memory Cache for VP,HPL, Compression*,Partitioning*
Parallel Operations*,Distributed Operations,Private Memory Cache for VP,HPL, Compression*,Partitioning*
Parallel Operations*,Distributed Operations,Private Memory Cache for VP,HPL, Compression*,Partitioning*
Platforms
*Note:UNIX is AIX, HP-UX, Solaris
Linux*UNIXWindowsMac-OS
Linux*UNIXWindowsMac-OS
Linux*UNIXWindowsMac-OS
Linux*UNIXWindowsMac-OS
Linux*UNIX
Linux*UNIXWindows Mac-OS
Linux RHELAIX Power
Linux*UNIX
IWA Platform 64-bit Linux 64-bit Linux
© 2014 IBM Corporation4
Informix Product Features by Edition
Features Developer Innovator-C Express WorkgroupAdvanced
Workgroup EnterpriseEnterprise Hypervisor
Advanced Enterprise
Parallel Operations
Data Partitioning
Private Memory Cache VP
High Performance Loader
*Table, Index and Blob Compression Option Option
Distributed Operations
Enterprise Replication
High Availability Clusters
Shared Disk Secondary
Remote Stand-alone
Updatable Secondary
Informix Warehouse Accelerator
TimeSeries & Spatial Capabilities
Continuous Log Restore
SQL/Instance Administration
Open Administration Tool
Label Based Access Control
Web Feature Services
Column-level Encryption
Virtual Table Interface
DBSpace Prioritization
Direct I/O – Cooked File
Point-in-time Table Restore
Primary Storage Manager
Auto Statistics for Index Build
Last Commit Isolation
Multiple Triggers
Recovery Time Policy
© 2014 IBM Corporation5
Bundle Additions
SPSS and Cognos– Includes bundling SPSS Modeler V15.0 FP 1 Limited– Includes bundling SPSS Statistics V16.0 1 Limited– Includes bundling 5 Cognos BI 10.2 licenses– Strengthens BI + Predictive Analytics value proposition
• SPSS will be included in any new Informix Advanced Enterprise Edition• Cognos BI will be included for Advanced Growth and Advanced Enterprise Editions
IBM Mobile DB & IBM Mobile Sync – Includes bundling IBM Mobile Database 7.0.1– Includes bundling IBM Mobile Database Sync 1.0– Strengthens Mobile value proposition
• Mobile DB and Sync will be included in all for fee Informix Editions• Help propagate development of mobile applications on Informix and for use with
mobile analytics and BI applications• These were added to 12.10.xC1 and xC2, discontinued permanently starting in xC3.
© 2014 IBM Corporation6
Auto Compression
© 2014 IBM Corporation
Text/Byte Compression
Compression of in-partition data stored in Partition Blobs.
– Partition Blobs are simple large objects (mostly text and byte data types) in which the data is stored outside the row, but in same partition of the same dbspace as the row data.
• So even though the blob data is stored on a separate page from the row data, the partition blob page still belongs to the same partition.
© 2014 IBM Corporation8
Auto Compression - Overview
Create a compression dictionary for the data rows of a table automatically when it has the required numbers of rows.
Benefits of automatic compression:– Informix compresses data rows as the data is loaded.– Informix sets compression as a property of the table, so any new fragments
added also get compressed automatically.– You use current SQL admin API commands to set a table or fragment for auto
compression.– You have an SQL interface to create a compressed table.
© 2014 IBM Corporation9
Auto Compression Details
New data will be loaded as compressed.
A minimum of 2000 rows are needed for the compression dictionary to be created.
You can start compression with the same SQL admin API commands that you currently use.
You can also compress a table, using SQL syntax when you create the table.
© 2014 IBM Corporation10
Auto Compression How To
Using Admin API
– Execute function task(“table compress”, “my_table”, “my_database”);– Execute function task(“fragment compress”, “my_fragid”);
– Even if there are not enough rows to sample and create a compression dictionary, the SQL admin commands will succeed indicating “Auto compression is set”.
© 2014 IBM Corporation11
Auto Compression - How to
Using SQL
– Create table my_table on (my_col ….) compressed;
– After the table is created, as data is loaded into the table, a compression dictionary will be created when 2000 rows are inserted.
© 2014 IBM Corporation12
Auto Compression
When a table load uses light append, a dictionary will be automatically created: – The rows that loaded before the dictionary was created will be compressed.
During a normal insert, the new rows inserted will be compressed:– Data rows already in the table (before the compress) will not be compressed.
Auto compression is not supported for indexes and blobs.
© 2014 IBM Corporation13
Questions
© 2014 IBM Corporation14
Index Compression
© 2014 IBM Corporation
Leaf LeafLeafLeafLeafLeafLeafLeaf LeafLeaf
TwigTwigTwigTwig
Node Node
Root
Basic BTree
© 2014 IBM Corporation16
create index ix on t1 (last_name, first_name, zipcode)
Key RidList
Smith George 12345Smith Harold 12345Smith Harold 33445
0x400004, 0x256,D0x400004, 0x257,D0x500008, 0x256,D
Key RidList
Token, George, TokenToken , Token, TokenToken , Token, 33445
0x400004, 0x256,D0x400004, 0x257,D0x500008, 0x256,D
Keys get compressed tomake what gets storedlook something like this.
© 2014 IBM Corporation17
Index Compression Restrictions
Normal detached BTree indexes only Native types No functional indexes Only the leaf level is compressed Can be done as part of index creation or to existing indexes while
online Must contain at least 2000 unique key values Full HDR support
– Log the uncompressed key Uncompress not available at this time for indexes
© 2014 IBM Corporation18
SQL Syntax
CREATE INDEX idxname ON tabname (column_name1, …) COMPRESSEDEXTENT SIZE # NEXT SIZE #;
EXECUTE FUNCTION sysadmin:task (‘index compress’, ‘index_name’, ‘database’);
© 2014 IBM Corporation19
Index Compression Benefits
More index data fits into the bufferpool– Reduced I/O
Unlike Oracle, SQL Server, and even DB2, Informix index compression supports composite indexes with either “prefix” and “suffix” keys– Prefix – unique value is last value of composite
• Ex) Red, Honda, Civic, Oregon, VIN #– Suffix – unique value is first value of composite (key only scan)
• Ex) VIN #, Red, Honda, Civic, Oregon
© 2014 IBM Corporation20
Indexing Varchars vs Indexing Compressed Chars
Varchars are used to save space, in both the data row and the index, by storing only the relevant data and not padding with white space
We did an experiment to compare key-only scan performance of indexed varchars against indexed compressed chars
© 2014 IBM Corporation21
Varchars vs Compressed Chars
create table people (first_name varchar(64),middle_name varchar(64),last_name varchar(64),address varchar(255),city varchar(64),state char(2),sex varchar(6),ssn char(11),eye_color varchar(24),ancestry varchar(64));
© 2014 IBM Corporation22
Varchars vs Compressed Chars
select count(*) from people where last_name in ('SMITH', 'JOHNSON', 'WILLIAMS', 'BROWN', 'JONES') AND first_name in ('JAMES', 'JOHN', 'ROBERT', 'MARY', 'PATRICIA', 'LINDA');
Key only scan
create unique index ssn_people on people(ssn);
create index ix1_people on people(last_name, first_name, middle_name, ssn);
© 2014 IBM Corporation23
Query Results (Several Million Rows)Varchar Uncompressed
Char% better Compressed Char % better
Unbuffered 100s 5.38s 1758% 1.9s 183%
Buffered 6.4s 0.77s 731% 0.77s 0%
# of index nodes 514748 2857165 634095
© 2014 IBM Corporation24
Why Is Performance Better?
Varchars require IDS to read the data row even if it appears that a key only scan could be performed– Have to compare white space
• White space stored in row• White space removed from index key
© 2014 IBM Corporation25
Varchar query plan
© 2014 IBM Corporation26
Char query plan
© 2014 IBM Corporation27
Varchar index (oncheck –pT)
© 2014 IBM Corporation28
Char index (oncheck –pT)
© 2014 IBM Corporation29
Char index compressed (oncheck –pT)
© 2014 IBM Corporation30
Results
So to look at the earlier slide again:
In both cases, the queries were faster on the compressed character column. Results were not so good on the uncompressed varchar column, due to the byte by byte comparison earlier mentioned.
Index usage: there were almost 120,000 more index nodes on the character (634095) than there were on the varchar (514748).
Here a compressed character column index, loses almost 78% of index nodes (634095 vs uncompressed char 2857165); and it is faster performance wise.
© 2014 IBM Corporation31
Index Compression
Compress an existing index:– execute function task(“index compress”, “j”, “testdb”);
Repack and shrink an existing index:– execute function task(“index repack shrink”, “j”, “testdb”);
Estimate Compression on an index:– execute function task(“index estimate_compression”,“176_576”,
“stores_demo”, “ifxjson”);– You can estimate compression only for a detached B-tree index on a
fragmented or non-fragmented table.
Pushes all nodes forward on the same page as they are found and compresses them, leaving free space at the end.
Repack consolidates partially filled index pages and fills them, leaving empty pages behind; repack if you intend to shrink.
Shrink reduces an index overall size by returning empty pages to its dbspace; we will not shrink beyond the index first extent size.
© 2014 IBM Corporation32
Some index repack and shrink results, before and after. An Index before compression, repack and shrink, 20480 pages,
oncheck –Pt output.
After: 2778 Total Pages Allocated, 86.43 % reduction 15.57 % of index Space remains.
© 2014 IBM Corporation33
Before:
After:
New SQL
© 2014 IBM Corporation
Minus, Plus & Intersect Support - Functionality
New Extensions to the existing UNION/UNION ALL SET operation.
Same rules of UNION also applies e.g.
Both query block should have exact same number of columns.
Comparable data types.
Projection clause will not have BYTE or TEXT.
© 2014 IBM Corporation35
Minus, Plus & Intersect Support - Functionality
Order by should be at the end.
Precedence will be from left to right, unless they are grouped using parentheses.
There are existing restrictions for UNION and they will apply to these operators too.
© 2014 IBM Corporation36
Intersect and Minus/Except
Intersect returns only those rows which are returned by both queries and results will be distinct.
Minus/Except will returns distinct rows from the left query which are not found in the right side query.
The All keyword is not available for the intersect and minus/except operations.
NULL friendly, means when comparing NULL to NULL they are considered equal.
© 2014 IBM Corporation37
Minus, Plus & Intersect Support - Examples
create table t1 (col1 int); insert into t1 values (1); insert into t1 values (2); insert into t1 values (2); insert into t1 values (2); insert into t1 values (3); insert into t1 values (4); insert into t1 values (4); insert into t1 values (NULL); insert into t1 values (NULL); insert into t1 values (NULL);
create table t2 (col1 int); insert into t2 values (1); insert into t2 values (3); insert into t2 values (4); insert into t2 values (4); insert into t2 values (NULL);
© 2014 IBM Corporation38
Minus, Plus & Intersect Support - Examples (cont’d)
Select * from t1 intersect select * from t2;– Returns NULL, 1,3 and 4
Select * from t1 minus/except select * from t2;– Returns 2
© 2014 IBM Corporation39
Minus, Plus & Intersect Support - Rewrite
Select col1, col2 from t1 intersect select col1, col2 from t2;
Rewrite:
Select distinct * from (select col1, col2 from t1) t11 where exist (select col1, col2 from (select * from t2) t22 where
t11.col1 = t22.col1 and t11.col2 = t22.col2 );
© 2014 IBM Corporation40
Minus, Plus & Intersect Support - Minus/Except Rewrite
select col1, col2 from t1 minus select col1, col2 from t2;
select distinct * from (select col1, col2 from t1) t11 where not exists (select * from (select col1, col2 from t2) t22 where t11.col1 = t22.col1 and t11.col2 and t11.col2= t22.col2);
© 2014 IBM Corporation41
Select ….. INTO TABLE
You can now create persistent tables in IDS that can be accessed across sessions using the result set of a SELECT statement.
Combines the functionality of CREATE TABLE and INSERT into a single statement.
Supports all the Storage options of CREATE TABLE on IDS.
You can now change the EXTENT SIZE and NEXT SIZE of an existing table without having to unload the table first.
Allows you to create new schema from existing tables and populate it with only a subset of data for testing purposes.
Supported in a Flexible Grid environment.
Functionality supported via two different syntax enhancements:
– SELECT col1, col2 FROM sourcetab INTO newtab;– CREATE TABLE newtab AS SELECT col1, col2 FROM sourcetab;
© 2014 IBM Corporation42
4GB Stored Procedure Language
The maximum length of SQL statements and SPL routines is now 4 GB– The actual limit may depend on the available memory on the system.
Valid for Client SDK 3.70.xC5, JDBC 3.70.xC5 or higher.
Transient extended types, collection types are limited to 65535
sysadmin task command length is limited to 2020 (slot table has 28 bytes required).
Perhaps writing SQL or SPL possibly so large should be avoided for programmer maintenance issues …..
© 2014 IBM Corporation43
CASE Statement in SPL
CASE statement without else clause CASE mod(r,314) WHEN 1 THEN select tabid into r from informix.systables where tabid = 1; WHEN 313 THEN LET r = r + 4; WHEN 3 THEN LET r = r + 3;END CASE;
CASE statement with else clause
CASE r WHEN 317 THEN select tabid into r from informix.systables where tabid = 1; WHEN 312 THEN LET r = r + 4; WHEN 3 THEN LET r = r + 3; ELSE LET r = 1;END CASE;
© 2014 IBM Corporation44
New Mathematical Functions – Oracle Compatibility
SIGN(num)– Returns the sign of the input numeric value (returns integer):
• -1 for n < 0, 0 for n = 0, and 1 for n > 0 COSH(num)
– Returns hyperbolic cosine (returns float) SINH(num)
– Returns hyperbolic sine (returns float) TANH(num)
– Returns hyperbolic tangent (returns float) ACOSH(num)
– Returns arc hyperbolic cosine (returns float) ASINH(num)
– Returns arc hyperbolic sine (returns float) ATANH(num)
– Returns arc hyperbolic tangent (returns float) LN(num)
– Alias for existing LOGN(num) function (returns float)
© 2014 IBM Corporation45
Multiple distinct Operators Example
– Legal syntax• Applied to each predicate
– Illegal syntax • Nested operators
select avg(unique col1), count (distinct col2) from my_tab;
select distinct avg(distinct col1) from my_tab;
© 2014 IBM Corporation46
SQL Enhancements – nulls first / last Example
Default null sort order with the asc keyword
Modified null sort order with the asc nulls last keywords
© 2014 IBM Corporation47
SQL Enhancements – nulls first / last Example
Default null sort order with the desc keyword
Modified null sort order with the desc nulls first keywords
© 2014 IBM Corporation48
SQL Enhancements – Windows XML Enhancement
The XML4C and XSLT4C libraries have been ported to Windows 64 enabling XML functionality on that port.
© 2014 IBM Corporation49
Informix Genero v2.50 - Features
Attachable dialogs & forms UTF-8 support GAS/GWC
– MS Silverlight support for native Genero constructs– HTML5 theme improvements:
• Compatible with GDC (splitters, tabbed MDI, etc.)• Strong dependency on HTML5 standard• Desktop integration, file management, clipboard• Frontcalls to hardware: cameras, etc.• Windows 8 Metro tablets• Windows Phone 8 native browser support
SSO – Security– Extensible framework based on web services:
• Authentication• Authorization (OpenID, SAML)
Web services – WS Security
© 2014 IBM Corporation50
51
Primary Storage Manager (PSM)Faster and more functional than ISM
Easy way to configure OnBar to take backups and restores Easy way for embedded solutions to take parallel backups
using OnBar Easy way for Ontape customers to migrate to OnBar Replaces ISM
IS M P S M
N um be r o f P ro gra m F ile s 46 2
S ize o f p ro g ra m s s h ippe d w ith ID S 25 M B 2 M B
N um be r o f da e m o n pro gra m s 9 0
S ta rtup S te ps 40 2
M a x im um N um be r o f pa ra lle l s tre a m s
4 U n lim ite d
M a x im um tra ns fe r buff e r s ize 64K 2G B
In te gra te d w ith D a ta ba s e S c he du le r N o Y e s
© 2014 IBM Corporation
Informix Warehouse Accelerator
© 2014 IBM Corporation
Informix Warehouse Accelerator - Cluster Environment IWA now has the ability to create an accelerator server across
multiple physical servers as a cluster– An MPP implementation– Enables the creation of significantly larger accelerators and marts
• No longer limited by the amount of memory a single server can support
Informix Warehouse Accelerator
SQL
Results
© 2014 IBM Corporation53
TC
P/IP
Informix Warehouse Accelerator
Informix: Routes SQL queries to accelerator User need not change SQL or apps. Can always run query in Informix,
e.g., if– too short an est. execution time
Bulk Loader
SQL Queries (from apps)
Informix Warehouse Accelerator
Compressed DB partition
QueryProcessor
Data Warehouse
Informix SQL
(via DRDA)
Query Router
Informix Warehouse Accelerator: Connects to Informix via TCP/IP & DRDA Analyzes, compresses, and loads
Copy of (portion of) warehouse Processes routed SQL query and
returns answer to Informix
Results
© 2014 IBM Corporation54
Informix Warehouse Accelerator
Partition Update: Example
© 2014 IBM Corporation55
Refreshing Data on IWA – Multiple Options
Methods Frequency of Deployment Considerations
Complete Load Nightly or a small number of times per day
For initial load and for periodic complete refresh of mart
Automatic or manual Partition Update
Hourly, nightly or on-demand
For partitioned FACT table
For non-partitioned or partitioned dimension tables
Trickle Feed Continuous For Inserts (Append) to FACT table and I/U/D for dimension tables
Complete reload periodically following Trickle Feed
© 2014 IBM Corporation56
Informix Warehouse Accelerator
Partition Update: Refresh data in a data mart at partition level– Provides efficient refresh of data mart data
– Enables fast refresh of fact tables which are usually very large• Dimension table (usually very small) full load refresh is very fast
– Refresh can be done on• The entire table or
• The partition (or subset) that contains data updates
© 2014 IBM Corporation57
Informix Warehouse Accelerator – Trickle Feed The source database can now be configured to push new data into
the marts every N seconds without any further administrative intervention– Inserts only, deleted / modified fact data is not pushed into the mart
• See next slide for more details– Is controlled by the ifx_setupTrickleFeed() function
• Available in the sysadmin database– In an H/A cluster, this function can only be executed on the cluster primary
Syntax
– Whereaccel_name – the name of the accelerator which supports the martmart_name – the name of the data martbuffer_time – the number of seconds between refreshes
1 – 8639999 (2,400 hours) – refresh both the fact and dimension tables-1 – -8639999 (2,400 hours) – refresh just the fact table
To use this feature, MAX_PDQPRIORITY in the source instance must be set >= 50 for the initial data load and all subsequent loads
ifx_setupTrickleFeed(‘accel_name’, ‘mart_name’, buffer_time )
© 2014 IBM Corporation58
Informix Warehouse Accelerator
New SQL Support for query acceleration– Several SQL functions and expressions now support query acceleration.
– Example:• COUNT (DISTINCT CASE …)
• Multiple DISTINCT with aggregates, such as COUNT (DISTINCT…), in a query
• LEN / LENGTH
• SUBSTR / SUBSTRING
• TODAY
• TRIM
• YEAR
• UNITS
• UNION, UNION ALL• RANK, DENSE_RANK, DENSERANK, CUME_DIST, PERCENT_RANK, NTILE,
ROW_NUMBER, ROWNUMBER, FIRST, LAST, RATIO_TO_REPORT, RATIOTOREPORT
© 2014 IBM Corporation59
Informix Warehouse Accelerator
Additional platform support with Solaris Intel/AMD
Client
Informix database server
Optimizer
Use Accelerator
?
SQL query
Result set
No
Informix Warehouse Accelerator
Yes
Linux Intel x86 64-bit
Data Warehouse or mixed OLTP/Warehouse system
TCP/IP
AIX 64-bitHP IA 64-bit Solaris SPARC 64-bitLinux Intel x86 64-bitSolaris x86 64-bit (new support)
W Accelerator supported on Linux Intel/AMD 64-bit.
W Informix Server for the accelerator supported on the 64-bit Aix, HP IA, Linux Intel x86, Solaris Sparc and Solaris x86.
W Adds more cross platform integration.
© 2014 IBM Corporation60
Informix Warehouse Accelerator
Using the accelerator with secondary servers: Example
© 2014 IBM Corporation61
New OLAP Functions
Ranking functions– RANK, DENSE_RANK, DENSERANK– PERCENT_RANK, CUME_DIST, NTILE
Numbering functions– ROW_NUMBER, ROWNUMBER
Aggregate functions– SUM, COUNT, AVG, MIN, MAX– STDEV, VARIANCE, RANGE
FIRST_VALUE/LAST_VALUE functions– FIRST_VALUE, LAST_VALUE
LEAD/LAG functions– LEAD, LAG
RATIO_TO_REPORT function– RATIO_TO_REPORT, RATIOTOREPORT
All of these are accelerated by IWA
© 2014 IBM Corporation62
Example - Ranking functions
Order by clause is required Window clause is not allowed Duplicate value handling is different between rank() and
dense_rank()– rank() - assigns the next relative ordinal to the next result set value– dense_rank() - assigns the next absolute ordinal to the next result set value
(alias is denserank())
© 2014 IBM Corporation63
Rank Example
select emp_num, sales, rank() over (order by sales) as rnk, dense_rank() over (order by sales) as dense_rnk from sales;
emp_num sales rnk dense_rnk
101 2,000 1 1 102 2,400 2 2 103 2,400 2 2 104 2,500 4 3 105 2,500 4 3 106 2,650 6 4
© 2014 IBM Corporation64
Top-N Rank
Use a derived table to obtain top-n rank
select * from (select name, salary rank() over (order by salary desc) as rnk from employee) where rnk <= 5;
© 2014 IBM Corporation65
Rank in Sports Stats – All Time Baseball Home Runs
© 2014 IBM Corporation66
Rank Player Home Runs
….
15 Mike Schmidt 548
16 Mickey Mantle 536
17 Jimmy Foxx 534
18 Willie McCovey 521
18 Frank Thomas 521
18 Ted Williams 521
21 Ernie Banks 512
21 Eddie Matthews 512
23 Mel Ott 511
….
26 Albert Pujols 502
Current Home Run Hitters list All Time in Major League Baseball (partial)– Presently, the player Albert
Pujols is active with 502 home runs (as of 05-13-2014) and stands a chance at age 33 to surpass many of the names on this list.
– When he hits his 513th home run, he will rise into 21st place alone, possibly this year.
– The current occupants of 21st place will then be in 22nd place and forever still tied.
Rank in Sports Stats – All Time Women’s Soccer Goals
Rank Name Goals
1 Abby Wambach 165
2 Mia Hamm 158
3 Christine Sinclair 147
4 Kristine Lilly 130
5 Birgiz Prinz 128
6 Julie Fleeting 120
7 Elisabetta Vignotte 107
8 Sun Wen 106
9 Carolina Morace 105
9 Michelle Akers 105
11 Han Duan 101
12 Tiffeny Milbrett 100
© 2014 IBM Corporation67
Presently, neither Morace nor Akers are active players.
Time Series
© 2014 IBM Corporation
Time Series - Write your own program to load data faster
Loader API included in the product Loader APIs can directly be used in SQLs in application Pre-requisites:
– The name of table and TimeSeries column should be in lower case– Primary key is defined for each row– If primary key is CHAR(n), each value should be n bytes long OR
user VARCHAR(n)– Data to be loaded should be compatible to TimeSeries data type– Data must be compatible to Informix data types
© 2014 IBM Corporation69
Loader API SQL Functions
TSL_Init: Initialize the global and session environments for a specific table and TimeSeries column
TSL_Attach: Initialize a session environment
TSL_SetLogMode: Messages verbosity and destination of messages
TSL_GetLogMessage: Get the generated messages if they were put on a queue
TSL_GetKeyContainer: Return the container used by a specific TimeSeries
TSL_Put: Buffer the records from a lvarchar or a CLOB
TSL_Flush: Flush buffers into the TimeSeries
TSL_SessionClose: Close the loader session
TSL_Shutdown: Close the global environment
© 2014 IBM Corporation70
IBM Informix TimeSeries Plug-in for Data Studio
Eclipse-based Tools for Working with TimeSeries Data– Enables rapidly loading TimeSeries data– Custom file formats are easily created using graphical interface
New Features– Loading from another database product through JDBC
• Enables extract-load without intermediary files– Non-GUI tooling execution
• Enables automation in production environments
© 2014 IBM Corporation71
Loading TimeSeries Data from a Database
Select the connection profile for the source database– Defined using New → Database Connection Profile
Specify a query that results in the data to be loaded
A preview of the record structure that results from the query can be seen by clicking the “Preview” button
© 2014 IBM Corporation72
TS Relational Views in Centaurus
Improvements:– Implement new VTI capabilities to push the predicates down to the
implementation.– Process the new (recursive) predicates in the implementation to push to the
base table
Items in blue are now pushed down:SELECT id, col2, tstamp, value FROM test_tab_vWHERE udr1(id,3,7) = 1AND udr1(id,col2,7) = 1AND (id+1) = 4 ← id = 3 would be pushed downAND udr2(id)-- Now on TS columnAND udr3(value, 2, 3) > 4AND udr4(value)AND (value + 1) > 1.02-- Now on the timestamp as beforeAND tstamp BETWEEN datetime(2010-11-10 01:00:00.00000) year to fraction(5)AND datetime(2010-11-10 03:00:00.00000) year to fraction(5)
© 2014 IBM Corporation73
Rolling Window Containers
Informix 12.10 adds rolling window containers– Easy control of date driven roll-on/roll-off partitioning– Instant space management – no more slow deletes– Allows active/dormant window control– Windows based on units of a day, a week, a month or a year
Jan Feb Mar Apr May Jun
Jan Jul
Jan Feb Aug
Feb Mar Sep
Dormant Active
6 Month Rolling Window
Feb Mar Apr May Jun
JulMar Apr May Jun
AugJulApr May JunJan
Deleted
Jul
Aug
Sep
Oct
New
© 2014 IBM Corporation74
HA, SDS, CM, ER & Flexible Grid Enhancements
SDS_ALTERNATE – Parameter allows SDS network communication with a primary in case of primary server failure via smart blobspace.
ER works directly with Storage Provisioning feature
cdr check queue can be used to monitor transaction queues within ER. Supports reporting on the control, send and receive queues
Informix 12.10 introduces the S (send-only) participant type– Can be used in the cdr define replicate command
•In this case g_er1 has a S participant qualifier indicating the replicate can only send data changes from
this replicate, it can not receive updates g_er2 and g_er3 don’t have a participant qualifier indicating the replicate is R/W in these
instances
© 2014 IBM Corporation75
cdr define repl -c g_er1 -C always -f y price_book \“S stores@g_er1:informix.my_prices" "select * from my_prices where prod_code > 500" \“stores@g_er2:informix.my_prices" "select * from my_prices" \“stores@g_er3:informix.my_prices" "select * from my_prices"
Flexible Grid and Flat File Transfers
Informix 12.10 introduces the capability of replicating flat files to instances in a cluster– Uses the ifx_grid_copy() function to read from / write to a specific directory
on servers• Configured with GRIDCOPY_DIR parameter
Must be $INFORMIXDIR or subdirectory thereof Does not have to be identical across the cluster
The ifx_grid_copy() operation can specify different source and target locations for the files– Can read from $INFORMIXDIR/scripts_repository but send to
$INFORMIXDIR/custom_scripts on all the targets
76 © 2014 IBM Corporation
Grid Queries
© 2014 IBM Corporation
SQL Enhancements – Grid Queries In a Flexible Grid environment, it is possible for a cluster of
instances to contain the same dbspaces and tables but no replicated data– Common in retail and or distribution environments where stores / warehouses
have different SKUs (products) and inventory levels
Currently, to gather data from multiple instances requires executing multiple SQL operations into a temporary table or joining the results
Informix 12.10 introduces the grid query operation which, when executed from an administrative node, will gather data from the requested nodes with a single SQL statement
78 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
Grid queries can be executed against the entire cluster or against a subset of the instances – These subsets are defined as “regions” but have nothing to do with geography
The cdr define region command is used to define regions– Syntax
region_name must be unique across all clusters
• Will be used by SQL statements and cdr commands so have to be able to resolve to the correct region
list_of_instances is a whitespace separated list of server group names
cdr define region [--grid | -g ]=grid_name region_name [list_of_instances]
79 © 2014 IBM Corporation
SQL Enhancements – Grid Queries Example
– Instances can be members of more than one region if needed– Regions can contain a subset of instances of another region
• The great_west region can contain g_inst_1, g_inst_2, g_inst_3, g_inst_4• The pac_coast region can contain g_inst_1, g_inst_3
cdr define region –g=my_grid region_1 g_inst_1 g_inst_2;
cdr define region -–grid=my_grid region_2 g_inst_3 g_inst_4;
80 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
Regions can not be modified, have to drop and recreate– Use the cdr delete region command
cdr delete region region_1;
81 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
After defining any regions that are needed, next step to executing grid queries is to verify the consistency of the tables across the grid– Ensures the table(s) schemas are identical across the grid
• All the columns and data types match so a consistent response to the query can occur
– Prevents table alters from occurring unless executed through a grid operation• Connect to the grid, execute alter, disconnect from the grid
82 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
The cdr change gridtable command is used to identify tables for grid query operations aka “gridtables”– Syntax
Where[--add | -a | --delete | -d] – the following table(s) can be added or removed from the gridtable list[--all | -A | list_of_tables ] – either all or list of tables (white space separated)
cdr change gridtable [--grid | -g]=grid_name
[--database | -D]=database_name
[--add | -a | --delete | -d]
[--all | -A | list_of_tables ]
83 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
Examples– Notice long and short versions of the commands are used
Add all ds2 database tables
Delete the customer table from the gridtables list
Add the my_tab_1 and my_tab_2 tables to the gridtables list
cdr change gridtable –g=my_grid -D=ds2 -a –all;
cdr change gridtable –g=my_grid -D=ds2 –d customer;
cdr change gridtable –-grid=my_grid --database=ds2 --add my_tab_1 my_tab_2
84 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
With any regions defined, and tables added to the gridtable list, can now execute grid queries
Restrictions– Only query operations are supported
– Queries must “simple”, can not contain subqueries, joins, unions or the intersect, minus, or except qualifiers
• The grid query itself *can* be a subquery nested inside another query though
– Queries can only be executed against “normal” tables, not against views, synonyms or external tables
• Exception is sysmaster tables which are allowed
– Queries can only be executed against data types that are supported by distributed operations
• Excludes TimeSeries and any extended types that can’t be used in distributed operations
85 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
New syntax to the select operation – the grid clause– Syntax at the statement level
– The optional all keyword determines whether or not all matching rows are returned ( a union all qualifier) from all instances or just the unique values (a basic union qualifier)
• Default is union or unique values
select column_names from table_names grid [all] [‘region_name’ | ‘grid_name’] where . . . . .
86 © 2014 IBM Corporation
SQL Enhancements – Grid Queries New syntax to the select operation – at the session level
– Rather than modifying existing statements, the scope and union qualifier can be set at the session level
• With the set environment select_grid operation
Use the default union operator on the grid queries
Use the union all operator on the grid queries
set environment select_grid [‘grid_name’ | ‘region_name’]
set environment select_grid_all [‘grid_name’ | ‘region_name’]
87 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
New syntax to the select operation – at the session level (cont.)
– Once set, all queries will execute as grid queries
– Only one default setting can be active at a time• To change from one setting to the other re-execute the set environment command
– These settings can be changed for any individual statement(s) by including a grid clause within the statement
88 © 2014 IBM Corporation
SQL Enhancements – Grid Queries To turn off grid queries
– Use the option that matches the command enabling grid queries
– Individual statement(s) can still execute as a grid query by including the grid clause in the statement(s)
set environment select_grid default;
set environment select_grid_all default;
89 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
But what happens if a node with the grid / region is not available when the grid query executes?– It depends, can either
• Abort the query and return an error• Execute the query with the available nodes
Can query later to see how many and which nodes didn’t participate
Set at a session level with the set environment ifx_node_skip operation
90 © 2014 IBM Corporation
SQL Enhancements – Grid Queries Syntax
– Wheredefault | off – the default setting. Query is aborted and an error returned
on – rows are returned from the nodes available at the time the query was executed
When the environment is set to on, any skipped nodes are captured and can be returned to the calling session– Important – this information is gathered on a statement-by-statement basis
and only lasts until the next grid query is executed in that session
set environment ifx_node_skip [ default | off | on ]
91 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
To determine how many nodes, if any, were skipped and their names
Returns an integer value with the number of skipped nodes
Returns an lvarchar with the name of one of the skipped nodes• To get all the node names, you will need to execute this the same number of times
as returned by ifx_gridquery_skipped_nodes_count() All names aren’t returned at once to facilitate easier parsing of the result set The function operates like a fetch against a cursor
execute function ifx_gridquery_skipped_nodes_count()
execute function ifx_gridquery_skipped_nodes()
92 © 2014 IBM Corporation
SQL Enhancements – Grid Queries Example
set environment select_grid my_region;set environment ifx_node_skip on;
select . . . . from . . . . .; {results returned}
execute function ifx_gridquery_skipped_nodes_count();
3
execute function ifx_gridquery_skipped_nodes();g_inst_2
execute function ifx_gridquery_skipped_nodes();g_inst_19
execute function ifx_gridquery_skipped_nodes();g_inst_38
93 © 2014 IBM Corporation
SQL Enhancements – Grid Queries
In the event a schema change needs to be made– The change must be made through a grid operation as explained earlier
– A metadata flag will be set indicating an alter operation is in-flight• Prevents any grid queries from executing
– As tables are being altered and acknowledged as completed, a cdr remaster gridtable operation is automatically executed to re-verify consistency across the cluster
– When the cdr remaster gridtable operation returns 0 (zero), the metadata flag is removed and grid queries can be resumed
– The cdr remaster gridtable operation can be invoked by an administrator to check the status of the alter operation
• Can know when to turn on grid query applications
94 © 2014 IBM Corporation