Hybrid Row-Column Partitioning in Teradata · Experimental Results PI Regular PI table with no...
36
Hybrid Row-Column Partitioning in Teradata Luca Gasparini October 17, 2017 HADP Seminar
Transcript of Hybrid Row-Column Partitioning in Teradata · Experimental Results PI Regular PI table with no...
Hybrid Row-Column Partitioning in Teradata
Luca GaspariniOctober 17, 2017HADP Seminar
Data Partitioning
● Easier management● Assist backup/recovery● Enhance Performance
Data Partitioning
SELECT avg(C)FROM TWHERE A between 3 and 9
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
No Partitioning
Data Partitioning
SELECT avg(C)FROM TWHERE A between 3 and 9
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
No Partitioning
Data Partitioning
Horizontal Partitioning Vertical Partitioning
SELECT avg(C)FROM TWHERE A between 3 and 9
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
Data Partitioning
Horizontal Partitioning Vertical Partitioning
SELECT avg(C)FROM TWHERE A between 3 and 9
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
Data Partitioning
Horizontal Partitioning Vertical Partitioning
SELECT avg(C)FROM TWHERE A between 3 and 9
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
Data Partitioning
SELECT avg(C)FROM TWHERE A between 3 and 9
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
Column & Row Partitioning
Wikipedia:● Integrated Data Warehouse Hardware and
Software● Unified Data Architecture● Big Data Analytics● Professional Services● Customer Services
Teradata Parallel Database
PE PE PE PE....
BYNET
AMP AMP AMP AMP AMP
VDISK VDISK VDISK VDISK VDISK
....
....
Multilevel PartitioningCREATE TABLE table(
ID INTEGER,date DATE,c
1 TYPE,
....c
n TYPE
)
Multilevel PartitioningCREATE TABLE table(
ID INTEGER,date DATE,c
1 TYPE,
....c
n TYPE
)
2015 2016 2017
Multilevel PartitioningCREATE TABLE table(
ID INTEGER,date DATE,c
1 TYPE,
....c
n TYPE
)
2015 2016 2017
ID1
ID2 .... ID
1ID
2 .... ....
Multilevel PartitioningCREATE TABLE table(
ID INTEGER,date DATE,c
1 TYPE,
....c
n TYPE
)
2015 2016 2017
ID1
ID2 .... ID
1ID
2 .... ....
C1.1
C1.2 .... C
2.1C
2.2 ........
File System
● doesn't care about partitioning scheme● rowid based (unique 16-byte key)
rowid
3 types of indices:
● PI (Primary Index)● PA (Primary AMP Index)● NoPI (no Primary Index)
PI & PA
Rows are hash-distributed over AMPs on value of PI/PA column
NoPI
Rows are distributed randomly over AMPs
rowid
PI table PA/NoPi table
internal partition number row hash uniqueness internal partition number hash bucket uniqueness
64 bits 32 bits64 bits 32 bits 64 bits 20 bits 44 bits
rowid
PI table PA/NoPi table
internal partition number row hash uniqueness internal partition number hash bucket uniqueness
64 bits 32 bits64 bits 32 bits 64 bits 20 bits 44 bits
PI tables PA tables NoPi tables
Hash-distribution to AMP
YES YES NO
Row-hash Ordering on AMP
YES NO NO
rowid
PI table PA/NoPi table
internal partition number row hash uniqueness internal partition number hash bucket uniqueness
64 bits 32 bits64 bits 32 bits 64 bits 20 bits 44 bits
∏i=1
n
di<(263−1)
CPN (Combined Partition Number)
(∑i=1
n−1
( pi−1 )× ∏j= i+1
n
d j )+ pn
CPN (Combined Partition Number)
(∑i=1
n−1
( pi−1 )× ∏j= i+1
n
d j )+ pn
number of partition at level i
degree of partitioning at level j
CPN (Combined Partition Number)
(∑i=1
n−1
( pi−1 )× ∏j= i+1
n
d j )+ pn
number of level n partitions below level i
index of partition at level n-1
index of last partition
CPN (Combined Partition Number)
(∑i=1
n−1
( pi−1 )× ∏j= i+1
n
d j )+ pn
1 2
1 2 1 2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Level 1
Level 2
CPN
1 2 3 4 1 2 3 4 1 3 4 1 2 3 42Level 3
CPN (Combined Partition Number)
(∑i=1
n−1
( pi−1 )× ∏j= i+1
n
d j )+ pn
1 2
1 2 1 2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
((2 - 1) * (2 * 4)) + ((1 - 1) * 4) + 2 = 10
Level 1
Level 2
CPN
1 2 3 4 1 2 3 4 1 3 4 1 2 3 42Level 3
Partitioning Expressions
PARTITION BY(…
)
Partitioning Expressions
PARTITION BY(partition_expr
1,
…,partition_expr
n
)
COLUMN [[NO] AUTO COMPRESS]
RANGE_N(test-value BETWEENrange [, range] …
)
CASE_N(condition-expr1,
…,condition-expr
n
)
Partitioning Expressions
CREATE TABLE Sales(TxnNo INTEGER, TxnDate DATE, ItemNo INTEGER, Quantity INTEGER,)
NO PRIMARY INDEX,PARTITION BY(
COLUMN,RANGE_N(TxnDate BETWEEN
Date ´2011-01-01´ AND DATE ´2011-12-31´ EACH INTERVAL ´1´ DAY))
Teradata Columnar
● ROW format ● COLUMN format
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
Teradata Columnar
● ROW format ● COLUMN format
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
Teradata Columnar
● ROW format ● COLUMN format
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
Teradata Columnar
● COLUMN format
A B C D
10 11 2 8
6 3 4 7
1 12 9 10
5 0 2 8
PI table PA/NoPi table
internal partition number row hash uniqueness internal partition number hash bucket uniqueness
64 bits 32 bits64 bits 32 bits 64 bits 20 bits 44 bits
Experimental Results
PI Regular PI table with no partitioning
RPPI Partitioned primary index table with 84 monthly row partitions
CP NoPI Single-column CP table with COLUMN format and autocompression
CRP Multilevel partitioned table with column partitioning at the first level and row partitioning (by month) at the second level
RCP Multilevel partitioned table with row partitioning (by month) at the first level and column partitioning at the second level
Experimental Results
PI RPPI CP CRP RCP0
100
200
300
400
500
600
700
800
900
1000
877 889
319 308 308
Table Size
Bill
ion
s o
f b
yte
s
Experimental Results
PI RPPI CP CRP RCP0
100
200
300
400
500
600
700
800
900
562 558
812
856 844
Full Table Scan
time
[s]
Experimental Results
PI RPPI CP CRP RCP0
50
100
150
200
250
300
111 113
86 86 86
111 113
156 156161
114 116
266 270 269
Aggregation
3 of 169 of 1615 of 16tim
e [
s]
