Supercharge Your GIS PlatformPostgres Plus with PostGIS

3/6/2014

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

2

• Introduction• PostgreSQL and buzz it has created• Postgres Plus Advanced Server

• Architecture• Oracle Compatibility• Performance Feature• Security Features• High Availability Features• DBA Tools

• How to start adopting

• User Stories

• PostgreSQL and PostGIS

Agenda

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

3

Postgres’ IMPACT

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

4

• Independent & Thriving Development Community for over 20 years• Thousands of active deployments worldwide in public and private

sector organizations of all sizes

PostgreSQL and the Community

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

5

Postgres with PostGIS

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

6

PostgreSQL – Postgres Plus Users, Globally

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

7

Malaysia

Philippines

Singapore

Vietnam Thailand

Indonesia

PostgreSQL – Postgres Plus Users, across ASEAN

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

8

EnterpriseDB – The Company• The Enterprise PostgreSQL company was founded in 2004,

first product GA in 2005

• 2,000+ customers across all market segments

• 70,000+ downloads/week of PostgreSQL and related products

• Enabling database consolidation using PostgreSQL and advanced Oracle compatibility

• Saving customers millions through the power of open source

• Strong financial backing:

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Postgres Plus - Recognized

by Gartner’s Magic

Quadrant as a challenger

in the Database Market

9

The EnterpriseDB Advantage• Products and Tools

• Advanced database server software• Deep Oracle compatibility• Bundled development and management tools

• Technical Support and Services• Around the clock support• Expert consulting• Oracle migration services• Remote management and monitoring

• Professional Training• Learn PostgreSQL from the experts• Web and on-site training• Training for developers and DBAs CO

NFI

DEN

TIAL

© 2

011

Ente

rpris

eDB.

All

right

s re

serv

ed.

10

• Fast development cycles• Thousands of developers• Better code• Lower cost

• 24/7 support• Services and training• Certification• Indemnification• Product strategy

Bringing the Advantage to Enterprises

Open Source Software

CommercialSoftwareEnterpriseDB

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

11

About PPAS

COMPANY

PERFORMANCE

COMPATIBILITYSECURITY

TOOLS

PostgreSQL

xDB REPLICATION

Server

HIGH AVAILABILITY

and Scalability

PERFORMANCE and

Manageability SECURITY

and Cloud

Database MIGRATION

Toolkit

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Architectural Overview

11

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Process Architecture

Postmaster

BGWRITER STATS COLLECTOR

ARCHIVERAUTO--VACUUM

BGWRITER STATS COLLECTOR

Data Files

WALSegments

ArchivedWAL

Shared MemoryShared Buffers Process ArrayWAL Buffers

WAL Writer LOG WRITER

Error Log Files12

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

14

Database Limitations

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

• Limitations are generally defined by• Operating System Limits• Compile-Time Parameters– Data Type Usage

General Database Limitations

Limit Value

Maximum Database Size Unlimited

Maximum Table Size 32 TB

Maximum Row Size 1.6 TB

Maximum Field Size 1 GB

Maximum Rows per Table Unlimited

Maximum Columns per Table 250-1600 (Depending on Column types)

Maximum Indexes per Table Unlimited

15

Oracle Compatibility

• Run applications written for Oracle virtually unchanged

• No need to re-train Oracle DBAs and developers

• Support for PL/SQL language and OCI interoperability

• Replication for easy sharing of data • Dramatic Cost Savings• No Vendor Lock-in

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

16

Survey: Re-Use of Oracle DBA Skills

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

17

Compatibility Means• SQL Extension support

• Decode, NVL, Substr, NVL2,replace,translate, table() • Date/time functions: add_months, extract, next_day,months_between,trunc, current_date support• Support for cascade option in drop table

• PL/SQL support• REF Cursors, Implicit and explicit cursors• Looping, variable declarations, conditional statements• Collections: Associative Arrays, Varrays, Nested tables• Bulk binding• Named parameters• User Defined Exceptions• Explicit Transaction Control - within a stored procedure

• Tools• EDB*Plus – SQL*Plus look-a-like• EDB*Loader – SQL*Loader equivalent• EDB*Wrap – similar to the PL/SQL wrapper CO

NFI

DEN

TIAL

© 2

011

Ente

rpris

eDB.

All

right

s re

serv

ed.

18

Compatibility Means (cont.)

• Packages• Stored procedures• Functions• Triggers• Optimizer Hints• Database Links• Hierarchical Queries• Synonyms – Public and

Private• Sequences• Materialized Views CO

NFI

DEN

TIAL

© 2

011

Ente

rpris

eDB.

All

right

s re

serv

ed.

• Rownum• Object types

o Create type … as objecto Create type … as tableo Create type …as varrayo Constructor and collection

methods• PL/SQL like SUBTYPE, which

inherits form base type• Users/Roles• Dynamic SQL

• Features

19

Compatibility Means (cont.)• Data Types

• Integer, number, char, double precision, float, varchar2, blob, clob, xmltype, rowid, boolean

• Built-in Packages• DBMS_:

o SQL, LOB, JOB, PIPE, ALERT, OUTPUT, UTILITY, PROFILER, RLS• UTL_:

o FILE, MAIL, SMTP, ENCODE, TCP

• Oracle-like Data Dictionary• ALL_, DBA_, USER_ views• Most commonly accessed views

• Diagnostics - DRITA• System and session waits

o Not exposed in PostgreSQL o Part of Advanced Server

• Statspack-like reporting

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

20

Flexible Partitioning Scheme in PostgreSQL• Partitioning Example in PostgreSQL

o CREATE TABLE employeeso Create Child Tableso create table emp_mgmt (check (job in (MANAGER'))) inherits (employees);o create table emp_sales (check (job in ('SALESMAN'))) inherits (employees);o Create the partitioning functiono CREATE OR REPLACE FUNCTION partition_emp();o Create the trigger for partitiono CREATE TRIGGER employees_partition_trigger BEFORE INSERT ON employees

FOR EACH ROW EXECUTE PROCEDURE partition_emp();

• Advantages• Flexible• Customize your Partitioning Policy

• Disadvantages• Complex• Difficult to add/delete/split Partitions

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

21

• Partitioning Example in PPASCREATE TABLE employees(empno numeric(4,0),ename varchar(10),job varchar(9),hiredate timestamp)PARTITION BY LIST (job)(PARTITION emp_mgmt VALUES ('MANAGER', 'PRESIDENT') tablespace tbsp_1, PARTITION emp_sales VALUES ('SALESMAN') tablespace tbsp_2, PARTITION emp_ops VALUES ('CLERK') tablespace tbsp_3);

• PPAS Offers• LIST and RANGE Partitioning Syntax• Easy to manage and maintain partitions• You can still take benefit of PostgreSQL syntax for complex partitioning e.g. Dynamic Partitions• Easy to Add/Delete/Split/Swap/Detach/Attach partitions• Brings the best of both the world!

Compatible and Flexible Partitioning Scheme in PPAS

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

22

Scaling with Partitioning in PPAS v9.3

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

100

250

500

1000

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Time in Seconds

Nu

mb

er

of

Ta

ble

Pa

rtit

ion

s

Partitioning Performance: PPAS 9.3 v. PostgreSQL 9.3 v. PPAS 9.2

• Postgres Plus Advanced Server 9.3 is up to 460 times faster with row insertions into partitioned tables than PostgreSQL 9.3 or PPAS 9.2

• Better performance for bulk loading and disaster recovery

23

• Online Migration Toolkit enables point and click migration from Oracle• Automatically Migrates:

Database Migration Toolkit

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

• Data• Schemas• Stored Procedures• Triggers• Functions• Sequences• Packages• Views• Database Links• Synonyms

24

• Developed for High Performance Transaction Environments (OLTP)

• DynaTune: • Dynamic tuning of the database server to make the

optimal usage of the system resources available on the host machine

• Index Advisor: • Helps determine which columns you should index

to improve performance in a given workload.• Query Optimization Hints, Hi-Speed Bulk Loader,

Multi-Threaded Replication

Performance enhancement in PPAS

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

25

• Infinite Cache • High performance

horizontal scaling architecture for cache memory

• Cache expands with inexpensive commodity hardware

Scalability with Infinite Cache

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

26

Scalability with Infinite Cache• Architecture: How it Works

• Disk access for data is slower than RAM access• Infinite Cache puts disk data into the RAM cache• Additional requests for that data are read from the faster cache• The cache scales as needed with inexpensive hardware

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.Application

High speed

pg_cache buffers in RAM

Slow speed disk

High Speed

Network

Updated Data Blocks

Update

Read

If not in buffer

Database Server

InfiniteCacheRAM

blades

High speed

27

Scalability with Infinite Cache

• Single Machine Performance

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Advanced Server is 16X faster on a single machine with large amounts of memory (e.g. greater than 2 GB)

Infinite Cache can be usedon a single machine!

28

Scalability with Infinite Cache

• Infinite Cache - Features• Expands and Distributes buffer cache across multiple machines• Designed for Read-Mostly applications (e.g. Content Management, Query Intensive, Business Intelligence, Data

Warehouse)• Cache is transparent to client applications (no cache coding needed)• Compression feature enables caching entire databases (e.g. put a 250 GB database into 32 GB RAM Cache)• Cache can be pre-warmed for immediate results• Cache scales using inexpensive commodity hardware• Infinite Cache can be run to boost single machine performance!

• Created For• DBAs and data managers overseeing large amounts of data requiring fast response times for queries and

reporting loads• Developers who don’t want to write specialized caching code

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

29

• Red Hat Reference Architecture Series:

• Comparing BenchmarkSQL Performance

on Red Hat Enterprise Linux 5 to Windows

Server Enterprise (2010)

Performance Benchmark to SQL Server

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

30

Scalability MySQL Vs PostgreSQL

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

31

Security

• Object level privileges assigned to roles and users• Virtual Private Database• Kerberos and LDAP authentication• Host base authentication• SSL communication• Data Level Encryption (AES, 3DES, etc)• Ability to utilize 3rd party Key Stores in a full PKI Infrastructure• Foundation for full compliance with the strictest of security standards (PCI Data Security

Standard)• Flexible field level encryption and row level security

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

“By default, PostgreSQL is probably the most security-aware database available ...” Database Hacker's Handbook- David Litchfield

32

PL/Secure and EDB*Wrap

• Encrypt your Pl/pgsql stored programs with PL/Secure• Encrypt your SPL procedures with EDB*Wrap

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

33

SQL/Protect

• DBA Managed SQL Injection Protection • Preventing attacks is normally the responsibility of the application developer, but with SQL/Protect, DBAs can

now provide another layer of protection to prevent corruption or co-opting of the database.

• Multiple Prevention Techniques• Unauthorized Relations • Utility Commands (e.g. DDL)• SQL Tautology • Unbounded DML

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

34

• Multi-Master Replication – Active-Active Technology• Near real-time Replication• Multi-Version Concurrency Control (MVCC)• Point-in-Time Recovery• Log Shipping for Standby (~ Oracle® Data Guard)

High Availability

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Availability

35

Backup and Recovery Options

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

• Physical and Logical Backup• Logical Backup using pg_dump• Instance level logical backup using pg_dumpall• Table level, Schema level or DB level logical backup• pg_restore• Physical Backups• pg_basebackup• Compressed backup• Recovery with WAL and archived WAL• Point in Time Recover

36

• Using OS Clustering• Using a shared disk for data

Active – Passive OS HA Clustering

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

37

• Uses WAL (Write Ahead Log)• WAL is continuously shipped through

an opened TCP channel to Hot Standby Node

• Hot Standby node can be used for read scalability

Hot Streaming Replication (Hot Standby Mode)

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

38

• Uses WAL (Write Ahead Log)• Archived WAL is shipped to Hot

Standby Node• Hot Standby node can be used for

read scalability

Log Shipping Replication (Hot Standby Mode)

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

39

EnterpriseDB Postgres Plus Failover Manager

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Agent Agent

40

• Using pg-pool• Using redundant disk for data• Hot Standby node can be used

for read scalability

HA with read scaling (with pg-pool)

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

41

• Trigger based replication• Publication – Subscription• Snapshot and Continuous• Cascaded Replication• Read Scalability• Master/Slave DB can be:

• Oracle• MSSQL• Postgres Plus Advanced Server• Either Master or Slave should be

PPAS/PostgreSQL

xDB Single Master Replication (SMR)

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

42

• High Availability uses• Geographic distribution of load• For creation of Testing/staging env using snapshot replication• Segregate OLTP and reporting

xDB Replication Use Cases

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Replication

43

xDB Single-Master Replication (SMR)

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Table D

Table C

Table B

Table A

Table C

Table D

ProcsObjects

ReportsQueries

Continuousor Scheduled----------------

Filteredor All Rows

TransactionReplication

ImprovedOLTP

Performance

InexpensiveQuery /

Reporting

Oracle Server Postgres Plus Advanced Server

44

xDB SMR Heterogeneous Replication Support

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Source/Target

Oracle MS SQL Server

2005/2008

PostgreSQL

Advanced Server(Oracle mode)

Advanced Server

(PostgreSQL mode)

Oracle No No Yes Yes Yes

MS SQL Server

2005/2008No No Yes Yes Yes

PostgreSQL No Yes Yes Yes Yes

Advanced Server (Oracle mode)

Yes Yes No Yes No

Advanced Server

(PostgreSQL mode)

No Yes Yes Yes Yes

45

• Trigger based replication• 2 or more Masters can be Sync• Auto Conflict Detection & Resolution• Read & Write Scalability

xDB Multi Master Replication (MMR)

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

46

• Java-based Replication Server and Replication Console• Delta changes recorded in shadow tables via post Insert/Update/Delete triggers• Data Replication through JDBC channel

xDB MMR Architecture

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

47

Enterprise Class DBA Tools

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

48

Remotely Manage, Monitor And Tune

• Management En-Mass Design

• Enterprise Performance Monitoring

• Proactive Alert Management

• Graphical Administration

• SQL Workload Profiling

• Simplified Capacity Planning

• Full SQL IDE

Postgres Enterprise Manager

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

49

PEM Distributed Architecture

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

50

• Performance Monitoring Dashboards

• Capacity Manager• Postgres Expert• Alert Management• Browser based console

and dashboard• Audit Manager• Team Support• Distributed Architecture• Convenient Access

PEM For DBAs: Centralized Tool

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

51

Helps you apply patches and updates for all the products under PPAS:

Stackbuilder Plus

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

52

•A solution to aid in the creation of highly available configurations of Postgres

•Monitors the health of a Postgres HA configuration

•Automates the failover process in the event of a failure

•Used in conjunction with Streaming Replication

Postgres Plus Failover Manager

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Slave

Witness

Master

53

Failover Manager Architecture

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Agent Agent

54

• Automatic Failover from master to replica node• Configurable fencing operation

• By default uses VIP• Parameter to specify alternative operation

• Ex: reconfigure a load balancer

• Manual failover configuration possible• Email notifications when cluster status changes• Witness node provides protection against ‘split

brain’ scenarios• User configurable wait times• Built on PPCD/JGroups technology

Failover Manager Features

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

55

Replicate below objects from MS SQL Server to

Postgres Plus Advanced Server:

• Schemas• Tables• Constraints• Indexes• Table data• Views

Replicating & Migrating Microsoft SQL Server

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

56

• Split input files into multiple files• Load single tables in parallel• Magnitudes better load time performance• Parallel load capabilities also in Migration Toolkit and Migration Studio

Parallel Data Loader

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

57

Use Cases:- Sistema

de Información Territorial Estatal en Línea, México

- National Oceanic and Atmospheric Administration

- TriMet

- http://www.ridethecity.com/

- FCC: Making the National Broadband Map with OpenGeo Suite

- North Dakota State Water Commission

Comparative Studies:- http://

www.gise.cse.iitb.ac.in/wiki/images/c/c4/Finalreport.pdf

- bit.ly/1e4meWo

Some Use Case for Postgres and PostGIS

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

58

• OpenGeo

• QuantumGIS

• GeoServer

• ArcGIS

Popular GIS Applications are Compatible with PostgreSQL

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

59

You have a ‘Real’ alternative to Oracle or other conventional proprietary Databases

Conclusion

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

Economical

TechnicallySoun

d

Easy to

Adopt

60

How to start adoption

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.

REPLICATION SERVER

BACKUP / DR SERVER

NEW APPLICATION

MIGRATION✔✔ ✔

✔

61

CON

FID

ENTI

AL©

201

1 En

terp

riseD

B. A

ll rig

hts

rese

rved

.