Powering GIS Application with PostgreSQL and Postgres Plus

Supercharge Your GIS PlatformPostgres Plus with PostGIS

3/6/2014

ll rig

• Introduction• PostgreSQL and buzz it has created• Postgres Plus Advanced Server

• Architecture• Oracle Compatibility• Performance Feature• Security Features• High Availability Features• DBA Tools

• How to start adopting

• User Stories

• PostgreSQL and PostGIS

Agenda

ll rig

Postgres’ IMPACT

ll rig

• Independent & Thriving Development Community for over 20 years• Thousands of active deployments worldwide in public and private

sector organizations of all sizes

PostgreSQL and the Community

ll rig

Postgres with PostGIS

ll rig

PostgreSQL – Postgres Plus Users, Globally

ll rig

Malaysia

Philippines

Singapore

Vietnam Thailand

Indonesia

PostgreSQL – Postgres Plus Users, across ASEAN

ll rig

EnterpriseDB – The Company• The Enterprise PostgreSQL company was founded in 2004,

first product GA in 2005

• 2,000+ customers across all market segments

• 70,000+ downloads/week of PostgreSQL and related products

• Enabling database consolidation using PostgreSQL and advanced Oracle compatibility

• Saving customers millions through the power of open source

• Strong financial backing:

ll rig

Postgres Plus - Recognized

by Gartner’s Magic

Quadrant as a challenger

in the Database Market

The EnterpriseDB Advantage• Products and Tools

• Advanced database server software• Deep Oracle compatibility• Bundled development and management tools

• Technical Support and Services• Around the clock support• Expert consulting• Oracle migration services• Remote management and monitoring

• Professional Training• Learn PostgreSQL from the experts• Web and on-site training• Training for developers and DBAs CO

• Fast development cycles• Thousands of developers• Better code• Lower cost

• 24/7 support• Services and training• Certification• Indemnification• Product strategy

Bringing the Advantage to Enterprises

Open Source Software

CommercialSoftwareEnterpriseDB

ll rig

About PPAS

COMPANY

PERFORMANCE

COMPATIBILITYSECURITY

PostgreSQL

xDB REPLICATION

Server

HIGH AVAILABILITY

and Scalability

PERFORMANCE and

Manageability SECURITY

and Cloud

Database MIGRATION

Toolkit

ll rig

Architectural Overview

ll rig

Process Architecture

Postmaster

BGWRITER STATS COLLECTOR

ARCHIVERAUTO--VACUUM

BGWRITER STATS COLLECTOR

Data Files

WALSegments

ArchivedWAL

Shared MemoryShared Buffers Process ArrayWAL Buffers

WAL Writer LOG WRITER

Error Log Files12

ll rig

Database Limitations

ll rig

• Limitations are generally defined by• Operating System Limits• Compile-Time Parameters– Data Type Usage

General Database Limitations

Limit Value

Maximum Database Size Unlimited

Maximum Table Size 32 TB

Maximum Row Size 1.6 TB

Maximum Field Size 1 GB

Maximum Rows per Table Unlimited

Maximum Columns per Table 250-1600 (Depending on Column types)

Maximum Indexes per Table Unlimited

Oracle Compatibility

• Run applications written for Oracle virtually unchanged

• No need to re-train Oracle DBAs and developers

• Support for PL/SQL language and OCI interoperability

• Replication for easy sharing of data • Dramatic Cost Savings• No Vendor Lock-in

ll rig

Survey: Re-Use of Oracle DBA Skills

ll rig

Compatibility Means• SQL Extension support

• Decode, NVL, Substr, NVL2,replace,translate, table() • Date/time functions: add_months, extract, next_day,months_between,trunc, current_date support• Support for cascade option in drop table

• PL/SQL support• REF Cursors, Implicit and explicit cursors• Looping, variable declarations, conditional statements• Collections: Associative Arrays, Varrays, Nested tables• Bulk binding• Named parameters• User Defined Exceptions• Explicit Transaction Control - within a stored procedure

• Tools• EDB*Plus – SQL*Plus look-a-like• EDB*Loader – SQL*Loader equivalent• EDB*Wrap – similar to the PL/SQL wrapper CO

Compatibility Means (cont.)

• Packages• Stored procedures• Functions• Triggers• Optimizer Hints• Database Links• Hierarchical Queries• Synonyms – Public and

Private• Sequences• Materialized Views CO

• Rownum• Object types

o Create type … as objecto Create type … as tableo Create type …as varrayo Constructor and collection

methods• PL/SQL like SUBTYPE, which

inherits form base type• Users/Roles• Dynamic SQL

• Features

Compatibility Means (cont.)• Data Types

• Integer, number, char, double precision, float, varchar2, blob, clob, xmltype, rowid, boolean

• Built-in Packages• DBMS_:

o SQL, LOB, JOB, PIPE, ALERT, OUTPUT, UTILITY, PROFILER, RLS• UTL_:

o FILE, MAIL, SMTP, ENCODE, TCP

• Oracle-like Data Dictionary• ALL_, DBA_, USER_ views• Most commonly accessed views

• Diagnostics - DRITA• System and session waits

o Not exposed in PostgreSQL o Part of Advanced Server

• Statspack-like reporting

ll rig

Flexible Partitioning Scheme in PostgreSQL• Partitioning Example in PostgreSQL

o CREATE TABLE employeeso Create Child Tableso create table emp_mgmt (check (job in (MANAGER'))) inherits (employees);o create table emp_sales (check (job in ('SALESMAN'))) inherits (employees);o Create the partitioning functiono CREATE OR REPLACE FUNCTION partition_emp();o Create the trigger for partitiono CREATE TRIGGER employees_partition_trigger BEFORE INSERT ON employees

FOR EACH ROW EXECUTE PROCEDURE partition_emp();

• Advantages• Flexible• Customize your Partitioning Policy

• Disadvantages• Complex• Difficult to add/delete/split Partitions

ll rig

• Partitioning Example in PPASCREATE TABLE employees(empno numeric(4,0),ename varchar(10),job varchar(9),hiredate timestamp)PARTITION BY LIST (job)(PARTITION emp_mgmt VALUES ('MANAGER', 'PRESIDENT') tablespace tbsp_1, PARTITION emp_sales VALUES ('SALESMAN') tablespace tbsp_2, PARTITION emp_ops VALUES ('CLERK') tablespace tbsp_3);

• PPAS Offers• LIST and RANGE Partitioning Syntax• Easy to manage and maintain partitions• You can still take benefit of PostgreSQL syntax for complex partitioning e.g. Dynamic Partitions• Easy to Add/Delete/Split/Swap/Detach/Attach partitions• Brings the best of both the world!

Compatible and Flexible Partitioning Scheme in PPAS

ll rig

Scaling with Partitioning in PPAS v9.3

ll rig

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Time in Seconds

Partitioning Performance: PPAS 9.3 v. PostgreSQL 9.3 v. PPAS 9.2

• Postgres Plus Advanced Server 9.3 is up to 460 times faster with row insertions into partitioned tables than PostgreSQL 9.3 or PPAS 9.2

• Better performance for bulk loading and disaster recovery

• Online Migration Toolkit enables point and click migration from Oracle• Automatically Migrates:

Database Migration Toolkit

ll rig

• Data• Schemas• Stored Procedures• Triggers• Functions• Sequences• Packages• Views• Database Links• Synonyms

• Developed for High Performance Transaction Environments (OLTP)

• DynaTune: • Dynamic tuning of the database server to make the

optimal usage of the system resources available on the host machine

• Index Advisor: • Helps determine which columns you should index

to improve performance in a given workload.• Query Optimization Hints, Hi-Speed Bulk Loader,

Multi-Threaded Replication

Performance enhancement in PPAS

ll rig

• Infinite Cache • High performance

horizontal scaling architecture for cache memory

• Cache expands with inexpensive commodity hardware

Scalability with Infinite Cache

ll rig

Scalability with Infinite Cache• Architecture: How it Works

• Disk access for data is slower than RAM access• Infinite Cache puts disk data into the RAM cache• Additional requests for that data are read from the faster cache• The cache scales as needed with inexpensive hardware

ll rig

.Application

High speed

pg_cache buffers in RAM

Slow speed disk

High Speed

Network

Updated Data Blocks

Update

If not in buffer

Database Server

InfiniteCacheRAM

blades

High speed

• Single Machine Performance

ll rig

Advanced Server is 16X faster on a single machine with large amounts of memory (e.g. greater than 2 GB)

Infinite Cache can be usedon a single machine!

• Infinite Cache - Features• Expands and Distributes buffer cache across multiple machines• Designed for Read-Mostly applications (e.g. Content Management, Query Intensive, Business Intelligence, Data

Warehouse)• Cache is transparent to client applications (no cache coding needed)• Compression feature enables caching entire databases (e.g. put a 250 GB database into 32 GB RAM Cache)• Cache can be pre-warmed for immediate results• Cache scales using inexpensive commodity hardware• Infinite Cache can be run to boost single machine performance!

• Created For• DBAs and data managers overseeing large amounts of data requiring fast response times for queries and

reporting loads• Developers who don’t want to write specialized caching code

ll rig

• Red Hat Reference Architecture Series:

• Comparing BenchmarkSQL Performance

on Red Hat Enterprise Linux 5 to Windows

Server Enterprise (2010)

Performance Benchmark to SQL Server

ll rig

Scalability MySQL Vs PostgreSQL

ll rig

Security

• Object level privileges assigned to roles and users• Virtual Private Database• Kerberos and LDAP authentication• Host base authentication• SSL communication• Data Level Encryption (AES, 3DES, etc)• Ability to utilize 3rd party Key Stores in a full PKI Infrastructure• Foundation for full compliance with the strictest of security standards (PCI Data Security

Standard)• Flexible field level encryption and row level security

ll rig

“By default, PostgreSQL is probably the most security-aware database available ...” Database Hacker's Handbook- David Litchfield

PL/Secure and EDB*Wrap

• Encrypt your Pl/pgsql stored programs with PL/Secure• Encrypt your SPL procedures with EDB*Wrap

ll rig

SQL/Protect

• DBA Managed SQL Injection Protection • Preventing attacks is normally the responsibility of the application developer, but with SQL/Protect, DBAs can

now provide another layer of protection to prevent corruption or co-opting of the database.

• Multiple Prevention Techniques• Unauthorized Relations • Utility Commands (e.g. DDL)• SQL Tautology • Unbounded DML

ll rig

• Multi-Master Replication – Active-Active Technology• Near real-time Replication• Multi-Version Concurrency Control (MVCC)• Point-in-Time Recovery• Log Shipping for Standby (~ Oracle® Data Guard)

High Availability

ll rig

Availability

Backup and Recovery Options

ll rig

• Physical and Logical Backup• Logical Backup using pg_dump• Instance level logical backup using pg_dumpall• Table level, Schema level or DB level logical backup• pg_restore• Physical Backups• pg_basebackup• Compressed backup• Recovery with WAL and archived WAL• Point in Time Recover

• Using OS Clustering• Using a shared disk for data

Active – Passive OS HA Clustering

ll rig

• Uses WAL (Write Ahead Log)• WAL is continuously shipped through

an opened TCP channel to Hot Standby Node

• Hot Standby node can be used for read scalability

Hot Streaming Replication (Hot Standby Mode)

ll rig

• Uses WAL (Write Ahead Log)• Archived WAL is shipped to Hot

Standby Node• Hot Standby node can be used for

read scalability

Log Shipping Replication (Hot Standby Mode)

ll rig

EnterpriseDB Postgres Plus Failover Manager

ll rig

Agent Agent

• Using pg-pool• Using redundant disk for data• Hot Standby node can be used

for read scalability

HA with read scaling (with pg-pool)

ll rig

• Trigger based replication• Publication – Subscription• Snapshot and Continuous• Cascaded Replication• Read Scalability• Master/Slave DB can be:

• Oracle• MSSQL• Postgres Plus Advanced Server• Either Master or Slave should be

PPAS/PostgreSQL

xDB Single Master Replication (SMR)

ll rig

• High Availability uses• Geographic distribution of load• For creation of Testing/staging env using snapshot replication• Segregate OLTP and reporting

xDB Replication Use Cases

ll rig

Replication

xDB Single-Master Replication (SMR)

ll rig

Table D

Table C

Table B

Table A

Table C

Table D

ProcsObjects

ReportsQueries

Continuousor Scheduled----------------

Filteredor All Rows

TransactionReplication

ImprovedOLTP

Performance

InexpensiveQuery /

Reporting

Oracle Server Postgres Plus Advanced Server

xDB SMR Heterogeneous Replication Support

ll rig

Source/Target

Oracle MS SQL Server

2005/2008

PostgreSQL

Advanced Server(Oracle mode)

Advanced Server

(PostgreSQL mode)

Oracle No No Yes Yes Yes

MS SQL Server

2005/2008No No Yes Yes Yes

PostgreSQL No Yes Yes Yes Yes

Advanced Server (Oracle mode)

Yes Yes No Yes No

Advanced Server

(PostgreSQL mode)

No Yes Yes Yes Yes

• Trigger based replication• 2 or more Masters can be Sync• Auto Conflict Detection & Resolution• Read & Write Scalability

xDB Multi Master Replication (MMR)

ll rig

• Java-based Replication Server and Replication Console• Delta changes recorded in shadow tables via post Insert/Update/Delete triggers• Data Replication through JDBC channel

xDB MMR Architecture

ll rig

Enterprise Class DBA Tools

ll rig

Remotely Manage, Monitor And Tune

• Management En-Mass Design

• Enterprise Performance Monitoring

• Proactive Alert Management

• Graphical Administration

• SQL Workload Profiling

• Simplified Capacity Planning

• Full SQL IDE

Postgres Enterprise Manager

ll rig

PEM Distributed Architecture

ll rig

• Performance Monitoring Dashboards

• Capacity Manager• Postgres Expert• Alert Management• Browser based console

and dashboard• Audit Manager• Team Support• Distributed Architecture• Convenient Access

PEM For DBAs: Centralized Tool

ll rig

Helps you apply patches and updates for all the products under PPAS:

Stackbuilder Plus

ll rig

•A solution to aid in the creation of highly available configurations of Postgres

•Monitors the health of a Postgres HA configuration

•Automates the failover process in the event of a failure

•Used in conjunction with Streaming Replication

Postgres Plus Failover Manager

ll rig

Witness

Master

Failover Manager Architecture

ll rig

Agent Agent

• Automatic Failover from master to replica node• Configurable fencing operation

• By default uses VIP• Parameter to specify alternative operation

• Ex: reconfigure a load balancer

• Manual failover configuration possible• Email notifications when cluster status changes• Witness node provides protection against ‘split

brain’ scenarios• User configurable wait times• Built on PPCD/JGroups technology

Failover Manager Features

ll rig

Replicate below objects from MS SQL Server to

Postgres Plus Advanced Server:

• Schemas• Tables• Constraints• Indexes• Table data• Views

Replicating & Migrating Microsoft SQL Server

ll rig

• Split input files into multiple files• Load single tables in parallel• Magnitudes better load time performance• Parallel load capabilities also in Migration Toolkit and Migration Studio

Parallel Data Loader

ll rig

Use Cases:- Sistema

de Información Territorial Estatal en Línea, México

- National Oceanic and Atmospheric Administration

- TriMet

- http://www.ridethecity.com/

- FCC: Making the National Broadband Map with OpenGeo Suite

- North Dakota State Water Commission

Comparative Studies:- http://

www.gise.cse.iitb.ac.in/wiki/images/c/c4/Finalreport.pdf

- bit.ly/1e4meWo

Some Use Case for Postgres and PostGIS

ll rig

• OpenGeo

• QuantumGIS

• GeoServer

• ArcGIS

Popular GIS Applications are Compatible with PostgreSQL

ll rig

You have a ‘Real’ alternative to Oracle or other conventional proprietary Databases

Conclusion

ll rig

Economical

TechnicallySoun

Easy to

How to start adoption

ll rig

REPLICATION SERVER

BACKUP / DR SERVER

NEW APPLICATION

MIGRATION✔✔ ✔

ll rig

Powering GIS Application with PostgreSQL and Postgres Plus

Technology

Transcript of Powering GIS Application with PostgreSQL and Postgres Plus

The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)

FLEXIBLE WITH POSTGRESQL, ORACLE® AND SQL SERVER® - Enterprises Use EDB Postgres …info.enterprisedb.com/rs/069-ALB-339/images/edb-postgres... · 2020-04-11 · EDB POSTGRES REPLICATION

Postgres-XC PostgreSQL Conference 2012 Michael ... PostgreSQL Conference 2012 Michael PAQUIER Tokyo, 2012/02/24

Postgres-XC Write Scalable PostgreSQL Cluster

Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open

Solution Brief VEXATA INCREASES POSTGRESQL PERFORMANCE · Solution Brief VEXATA INCREASES POSTGRESQL PERFORMANCE !!!! PostgreSQL (Postgres) is one of the most powerful open source

Günümüzde Postgres - 5. PostgreSQL Konferansıdialoglar.com/assets/presentations/utku_onder_olceklenebilir... · PostgreSQL: Dinamikveglobal ekosistem citus pgcrypto pg_cron pg_partman

Researching PostgreSQL Performance - PGCon · Researching PostgreSQL Performance ... OS Tuning echo "8589934592" > /proc/sys/kernel/shmmax ... postgres soft nofile 63536 postgres

Distributed PostgreSQL - Percona · Postgres-XL Fork of PostgreSQL 10 to provide Scalable PostgreSQL Hash distributed tables and replicated tables Multi-node transactions, Multi-node

High availability in Postgres-XC, the symmetric - PostgreSQL wiki

New features in PostgreSQL 9 - hagander.net features in PostgreSQL 9.0.pdf · New features in PostgreSQL 9.0 Open Source Days, ... postgres=# EXPLAIN (ANALYZE, ... Conditional triggers

Postgres-XC: Symmetric PostgreSQL Cluster

PostgreSQL 13 Beta 1 New Features With Examples (EN) · 5/27/2020 · postgres=# psql command prompt for PostgreSQL administrator. postgres=> psql command prompt for PostgreSQL

PostgreSQL 7.2 User’s Guide - Instituto de …celio/livrobd/postgres/user-7...PostgreSQL is an object-relational database management system (ORDBMS) based on POSTGRES, Version 4.2

The Future of Postgres Sharding - PostgreSQL EuropeTitle: The Future of Postgres Sharding - This presentation will cover the advantages of sharding and future Postgres sharding implementation

PostgreSQL is NOT your traditional SQL · 2018-09-28 · PostgreSQL Europe 2ndQuadrant Prague PostgreSQL Meetup Postgres Women TalTech Yildiz Technical University blog Github: @apatheticmagpie

Postgres Professional, Moscow Universitymegera/postgres/talks/pgcon-2016-fts.pdf · Russian PostgreSQL developers Oleg Bartunov, Teodor Sigaev PostgreSQL CORE Locale support PostgreSQL

High Availability Database Solutions for PostgreSQL & Postgres Plus An EnterpriseDB White Paper

Guia del Programador de PostgreSQL - index-of.co.ukindex-of.co.uk/SERVIDORES/Postgres-Programmer.pdf · Guia del Programador de PostgreSQL El equipo de desarrollo de PostgreSQL Editado

Crunchy Data PostgreSQL Operator · 2017-09-07 · PostgreSQL 9.5+ Container (crunchy-postgres) PostgreSQL Backup Container (crunchy-backup) PostgreSQL Upgrade Container (crunchy-upgrade)