Prestogres internals

Sadayuki FuruhashiFounder & Software Architect

Treasure Data, inc.

internalsPostgreSQL protocol gateway for Presto

A little about me...> Sadayuki Furuhashi

> github/twitter: @frsyuki

> Treasure Data, Inc. > Founder & Software Architect

> Open-source hacker > MessagePack - Efficient object serializer > Fluentd - An unified data collection tool > ServerEngine - A Ruby framework to build multiprocess servers > Prestogres - PostgreSQL protocol gateway for Presto > LS4 - A distributed object storage with cross-region replication > kumofs - A distributed strong-consistent key-value data store

Today’s talk

1. What’s Presto?

2. Prestogres design

3. Prestogres implementation

4. Prestogres hacks

5. Prestogres future works

1. What’s Presto?

What’s Presto?

A distributed SQL query engine for interactive data analisys against GBs to PBs of data.

What’s the problems to solve?> We couldn’t visualize data in HDFS directly using

dashboards or BI tools > because Hive is too slow (not interactive) > or ODBC connectivity is unavailable/unstable

> We needed to store daily-batch results to an interactive DB for quick response(PostgreSQL, Redshift, etc.) > Interactive DB costs more and less scalable by far

> Some data are not stored in HDFS > We need to copy the data into HDFS to analyze

HDFS

Hive

PostgreSQL, etc.

Daily/Hourly BatchInteractive query

CommercialBI Tools

Batch analysis platform Visualization platform

Dashboard

HDFS

Hive

PostgreSQL, etc.


✓ Less scalable ✓ Extra cost

CommercialBI Tools

Dashboard

✓ Extra work to manage 2 platforms

✓ Can’t query against “live” data directly

Batch analysis platform Visualization platform

HDFS

Hive Dashboard

Presto

PostgreSQL, etc.

Daily/Hourly Batch

HDFS

HiveDashboard

Daily/Hourly Batch

Interactive query

Interactive query

Data analysis platform

Presto

HDFS

HiveDashboard


Cassandra PostgreSQL Commertial DBs

SQL on any data sets


Presto

HDFS

HiveDashboard


Cassandra PostgreSQL Commertial DBs

SQL on any data sets CommercialBI Tools

✓ IBM Cognos ✓ Tableau ✓ ...


Prestogres

Presto

HDFS

Dashboard

Interactive query

CommercialBI Tools

✓ IBM Cognos ✓ Tableau ✓ ...

Prestogres

Today’s topic!

dashboard on chart.io: https://chartio.com/

https://chartio.com/

What can Presto do?> Query interactively (in milli-seconds to minues)

> MapReduce and Hive are still necessary for ETL > Query using commercial BI tools or dashboards

> Reliable ODBC/JDBC connectivity through Prestogres > Query across multiple data sources such as

Hive, HBase, Cassandra, or even internal DBs > Plugin mechanism

> Integrate batch analisys + visualizationinto a single data analysis platform

Presto’s deployment

> Facebook > Multiple geographical regions > scaled to 1,000 nodes > actively used by 1,000+ employees > who run 30,000+ queries every day > processing 1PB/day

> Netflix, Dropbox, Treasure Data, Airbnb, Qubole > Presto as a Service

Prestogres design of the ODBC/JDBC gateway

The problems to use Presto with BI tools

> BI tools need ODBC or JDBC connectivity > Tableau, IBM Cognos, QlickView, Chart.IO, ... > JasperSoft, Pentaho, MotionBoard, ...

> ODBC/JDBC is VERY COMPLICATED > Matured implementation needs LONG time

• psqlODBC: 58,000 lines • postgresql-jdbc: 62,000 lines • mysql-connctor-odbc: 27,000 lines • mysql-connector-j: 101,000 lines

A solution

> Creates a PostgreSQL protocol gateway > Uses PostgreSQL’s stable ODBC / JDBC driver

Other possible designs were…

a) MySQL protocol + libdrizzle: > Drizzle provides a well-designed library to implement

MySQL protocol server. > Proof-of-concept worked well:

• trd-gateway - MySQL protocol gateway for Hive > Difficulties: clients assumes the server is MySQL but,

• syntax mismatches: MySQL uses `…` while Presto “…” • function mismatches: DAYOFMONTH(…) vs EXTRACT(day…)

b) PostgreSQL + Foreign Data Wrapper (FDW): > JOIN and aggregation pushdown is not available

Other possible designs were…

c) PostgreSQL + H2 database + patch: > H2 is an embedded database engine written in Java > H2 has a PostgreSQL protocol implementation in Java > Difficulties:

• System catalog implementation is incomplete(pg_class, pg_namespace, pg_proc, etc.)

d) Reusing PostgreSQL protocol impl.: > Difficulties:

• complete implementation of system catalogs was too difficult

Prestogres design

pgpool-II + PostgreSQL + PL/Python > pgpool-II is a PostgreSQL protocol middleware for

replication, failover, load-balancing, etc. > pgpool-II originally has some useful code

(parsing SQL, rewriting SQL, hacking system catalogs, …) > Basic idea:

• Rewrite queries at pgpool-II and run Presto queries using PL/Python

select count(1)from access

select * frompython_func(‘select count(1) from access’)

rewrite!

Prestogres implementation

psql

pgpool-IIodbc

jdbc

PostgreSQL Presto

Authentication Create faked systemcatalogs for meta-queries

1. 2.

Rewriting queries Executing queries using PL/Python

3. 4.

Overview

Patched!

psql

pgpool-IIodbc

jdbc

PostgreSQL Presto

Authentication Create faked systemcatalogs for meta-queries

1. 2.

Rewriting queries Executing queries using PL/Python

3. 4.

Overview

Patched!

Prestogres

pgpool-IIpsql PostgreSQL Presto

$ psql -U me mydb

StartupPacket { database = “mydb”, user = “me”, … }


$ psql -U me mydb

prestogres.conf

system_db_dbname = ‘postgres’ system_db_user = ‘prestogres’

prestogres_hba.conf

host mydb me 0.0.0.0/0 trust presto_server presto.local:8080, presto_catalog hive, pg_database hive




$ psql -U me mydb

prestogres.conf


> CREATE DATABASE hive; > CREATE ROLE me; > CREATE FUNCTION setup_system_catalog; > CREATE FUNCTION start_presto_query;

libpq host=‘localhost’, dbname=‘postgres’, user=‘prestogres’ (system_db)

prestogres_hba.conf



$ psql -U me mydb

prestogres_hba.conf


prestogres.conf


StartupPacket { database = “hive”, user = “me”, … }


system catalog!


$ psql -U me mydb

“Q” SELECT * FROM pg_class;

"Query against a system catalog!”

Meta-query

system catalog!


$ psql -U me mydb

SELECT setup_system_catalog(‘presto.local:8080’, ‘hive’)“Q” SELECT * FROM pg_class;


Meta-query

PL/Python functiondefined at prestogres.py


$ psql -U me mydb

SELECT setup_system_catalog(‘presto.local:8080’, ‘hive’)“Q” SELECT * FROM pg_class;

> CREATE TABLE access_logs; > CREATE TABLE users; > CREATE TABLE events; …

Meta-query

SELECT * FROM information_schema.columns



$ psql -U me mydb

“Q” SELECT * FROM pg_class; “Q” SELECT * FROM pg_class;

Meta-query"Query against a system catalog!”


$ psql -U me mydb

“Q” select count(*) from access_logs;

regular table!

Presto Query"Query against a regular table!”


$ psql -U me mydb

“Q” select count(*) from access_logs; SELECT start_presto_query(… ‘select count(*) from access_logs’)

regular table!


PL/Python functiondefined at prestogres.py


$ psql -U me mydb

“Q” select count(*) from access_logs; SELECT start_presto_query(… ‘select count(*) from access_logs’)

> CREATE TYPE result_type (c0_ BIGINT); > CREATE FUNCTION fetch_results RETURNS SETOF result_type …

regular table!


1. start the query on Presto

2. define a function to fetch the result


$ psql -U me mydb

“Q” select count(*) from access_logs; “Q” SELECT * FROM fetch_results();


PL/Python functiondefined by start_presto_query

“Q” RAISE EXCEPTION …

Prestogres hacks

Multi-statement queries

BEGIN; SELECT 1; COMMIT;

Supporting Cursors

DECLARE CURSOR xyz FOR select …; FETCH

Security

security definer

Error message handling

raise exception ‘%’, E’…’ using errcode = …;

Faked current_database()

delete from pg_catalog.pg_proc where proname=‘current_database’;

create function pg_catalog.current_database()returns name as $$begin return ‘faked_name’::name;end$$ language plpgsql stable strict;

Future works

Future works

Rewriting CAST syntax

Extended query

CREATE TEMP TABLE

DROP TABLE

Check: www.treasuredata.com

Cloud service for the entire data pipeline, including Presto. We’re hiring!

Prestogres internals

Presentations & Public Speaking

Transcript of Prestogres internals