Integrating PostgreSQLReuven M. Lerner • reuven@lerner.co.il

Rails Israel Conference 2013October 9th, 2013 • Tel Aviv

1

Who am I?

• Web developer since 1993

• Linux Journal columnist since 1996

• PostgreSQL user since at least 1999

• Ruby/Rails developer, teacher since 2005

• PhD candidate in learning sciences at Northwestern University

2

But before all of that...I was a kid.

3

(Yes, a geeky kid.)

But before all of that...I was a kid.

3

4

5

6

7

Databases?!? Who needs them?

8

Er, I did. And do.

• Whadaya know? Databases are useful!

• They offer an abstraction layer for our data

• I don’t care how it is stored, so long as the data is safe, and I can get it back

9

What is a database?

Database

Store data confidently

Retrieve data flexibly

10

Relational databases

• Client-server

• All data is stored in two-dimensional tables

• Client-server communication is in SQL

• SQL is standard. Each database implements a superset of a subset of that standard.

11

“Impedance mismatch”

• You basically couldn’t find two languages that are more different from one another than Ruby and SQL:

• Ruby — object oriented, imperative

• SQL — table oriented, declarative

• Mixing them is ugly, as we all know

12

ActiveRecord!

• ActiveRecord is the default ORM (object-relational manager) in Rails

• ActiveRecord translates our Ruby method calls to SQL queries

• Query results are turned into Ruby objects and collections

13

The good news...

• ActiveRecord papers over the differences between databases!

14

... and the bad news

• ActiveRecord papers over the differences between databases!

15

The really bad news

• This means we’re:

• Writing extra code

• Slowing down our apps

• Putting logic in the wrong place

• Database logic shouldn’t be in your app (just like SQL shouldn’t be in your HTML)

16

PostgreSQL

17

PostgreSQL

17

PostgreSQL

17

PostgreSQL

17

PostgreSQL

17

PostgreSQL

17

PostgreSQL

• Very fast, very scalable.

• Amazingly flexible, easily extensible.

• Rock-solid — no crashes, corruption, major security issues for years

• Ridiculously easy administration

• It also happens to be free (MIT/BSD)

18

Brief history

• Ingres (Stonebreaker, Berkeley)

• Postgres (Stonebreaker, Berkeley)

• PostgreSQL project = Postgres + SQL

• About one major release per year

• Latest and greatest is 9.3

19

Features!

• MVCC

• Custom data types

• CTEs

• Functional, partial indexes

• Full-text search

• Server-side functions

20

Rails 4 to the rescue!

• As of Rails 4, many PostgreSQL features are supported, out of the box, by ActiveRecord

• The best of both worlds: Amazing database features, and we don’t have to use SQL!

21

But wait!

• What if I want to switch to another database?

• Won’t I lose the platform independence?

• Yes, you will.

• Fortunately, this will almost certainly not happen.

• And if it does, it’ll be super-painful anyway.

22

Wait again!

• This means that I can’t use SQLite in development, and PostgreSQL on the server.

• Yes, that’s right.

• You should use the same database in development and production. Please.

23

Data types

• Databases are like programming languages: The right choice of data type can have huge consequences in speed, size, and flexibility

• PostgreSQL has some special ones that can really come in handy!

24

Network types

• PostgreSQL supports INET, CIDR, and MACADDR types

• If you’re ever storing network addresses, you should use these, not a text type!

• Savings in space (and time), and faster comparisons (numeric vs. textual)

25

CREATE TABLE Logins (

id SERIAL PRIMARY KEY,

local_ip INET NOT NULL,

remote_ip INET NOT NULL);

INSERT INTO Logins (local_ip, remote_ip) values ('192.168.1.1', '10.1.2.3');

reuven=# INSERT INTO Logins (local_ip, remote_ip) values ('999.999.999.1000', '10.1.2.3');

ERROR: invalid input syntax for type inet: "999.999.999.1000"

LINE 1: INSERT INTO Logins (local_ip, remote_ip) values ('999.999.99...

26

Rails 4 migrations$ rails g model login local_ip:inet remote_ip:inet

class CreateLogins < ActiveRecord::Migration

def change

create_table :logins do |t|

t.inet :local_ip, nulls: :false

t.inet :remote_ip, nulls: :false

t.timestamps

end

end

end

27

Rails 4 appLogin.create!(local_ip:'192.168.1.1', remote_ip:'10.1.2.3')

Login.create!(local_ip:'192.168.1.1000', remote_ip:'10.1.2.3')

IPAddr::InvalidAddressError: invalid address

i = Login.first.local_ip

i.class

=> IPAddr

i.ipv4?

=> true

i.ipv6?

=> false

28

Managing a network?

• We also have CIDR and MAC addresses!

Netconfig.create!(net: '192.168.1.1/255.255.255.0')

29

UUIDs

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

CREATE TABLE evil_companies (

id UUID NOT NULL PRIMARY KEY DEFAULT uuid_generate_v1(),

name TEXT);

INSERT INTO evil_companies (name) VALUES ('HOT Internet');

INSERT INTO evil_companies (name) VALUES ('HOT TV');

INSERT INTO evil_companies (name) VALUES ('HOT Telephone');

INSERT INTO evil_companies (name) VALUES ('Voldemort, Ltd.');

30

Wait... extensions?

• Yes, PostgreSQL has extensions!

• Download and install them from PGXN

• (Think of them as gems for PostgreSQL)

• Some, like uuid-ossp, come by default

• Activate them with CREATE EXTENSION

31

32

With this in place...

SELECT * FROM evil_companies ;

┌──────────────────────────────────────┬─────────────────┐

│ id │ name │

├──────────────────────────────────────┼─────────────────┤

│ da7a333a-3069-11e3-b4fd-28cfe91f81e7 │ HOT Internet │

│ dc480be2-3069-11e3-a0f4-28cfe91f81e7 │ HOT TV │

│ ddadabfe-3069-11e3-af1d-28cfe91f81e7 │ HOT Telephone │

│ 4c33a128-306a-11e3-8000-28cfe91f81e7 │ Voldemort, Ltd. │

└──────────────────────────────────────┴─────────────────┘

33

Migrations

# Requires some manual tinkering

class CreateEvilCompanies < ActiveRecord::Migration

def change

enable_extension "uuid-ossp"

create_table :evil_companies, id: :uuid do |t|

t.text :name

t.timestamps

end

end

end

34

Migrations

# Requires some manual tinkering

class CreateEvilCompanies < ActiveRecord::Migration

def change

enable_extension "uuid-ossp"

create_table :evil_companies, id: :uuid do |t|

t.text :name

t.timestamps

end

end

end

34

In the app

EvilCompany.create!(name: 'HOT Internet')

EvilCompany.first.id

=> "07d3f537-c008-4006-9f2b-0592c7df3ebf"

35

JSON

• Remember the “impedance mismatch” between objects and databases?

• Now we have another one — we get data in JSON, turn it into database objects, and then turn it back into JSON

• But now PostgreSQL can store, retrieve, and query JSON!

36

CREATE TABLE bad_service (

id SERIAL,

info JSON

);

INSERT INTO bad_service (info)

values ('{"company":"HOT Internet", "badness":9}');

INSERT INTO bad_service (info) VALUES ('la la la');

ERROR: invalid input syntax for type json

LINE 1: insert into bad_service (info) values ('la la la');

^

DETAIL: Token "la" is invalid.

CONTEXT: JSON data, line 1: la...

37

JSON operators!

• As of PostgreSQL 9.3, we get operators

• Work with JSON, much like XML:SELECT info#>'{company}' from bad_service;┌────────────────┐│ ?column? │├────────────────┤│ "HOT Internet" │└────────────────┘

38

Migration

$ rails g model bad_service info:json

class CreateBadServices < ActiveRecord::Migration

def change

create_table :bad_services do |t|

t.json :info

t.timestamps

end

end

end

39

App

BadService.create!(info: '{"company":"HOT Internet"}')

b = BadService.first

b.info.class

=> Hash

b.info['company']

=> "HOT Internet"

40

Ranges

CREATE TABLE forecast (

forecast_for DATE,

hi_lo int4range);

INSERT INTO forecast (forecast_for, hi_lo) VALUES ('8-oct-2013', '[19,29]');

INSERT INTO forecast (forecast_for, hi_lo) VALUES ('9-oct-2013', '[20,32]');

INSERT INTO forecast (forecast_for, hi_lo) VALUES ('10-oct-2013', '[19,28]');

41

SELECT * FROM forecast ;

┌──────────────┬─────────┐

│ forecast_for │ hi_lo │

├──────────────┼─────────┤

│ 2013-10-08 │ [19,30) │

│ 2013-10-09 │ [20,33) │

│ 2013-10-10 │ [19,29) │

└──────────────┴─────────┘

SELECT * FROM forecast WHERE hi_lo @> 19;

┌──────────────┬─────────┐

│ forecast_for │ hi_lo │

├──────────────┼─────────┤

│ 2013-10-08 │ [19,30) │

│ 2013-10-10 │ [19,29) │

└──────────────┴─────────┘

42

Migration$ model forecast forecast_for:date hi_lo:int4range

class CreateForecasts < ActiveRecord::Migration

def change

create_table :forecasts do |t|

t.date :forecast_for

t.int4range :hi_lo

t.timestamps

end

end

end

43

Ranges in apps

Forecast.create!(forecast_for: Time.now, hi_lo: 20..30)

Forecast.first.hi_lo

=> 20...31

Forecast.first.hi_lo.class

Range

44

Arrays!

CREATE TABLE posts (

body text,

tags text[]

);

INSERT INTO posts (body, tags) VALUES ('Hello, world', '{intro, hello}');

INSERT INTO posts (body, tags) VALUES ('Hello again', '{DRY, boring, hello}');

45

SELECT * FROM posts;

┌──────────────┬────────────────────┐

│ body │ tags │

├──────────────┼────────────────────┤

│ Hello, world │ {intro,hello} │

│ Hello again │ {DRY,boring,hello} │

└──────────────┴────────────────────┘

SELECT * from posts where 'intro' = ANY(tags);

┌──────────────┬───────────────┐

│ body │ tags │

├──────────────┼───────────────┤

│ Hello, world │ {intro,hello} │

└──────────────┴───────────────┘

46

Arrays in migrations$ rails g model post body:text tags:text

class CreatePosts < ActiveRecord::Migration

def change

create_table :posts do |t|

t.text :body

t.text :tags, array: true

t.timestamps

end

end

end

47

Arrays in your app

Post.create!(body: 'First post!', tags: %w(first second third))

Post.first.tags

=> ["first", "second", "third"]

48

Normalization = DRY

• Everyone loves to talk about tagging as a great example of PostgreSQL arrays

• Um... don’t forget about normalization, the DRY of database design

• I know, it’s not cool to talk about it in the Age of NoSQL. But it really does work.

49

Premature denormalization is the

root of all database evil.

50

Hstore

• A key-value storage system in PostgreSQL!

• Any column can be defined as hstore

• Then you can query it — and you get a hash back!

• It’s sort of like Redis or memcached, but integrated into (and backed by) Postgres!

51

PostgreSQL

CREATE EXTENSION hstore;

CREATE TABLE posts2 (

body text,

tags hstore

);

INSERT INTO posts2 (body, tags)

VALUES ('Hello, there', '"hello" => 2, "boring" => 10');

INSERT INTO posts2 (body, tags)

values ('Hello again', '"DRY" => 5,

"hello" => 2, "boring" => 3');

52

SELECT * from posts2 where defined(tags, 'hello');

┌──────────────┬─────────────────────────────────────────┐

│ body │ tags │

├──────────────┼─────────────────────────────────────────┤

│ Hello, there │ "hello"=>"2", "boring"=>"10" │

│ Hello again │ "DRY"=>"5", "hello"=>"2", "boring"=>"3" │

└──────────────┴─────────────────────────────────────────┘

(2 rows)

SELECT tags -> 'boring' from posts2 ;

┌──────────┐

│ ?column? │

├──────────┤

│ 10 │

│ 3 │

└──────────┘

(2 rows)

53

And also...

SELECT * FROM posts2

WHERE (tags -> 'boring')::integer > 5;

┌──────────────┬──────────────────────────────┐

│ body │ tags │

├──────────────┼──────────────────────────────┤

│ Hello, there │ "hello"=>"2", "boring"=>"10" │

└──────────────┴──────────────────────────────┘

54

Migration$ rails g model newpost body:text tags:hstore

class CreateNewposts < ActiveRecord::Migration

def change

enable_extension "hstore"

create_table :newposts do |t|

t.text :body

t.hstore :tags

t.timestamps

end

end

end

55

In your app

Newpost.create!(:body => 'foo', :tags => {a:1, b:2})

Newpost.first.tags

=> {"a"=>"1", "b"=>"2"} # hash!

Newpost.first.tags['a']

=> "1" # now a string!

Newpost.first.tags[:a]

=> nil # not WithIndifferentAccess!

56

Indexes

• PostgreSQL offers different index types

• Specify these in your migrations

• Partial indexes (with a WHERE clause)

• Functional indexes (apply a function!)

• Full-text indexes (many languages)

• Geospatial indexes (install PostGIS!)

57

Some things are still missing

• “The Active Record way claims that intelligence belongs in your models, not in the database. As such, features such as triggers or foreign key constraints, which push some of that intelligence back into the database, are not heavily used.”

• — Rails Guide, ActiveRecord Migrations

58

Sigh.

59

60

Your data are your crown jewels!

61

Summary

• Rails good! PostgreSQL good!

• Rails + PostgreSQL = Doubleplus good!

• Rails 4 + PostgreSQL = Secure storage, flexible retrieval, and a lot of geeky DB fun!

62

Thanks!(Any questions?)

reuven@lerner.co.ilhttp://www.lerner.co.il/

@reuvenmlerner

054-496-8405“reuvenlerner” on Skype

63