Postgres & Redis Sitting in a Tree- Rimas Silkaitis, Heroku
-
Upload
redis-labs -
Category
Technology
-
view
251 -
download
3
Transcript of Postgres & Redis Sitting in a Tree- Rimas Silkaitis, Heroku
Rimas Silkaitis
Postgres & Redis Sitting in a Tree
App
Workers
App
http://oldblog.antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html
What’s changed since then?
redis_fdw
https://github.com/pg-redis-fdw/redis_fdw
redis_fdw
Foreign Data Wrapper
FDW
By using redis_fdw
•Cache results without leaving Postgres
•Cross reference data
•Reduce complexity in app code
•Maybe even replace PG functionality with that of Redis
app cloud
DEPLOY MANAGE SCALE
$ git push heroku master
Counting objects: 11, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (11/11), 22.29 KiB | 0 bytes/s, done.
Total 11 (delta 1), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Ruby app detected
remote: -----> Compiling Ruby
remote: -----> Using Ruby version: ruby-2.3.1
Rimas Silkaitis
Product
Heroku PostgresOver 1 Million Active DBs
Heroku RedisOver 100K Active Instances
Heroku Kafka
Configuring redis_fdw
neovintage::DB=> CREATE EXTENSION redis_fdw;
CREATE EXTENSION
neovintage::DB=> CREATE SERVER redis_server
neovintage::DB-> FOREIGN DATA WRAPPER redis_fdw
neovintage::DB-> OPTIONS (
neovintage::DB-> ADDRESS ‘127.0.0.1’,
neovintage::DB-> PORT ‘6379’
neovintage::DB-> );
CREATE SERVER
neovintage::DB=> CREATE USER MAPPING FOR PUBLIC
neovintage::DB-> SERVER redis_server OPTIONS (password ‘pass’);
CREATE USER MAPPING
$ heroku pg:links create DATABASE_URL REDIS_URL —as redis_db -a sushi
neovintage::DB=> CREATE FOREIGN TABLE redis_scalar (
neovintage::DB-> key text,
neovintage::DB-> value text
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS (
neovintage::DB-> database ‘0’
neovintage::DB-> );
CREATE FOREIGN TABLE
redis> SET presentation awesome
OK
redis>
neovintage::DB=> SELECT * from redis_scalar;
key | value
--------------+-------
presentation | awesome
(1 row)
More Options for PG Tables
• tabletype
• tablekeyprefix
• tablekeyset
• singleton_key
tabletype 'hash'
neovintage::DB=> CREATE FOREIGN TABLE redis_hash (
neovintage::DB-> key text,
neovintage::DB-> value text[]
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS ( database ‘0’, tabletype ‘hash’ );
CREATE FOREIGN TABLE
neovintage::DB=> SELECT * from redis_hash;
key | value
---------+------------
awesome | {today,10}
(1 row)
neovintage::DB=> SELECT key, json_object(value) from redis_hash;
key | json_object
---------+---------------
awesome | {“today”: “10”}
(1 row)
redis> HSET awesome today 10
1
Column Value Type redis_fdw return value
text[] array of text
text text as array
tabletype 'list'
neovintage::DB=> CREATE FOREIGN TABLE redis_list (
neovintage::DB-> key text,
neovintage::DB-> value text[]
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS ( database ‘0’, tabletype ‘list’ );
CREATE FOREIGN TABLE
neovintage::DB=> SELECT * from redis_list;
key | value
----------+------------
mylist | {hello,world}
yourlist | {awesome}
(2 row)
redis> RPUSH mylist “hello”
1
redis> RPUSH mylist “world”
1
redis> RPUSH yourlist “awesome”
1
tabletypes set and zset are similar to list
tablekeyset
neovintage::DB=> CREATE FOREIGN TABLE redis_set (
neovintage::DB-> value text
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS (
neovintage::DB-> database ‘0’,
neovintage::DB-> tabletype ‘set’,
neovintage::DB-> tablekeyset ‘myset’
neovintage::DB-> );
CREATE FOREIGN TABLE
neovintage::DB=> SELECT * from redis_set;
value
-------
hello
world
(2 row)
redis> SADD myset “hello”
1
redis> SADD myset “world”
1
http://oldblog.antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html
Counting Things
redis> INCR user:<id>
redis> EXPIRE user:<id> 60
neovintage::DB=> CREATE FOREIGN TABLE redis_counts (
neovintage::DB-> user_id text,
neovintage::DB-> count bigint
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS (
neovintage::DB-> database ‘0’,
neovintage::DB-> tablekeyprefix ‘user:’
neovintage::DB-> );
CREATE FOREIGN TABLE
neovintage::DB=> SELECT * from redis_counts;
user_id | count
----------+------------
user:2 | 10
user:3 | 200
(2 row)
GROSS
neovintage::DB=> SELECT * from user_counts;
user_id | count
----------+------------
2 | 10
3 | 200
(2 row)
neovintage::DB=> CREATE VIEW user_counts AS
neovintage::DB-> SELECT split_part(user_id, ‘:’, 2) as user_id
neovintage::DB-> , count
neovintage::DB-> FROM redis_counts;
neovintage::DB=> INSERT INTO user_count_snapshots
neovintage::DB-> (created_at, user_id, count)
neovintage::DB-> SELECT date_trunc(‘hour’, now())
neovintage::DB-> , user_id
neovintage::DB-> , count
neovintage::DB-> FROM user_counts;
Benefits
• Cross reference data in Postgres with high velocity information
• Issue one query to take snapshots of counts in Redis. Make data warehousing easier.
Slow Queries
SELECT * FROM foo WHERE ... ORDER BY rank DESC LIMIT 10
Workers
App
psudeo-codeFUNCTION get_top_commenters(): list = redis.get(“top:comments”) time = redis.get(“top:comments:refresh_time”) IF (Time.now - time) > 90 mutex do list = SQL_DB("SELECT ... ORDER BY rank LIMIT …”)
redis.set(“top:comments”, list) redis.set(“top:comments:refresh_time”, Time.now)
end END RETURN list END
neovintage::DB=> \d users
Table "public.users"
Column | Type | Modifiers
----------+---------+----------------------------------------------------
id | bigint | not null default nextval('users_id_seq'::regclass)
name | text |
comments | integer |
neovintage::DB=> select * from users;
id | name | comments
-----+---------+----------
1 | rimas | 10
2 | chuck | 10000
3 | lucy | 300
neovintage::DB=> CREATE FOREIGN TABLE top_commenters (
neovintage::DB-> cache_key text,
neovintage::DB-> commenter text[]
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS (
neovintage::DB-> database ‘0’,
neovintage::DB-> tabletype ‘list’
neovintage::DB-> );
CREATE FOREIGN TABLE
neovintage::DB=> INSERT INTO top_commenters (cache_key, commenter)
neovintage::DB-> SELECT ‘mylist’
neovintage::DB-> , array_agg(name)
neovintage::DB-> FROM users
neovintage::DB-> GROUP BY 1;
INSERT 0 1
redis> LRANGE mylist 0 3
1) chuck
2) lucy
3) rimas
neovintage::DB=> UPDATE top_commenters
set commenters = subquery.names
from (select array_agg(name) as names
from users) AS subquery
;
What if we need to show score or count?
zset
neovintage::DB=> CREATE FOREIGN TABLE top_commenters (
neovintage::DB-> value text,
neovintage::DB-> score numeric
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS (
neovintage::DB-> database ‘0’,
neovintage::DB-> tabletype ‘zset’,
neovintage::DB-> singleton_key ‘mycache’
neovintage::DB-> );
CREATE FOREIGN TABLE
neovintage::DB=> CREATE FOREIGN TABLE top_commenters (
neovintage::DB-> value text,
neovintage::DB-> score numeric
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS (
neovintage::DB-> database ‘0’,
neovintage::DB-> tabletype ‘zset’,
neovintage::DB-> singleton_key ‘mycache’
neovintage::DB-> );
CREATE FOREIGN TABLE
What if we need to update results?
neovintage::DB=> INSERT INTO top_commenters (cache_key, commenter)
neovintage::DB-> SELECT ‘mylist’
neovintage::DB-> , array_agg(name)
neovintage::DB-> FROM users
neovintage::DB-> GROUP BY 1;
ERROR: key already exists: mylist
UPSERT?
neovintage::DB=> INSERT INTO redis_set (cache_key, commenters)
(SELECT ‘mylist’
, array_agg(name) as top_commenters
FROM users
GROUP BY 1) as subquery
ON CONFLICT (cache_key)
DO UPDATE SET value = subquery.top_commenters
WHERE cache_key = ‘mylist’;
ERROR: there is no unique or exclusion constraint matching the ON
CONFLICT specification
Challenges
• You will get errors if keys already exist in Redis
• sets, lists, zsets can be more of a challenge to update in place if you have many processes trying to grab the same key
• Unique constraints on foreign tables in postgres aren’t a thing :-(
Tips
• Updates work well for scalar keys
• Atomic updates to lists, sets and zsets will require some creativity
• If you’re going to use zset, try the singleton_key when defining the foreign table in postgres.
• Get rid of nasty mutex code by running a cron job on a periodic basis that executes update queries.
Caveats
•Postgres 9.3+
•Redis 2.8+
Redis GEO
Is it possible to replace Postgis with Redis GEO ?
¯\_( )_/¯
zset under the hood
redis> GEOADD Sicily 13.361389 38.115556 "Palermo" 15.087269 37.502669
“Catania"
2
redis> GEOHASH Sicily Palermo
sqc8b49rny0
redis> GEOHASH Sicily Catania
sqdtr74hyu0
neovintage::DB=> CREATE FOREIGN TABLE redis_geo (
neovintage::DB-> city text,
neovintage::DB-> geohash text
neovintage::DB-> )
neovintage::DB-> SERVER redis_server
neovintage::DB-> OPTIONS (
neovintage::DB-> database ‘0’,
neovintage::DB-> tabletype ‘zset’,
neovintage::DB-> singleton_key ‘Sicily’ );
CREATE FOREIGN TABLE
neovintage::DB=> SELECT * FROM redis_geohash;
city | geohash
----------+------------
Palermo | sqc8b49rny0
Catania | sqdtr74hyu0
•Still need to install Postgis (geohash functions)
•Can’t use any of the GEO functions from redis
Maybe someday
Thank You!Rimas Silkaitis / neovintage.org / @neovintage