UI Improvements - Dan Clarizio, Eric Winchell - ManageIQ Design Summit 2016
Replication - Nick Carboni - ManageIQ Design Summit 2016
-
Upload
manageiq -
Category
Technology
-
view
364 -
download
4
Transcript of Replication - Nick Carboni - ManageIQ Design Summit 2016
Database Replication in ManageIQ
Introduction
Nick Carboni
Software Engineer at Red Hat
Appliance and database for ManageIQ (Platform Team)
Replication in ManageIQ
Multiple databases can be used for far away environments
Database replication allows for easy reporting and visibility
Database 1 Database 2
Global Database
Replication!
rubyrep
http://www.rubyrep.org
https://github.com/rubyrep/rubyrep
Last commit in 2011
rubyrep
Ruby + ActiveRecord
Triggers = Developer headaches
SQL
Configuration is … hardelsif commands.include? command status = commands[command][:command].run(args.slice(1, 1_000_000))
rubyrep in ManageIQ
We use our own fork
https://github.com/ManageIQ/rubyrep
Mainly support for new versions of ActiveRecord
rubyrep in ManageIQPR to add support for Rails 4.2
March 27, 2015
“Yup. It's basically dead. Our changes are also SOOO crazy different, that this is like a Frankenstein project now. I wonder if we should just write something different from scratch given our experience...or perhaps we should rethink replication altogether and come up with a plan.”
-@Fryguy
rubyrep in ManageIQ
Worker process
Database Synchronization role
Worker will run on the server with the role
Passive global database
pglogical
http://2ndquadrant.com/en-us/resources/pglogical/
https://github.com/2ndQuadrant/pglogical/
Inclusion into PostgreSQL core in progress
9.7+??
pglogical
PostgreSQL extension
Process runs on database server
Managed by PostgreSQL service
SQL stored procedure API
postgresql.conf configuration
pglogicalPostgreSQL logical decoding
https://www.postgresql.org/docs/9.4/static/logicaldecoding.html
Output plugin
pglogical_output
Consumer extension
pglogical
Replication slots
https://www.postgresql.org/docs/9.4/static/warm-standby.html#STREAMING-REPLICATION-SLOTS
pglogical in ManageIQ
AR connection adapter extension
Exposes a ruby method per pglogical stored procedure
Configure “subscriptions” to remote regions through global region UI
Exclude table configuration unchanged
“Advanced Settings” tab in remote region UI
Global Region Configuration
Performance testing
How long does it take to replicate a “backlog” of data?
Vary amount of data and artificial network latency
Measure time until “backlog” is zero
rubyrep - number of rows in rr_pending_changes table
pglogical - transaction log location compared to flush location
SELECT pg_xlog_location_diff(pg_current_xlog_location(), flush_location) AS lag_bytes FROM pg_stat_replication;
Also record time to insert test rows
Insert Performance
Rows pglogical rubyrep
1,000 0.158 (σ = 0.006) 0.189 (σ = 0.011)
10,000 1.499 (σ = 0.095) 1.84 (σ = 0.093)
100,000 16.105 (σ = 0.296) 18.042 (σ = 0.320)
1,000,000 147.508 (σ = 3.557) 195.399 (σ = 18.317)
Sample size = 10
No Latency
25ms Latency (50ms ping time)
100ms Latency (200ms ping time)
Q&A