High availability for puppet - 2016
-
Upload
zack-smith -
Category
Technology
-
view
645 -
download
1
Transcript of High availability for puppet - 2016
High Availability for Puppet
Russ Mull - @mullr
Senior Software Engineer
Zack Smith - @acidprime
Principal Professional Services Engineer
Enterprise Readiness: High Availability
Auto scaling
Active/ Active
Robust Backup and
Restore tooling
Disaster recovery
in Multi datacenter/
Geo diverse
environments
Eliminate Single
Points of Failure
Building capabilities that matter
Puppet
Runs continue
High Availability for Puppet - Puppetconf 2016
Building a new catalogClassification, Exported Records, Hiera Data and puppet code being synced from version control
4
run
Puppet Code
Classifier
High Availability for Puppet - Puppetconf 2016
Two of Everything!don’t forget about your external services like git, ldap etc
5
Cl
as
sifi
erPup
pet
Cod
e
Classifier
Pup
pet
Cod
eClassifier
Balancer“Load”
check
check backup
High Availability for Puppet - Puppetconf 2016
Puppet CA Replication Components
CA private key and cert
Signed Directory
Serial file
Certificate Revocation List (CRL)
8
serial
3E8
crl.pem
signed
101
ca_crt
High Availability for Puppet - Puppetconf 2016
CA Private key
ca
If you don’t care about revocation (security/revocation) this is the only file needed to replicate
9
ca_key.pem
ssl
ca_crt.pem
High Availability for Puppet - Puppetconf 2016
Signed Directory
signed
Used when checking for duplicate CN ( certs with the same name)
10
host1.company.com.pem
ca
High Availability for Puppet - Puppetconf 2016
Serial fileTracking the next numeric serial to be issued to new agent
12
serial
ca
3E8
decimal: 1000
decimal: 1001
3E9
High Availability for Puppet - Puppetconf 2016
Certificate Revocation ListTracking revoked certificates
13
crl.pem
ca
decimal: 1000
decimal: 1001
1000
1001
serial
3E9
High Availability for Puppet - Puppetconf 2016
Simply copy your ssldir ahead of the second installation
14
$ssldir
scp -r
$ssldir
installer
CA
When using old versions of PE delete the pe-internal* certs post transfer , pre install
DR site
1.
2.
High Availability for Puppet - Puppetconf 2016
Streaming ReplicationThis happens as the postgres database layer
18
PostgreSQL PostgreSQL
PuppetDBPDBPuppetDBPDB
22
Read (Standby)Write
5432
write ahead logs
R
W1
2
16MB
High Availability for Puppet - Puppetconf 2016
Split Reads and writesCan survive temporary failures of the write master
19
PostgreSQL PostgreSQL
PuppetDBPDB
Read (Standby)Write
5432
W R
WWW
queue
High Availability for Puppet - Puppetconf 2016
Promote Standby to WritableThis happens as the postgres database layer
20
PostgreSQL
WriteRead (standby)
PostgreSQL
PostgreSQL
Write
5432
PostgreSQL
Read (standby)
5432
High Availability for Puppet - Puppetconf 2016
Master side FailoverThis is know as “terminus” failover as its handled in the puppetdb terminus package code
22
PuppetDBPDB
[main]
server_urls = https://primary:8081, https://replica:8081
Primary
PuppetDBPDB
Replica1 2
High Availability for Puppet - Puppetconf 2016
Command Broadcastcommand_broadcast = true in puppetdb.conf
23
PuppetDBPDB
PuppetDBPDB
High Availability for Puppet - Puppetconf 2016
PuppetDB Replication ReconciliationReconciliation happens on an interval
24
PostgreSQL
8081
Write
PostgreSQL
PuppetDBPDB
Write
PuppetDBPDB
Sync Interval
High Availability for Puppet - Puppetconf 2016
Simple HAMonolithic master + Replica
26
P Primary R Replica
High Availability for Puppet - Puppetconf 2016
Large Environment InstallationMonolithic master + Compile masters + Replica
27
R Replica
BalancerLoad
P Primary
BalancerLoad
High Availability for Puppet - Puppetconf 2016
New: Agent Side Failover!Shipping in Puppet 4.6 and higher, PE 2016.4+
28
Primary Replica
1 2
High Availability for Puppet - Puppetconf 2016
Use Cases
● Puppet runs keep working
● Promote replica to master
29
When master is unreachable
High Availability for Puppet - Puppetconf 2016
When the master is unreachable
● Run puppet
● Promote replica to master
30
You can:
You can’t● Change classification
● Deploy new puppet code
● Issue new certs
● Use the Puppet Enterprise Console
● Use Application Orchestrator
High Availability for Puppet - Puppetconf 2016
Command Line Interface
32
puppet infra provision replica <hostname>
replica.mycorp.net
puppet infra enable replica
replica.mycorp.net
puppet infra status. . .
High Availability for Puppet - Puppetconf 2016
1. Provision Replica
33
puppet infra provision replica <hostname>
replica.mycorp.net
High Availability for Puppet - Puppetconf 2016
2. Monitor status of replication
34
puppet infra status
> Per-service ‘alerts’
> Visible in the UI as well
High Availability for Puppet - Puppetconf 2016
3. Enable replica
35
puppet infra enable replica
replica.mycorp.net
High Availability for Puppet - Puppetconf 2016
What’s a replica?
● Compile Master
● PuppetDB (r/w)
● RBAC, classifier, activity (r/o)
● Orchestrator data (not running)
● CA data (r/o using a proxy)
37
R Replica
PuppetDBPDB
Console
High Availability for Puppet - Puppetconf 2016
File Sync - Compile Masters
M
Master of Masters
MOM
Compile Master
COMC
Compile Master
COMC
Compile Master
COMC
High Availability for Puppet - Puppetconf 2016
File Sync - CA Replica data
Primary Master Replica Master
R ReplicaP Primary
8140
ssl ssl
High Availability for Puppet - Puppetconf 2016
Puppet Enterprise CA Proxy
Primary Master Replica Master
R ReplicaP Primary
sslCSR
High Availability for Puppet - Puppetconf 2016
PGLogical Replication
PostgreSQL
RBAC
NC
Classifier5432
5432
PostgreSQL
RBAC
NC
Classifier
Write Read (Standby)
PuppetDB
PDB
PuppetDB
PDBNot synced
High Availability for Puppet - Puppetconf 2016
PE HA - Replication
PuppetDB
PuppetDBPDB PuppetDBPDB
PostgreSQLPostgreSQL
PGlogical
PGlogical
FileSync
Primary Replica
RBAC
NC
Classifier
W
W
RBAC
NC
Classifier
R
R
High Availability for Puppet - Puppetconf 2016
Puppet Enterprise HA - Beta Signup
Interested in what you heard?
Please signup for our HA beta program through the Puppet Enterprise Support portal
45
High Availability for Puppet - Puppetconf 2016
46
https://goo.gl/Z85HLS
PE HA Beta Signup
Support Knowledge base
Z 8 5 H L S