Post on 16-Apr-2017
Indiana Jonesand the Temple of NFS Ganesha HA
Kaleb KeithleyGluster Summit, Berlin
6 October, 2016
These aren't the droids you're looking for
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
Setup Pacemaker, Gluster, and create volume
But I was going into Tosche Station to pick up some power converters!
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
My HA Cluster
Create HA Cluster
Don't call me a mindless philosopher, you overweight glob of grease.
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
Create Resource Agents
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
I find your lack of faith disturbing.
Set Constraints
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha-active ?
grace-active ?
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha-active ?
grace-active ?pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha-active ?
grace-active ?
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha-active ?
grace-active ?
Fast ship? You've never heard of the Millennium Falcon?
Start ganesha.nfsd
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
If there's a bright center to the universe, you're on the planet that it's farthest from.
ganesha_mon notices ganesha.nfsd is running
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-activeganesha-grace
Help me, Obi-Wan Kenobi; you're my only hope.
Ganesha.nfsd dies
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-activeganesha-grace
I used to bullseye womp rats in my T-16 back home, they're not much bigger than two meters.
ganesha_mon deletes grace-active
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-active
In my experience, there's no such thing as luck.
surviving ganesha.nfsds placed into NFS-GRACE
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-active
Who's the more foolish? The fool, or the fool who follows him?
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha_mon deletes ganesha-active
You came in that thing? You're braver than I thought.
VIP fails over
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsdblock
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
VIP2
You can't win, Darth. If you strike me down I will become more powerful than you could possibly imagine.
Next Steps
● Switch to storhaug (anyone fancy a trip to Norway?)
● Fencing?
I've got a bad feeling abou this.
● https://github.com/linux-ha-storage/storhaug● http://clusterlabs.org/
I've got to rest before I fall apart
Indiana Jonesand the Temple of NFS Ganesha HA
Kaleb KeithleyGluster Summit, Berlin
6 October, 2016
These aren't the droids you're looking for
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
Setup Pacemaker, Gluster, and create volume
But I was going into Tosche Station to pick up some power converters!
Four node clusterPacemaker + Corosync runningGlusterd running + four brick volume is started
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
pacemaker
glusterd
glusterfsd
My HA Cluster
Create HA Cluster
Don't call me a mindless philosopher, you overweight glob of grease.
Glusterd invokes ganesha-ha.sh –setupSetup create the pacemaker cluster, plus some other
pacemaker voodoo magic, e.g.+ authenticates w/ all members+ creates cluster w/ name and members+ starts cluster+ waits for cluster to elect a domain controller (DC)+ sets quorum policy if < 3 members+ disables stonith...
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
Create Resource Agents
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
I find your lack of faith disturbing.
Creates resources (resource agents)+ Floating (or virtual) IPs (IPaddr) and
portblock/unblock+ ganesha_mon, ganesha_grace, ganesha_nfsd
What they do:+ ganesha_mon checks to see if ganesha.nfsd is
running or not+ ganesha_grace sends dbus signals to put
ganesha.nfsd into grace+ ganesha_nfsd fixes up /var/lib/nfs/. (but storehaug)
Then lots of pacemaker constraint location voodoo to make VIPs fail over in a deterministic manner
Set Constraints
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha-active ?
grace-active ?
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha-active ?
grace-active ?pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha-active ?
grace-active ?
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha-active ?
grace-active ?
Fast ship? You've never heard of the Millennium Falcon?
VIPs want to be where ganesha-active attribute existsganesha_grace run when ganesha-grace attribute
exists.
ganesha_grace uses a pacemaker feature called “notify” available to clone resources. If ganesha_grace stops on node2, the other ganesha_grace RAs are notified.
Start ganesha.nfsd
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
If there's a bright center to the universe, you're on the planet that it's farthest from.
After everything is set up, gluster/glusterd starts the ganesha.nfsds on all the nodes
ganesha_mon notices ganesha.nfsd is running
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-activeganesha-grace
Help me, Obi-Wan Kenobi; you're my only hope.
ganesha_mon sees that ganesha.nfsd is running and creates ganesha-active and ganesha-grace attributes.
VIP RAs get happy, clients mount using the Floating IP addrs.
Time passes…...
Ganesha.nfsd dies
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-activeganesha-grace
I used to bullseye womp rats in my T-16 back home, they're not much bigger than two meters.
Oops, one of the ganesha.nfsds died.
ganesha_mon deletes grace-active
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-active
In my experience, there's no such thing as luck.
ganesha_mon notices and deletes the ganesha-grace attr.
surviving ganesha.nfsds placed into NFS-GRACE
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha-active
Who's the more foolish? The fool, or the fool who follows him?
Pacemaker notifies the other ganesha_grace RAs, which send their ganesha.nfsd a dbus message putting them into NFS-GRACE
The ganesha.nfsds do whatever they do in grace, e.g. migrate locks, etc.
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP2
block
unblock
ganesha
ganesha_mon deletes ganesha-active
You came in that thing? You're braver than I thought.
Then ganesha-mon deletes the ganesha-active attr….
VIP fails over
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsdblock
unblock
ganesha
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP1
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP3
block
unblock
ganesha
ganesha-activeganesha-grace
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP
block
unblock
pacemaker
glusterd
glusterfsd
mon
nfsd
grace
VIP4
block
unblock
ganesha
ganesha-activeganesha-grace
VIP2
You can't win, Darth. If you strike me down I will become more powerful than you could possibly imagine.
Which causes the floating IP addr to fail-over to the next node based on the location constraints.
When the operator restarts the ganesha.nfsd, everything “unwinds”. The ganesha.nfsds are put into grace, and the floating IP fails back to the node.
In storehaug (Jose’s version) gluster/glusterd don’t run ganesha-ha.sh –setup, that’s done by the admin (or some other thing TBD. gluster/glusterd doesn’t start the ganesha.nfsds. Instead they’re started by the ganesha_nfsd RA along with the Samba server after /var/lib/nfs is setup.
Next Steps
● Switch to storhaug (anyone fancy a trip to Norway?)
● Fencing?
I've got a bad feeling abou this.